Method and device for processing audio data, medium and equipment

Abstract

The invention relates to a method and device for processing audio data, a medium and equipment. Diffusive or spatially large audio objects may be identified for special processing. A decorrelation process may be performed on an audio signal corresponding to a large audio object to produce a decorrelated large audio object audio signal. These decorrelated large audio object audio signals may be associated with object locations, which may be fixed or varying over time. For example, a decorrelated large audio object audio signal may be rendered to a virtual or actual speaker position. The output of such a rendering process may be input to a scene simplification process. The decorrelation, association, and / or scene simplification process may be performed prior to the process of encoding the audio data.

Description

[0001] This application is a divisional application of the invention patent application with application number 201911130633.3, application date July 24, 2014, and invention name “Method and device, medium and equipment for processing audio data”. The invention patent application with application number 201911130633.3 is a divisional application of the invention patent application with application number 201480043090.0, application date July 24, 2014, and invention name “Processing of spatial dispersion or large audio objects”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to Spanish Patent Application No. P201331193, filed on July 31, 2013, and U.S. Provisional Application No. 61 / 885,805, filed on October 2, 2013, the entire contents of each of which are incorporated herein by reference. Technical Field

[0004] The present disclosure relates to processing audio data. More particularly, the present disclosure relates to processing audio data corresponding to diffuse or spatially large audio objects. Background Art

[0005] Since the introduction of sound in motion pictures in 1927, the technology used to capture the artistic intent of motion picture soundtracks and reproduce such content has been steadily evolving. In the 1970s, Dolby introduced a cost-effective means of encoding and distributing mixes using three screen channels and a single surround channel. In the 1990s, Dolby brought digital sound to movie theaters with a 5.1 channel format that provided discrete left, center, and right screen channels, left and right surround arrays, and a subwoofer channel for low-frequency effects. Dolby Surround 7.1, introduced in 2010, increased the number of surround channels by dividing the existing left and right surround channels into four "zones."

[0006] Cinema and home theater audio playback systems are becoming more and more versatile and complex. Home theater audio playback systems include an increasing number of speakers. As the number of channels increases and the speaker layout changes from a flat two-dimensional (2D) array to a three-dimensional (3D) array that includes elevation, reproducing sound in a playback environment is becoming a more complex process. Improved audio processing methods would be desirable. Summary of the invention

[0007] Improved methods for processing diffuse or spatially large audio objects are provided. As used herein, the term "audio object" refers to an audio signal (also referred to herein as an "audio object signal") and associated metadata that can be created or "authored" without reference to any particular playback environment. The associated metadata may include audio object position data, audio object gain data, audio object size data, audio object trajectory data, and the like. As used herein, the term "rendering" refers to the process of transforming an audio object into a speaker feed signal for a specific playback environment. The rendering process can be performed at least in part based on the associated metadata and based on playback environment data. The playback environment data may include an indication of the number of speakers in the playback environment and an indication of the position of each speaker within the playback environment.

[0008] Spatially large audio objects are not to be perceived as point sources, but should be perceived as covering a large spatial area. In some cases, large audio objects should be perceived as surrounding the listener. This audio effect may not be achieved simply by panning, but may require additional processing. In order to create a convincing spatial object size, or spatial diffuseness, a significant proportion of the speaker signals in the playback environment should be independent of each other, or at least uncorrelated (e.g., independent in terms of first-order cross-correlation or covariance). Sufficiently complex rendering systems, such as those used in theaters, may be able to provide this decorrelation. However, less complex rendering systems, such as those to be used in home theater systems, may not be able to provide sufficient decorrelation.

[0009] Some embodiments described herein may involve identifying diffuse or spatially large audio objects for special processing. A decorrelation process may be performed on audio signals corresponding to large audio objects to produce decorrelated large audio object audio signals. These decorrelated large audio object audio signals may be associated with object positions, which may be static or time-varying positions. The association process may be independent of the actual playback speaker configuration. For example, the decorrelated large audio object audio signals may be rendered to virtual speaker positions. In some embodiments, the output of this rendering process may be input to a scene simplification process.

[0010] Thus, at least some aspects of the present disclosure may be implemented in a method that may involve receiving audio data including an audio object. The audio object may include an audio object signal and associated metadata. The metadata may include at least audio object size data.

[0011] The method may involve, based on the audio object size data, determining a large audio object whose audio object size is greater than a threshold size and performing a decorrelation process on an audio signal of the large audio object to produce a decorrelated large audio object audio signal. The method may involve associating the decorrelated large audio object audio signal with an object position. The association process may be independent of an actual playback speaker configuration. The actual playback speaker configuration may ultimately be used to render the decorrelated large audio object audio signal to speakers of a playback environment.

[0012] The method may involve receiving decorrelation metadata for the large audio object. The decorrelation process may be performed at least in part based on the decorrelation metadata. The method may involve encoding the audio data output from the association process. In some embodiments, the encoding process may not involve encoding decorrelation metadata for the large audio object.

[0013] The object positions may include positions corresponding to at least some of the audio object position data of the received audio objects. At least some of the object positions may be fixed. However, in some embodiments, at least some of the object positions may vary over time.

[0014] The associating process may involve rendering decorrelated large audio object audio signals according to virtual speaker positions. In some examples, the receiving process may involve receiving one or more audio bed signals corresponding to the speaker positions. The method may involve mixing at least some of the received audio bed signals or the received audio object signals with the decorrelated large audio object audio signals. The method may involve outputting the decorrelated large audio object audio signals as additional audio bed signals or audio object signals.

[0015] The method may involve applying a level adjustment process to the decorrelated large audio object audio signal.In some implementations, the large audio object metadata may include audio object position metadata and the level adjustment process may rely at least in part on audio object size metadata and audio object position metadata of the large audio object.

[0016] The method may involve attenuating or deleting the audio signal of the large audio object after performing the decorrelation process. However, in some embodiments, the method may involve retaining the audio signal of the point source contribution corresponding to the large audio object after performing the decorrelation process.

[0017] The large audio object metadata may include audio object position metadata. In some such embodiments, the method may involve calculating contributions from virtual sources within an audio object area or volume defined by the large audio object position data and the large audio object size data. The method may also involve determining a set of audio object gain values ​​for each of a plurality of output channels based at least in part on the calculated contributions. The method may involve mixing decorrelated large audio object audio signals with audio signals for audio objects that are spatially separated from the large audio object by a threshold amount.

[0018] In some implementations, the method may involve performing an audio object clustering process after the decorrelation process. In some such implementations, the audio object clustering process may be performed after the association process.

[0019] The method may involve evaluating the audio data to determine the content type. In some such embodiments, a decorrelation process may be selectively performed based on the content type. For example, the amount of decorrelation to be performed may depend on the content type. The decorrelation process may involve a delay, an all-pass filter, a pseudo-random filter, and / or a reverberation algorithm.

[0020] The methods disclosed herein may be implemented via hardware, firmware, software stored in one or more non-transient media, and / or a combination thereof. For example, at least some aspects of the present disclosure may be implemented in a device including an interface system and a logic system. The interface system may include a user interface and / or a network interface. In some embodiments, the device may include a memory system. The interface system may include at least one interface between the logic system and the memory system.

[0021] The logic system may include at least one processor, such as a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, and / or combinations thereof.

[0022] In some embodiments, the logic system may be capable of receiving audio data including an audio object via the interface system. The audio object may include an audio object signal and associated metadata. In some embodiments, the metadata includes at least audio object size data. The logic system may be capable of determining a large audio object having an audio object size greater than a threshold size based on the audio object size data and performing a decorrelation process on the audio signal of the large audio object to produce a decorrelated large audio object audio signal. The logic system may be capable of associating the decorrelated large audio object audio signal with an object position.

[0023] The association process may be independent of the actual playback speaker configuration. For example, the association process may involve rendering the decorrelated large audio object audio signal according to the virtual speaker position. The actual playback speaker configuration may ultimately be used to render the decorrelated large audio object audio signal to the speakers of the playback environment.

[0024] The logic system may be capable of receiving decorrelation metadata for the large audio object via the interface system.The decorrelation process may be performed at least in part based on the decorrelation metadata.

[0025] The logic system may be capable of encoding the audio data output from the association process.In some implementations, the encoding process may not involve encoding decorrelation metadata for the large audio objects.

[0026] At least some of the object positions may be fixed. However, at least some of the object positions may vary over time. The large audio object metadata may include audio object position metadata. The object positions may include positions corresponding to at least some of the audio object position metadata of the received audio objects.

[0027] The receiving process may involve receiving one or more audio bed signals corresponding to speaker positions. The logic system may be capable of mixing at least some of the received audio bed signals or received audio object signals with the decorrelated large audio object audio signal. The logic system may be capable of outputting the decorrelated large audio object audio signal as an additional audio bed signal or audio object signal.

[0028] The logic system may be capable of applying a level adjustment process to the decorrelated large audio object audio signal.The level adjustment process may rely at least in part on the audio object size metadata and the audio object position metadata of the large audio object.

[0029] The logic system may be capable of attenuating or removing the audio signal of the large audio object after performing the decorrelation process. However, the apparatus may be capable of retaining the audio signal of the point source contribution corresponding to the large audio object after performing the decorrelation process.

[0030] The logic system may be capable of calculating contributions from virtual sources within an audio object area or volume defined by the large audio object position data and the large audio object size data. The logic system may be capable of determining a set of audio object gain values ​​for each of a plurality of output channels based at least in part on the calculated contributions. The logic system may be capable of mixing decorrelated large audio object audio signals with audio signals for audio objects spatially separated from the large audio object by a threshold amount.

[0031] The logic system may be capable of performing the audio object clustering process after the decorrelation process.In some implementations, the audio object clustering process may be performed after the association process.

[0032] The logic system may be capable of evaluating the audio data to determine the content type. The decorrelation process may be selectively performed based on the content type. For example, the amount of decorrelation to be performed depends on the content type. The decorrelation process may involve delays, all-pass filters, pseudo-random filters, and / or reverberation algorithms.

[0033] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. It should be noted that the relative sizes of the following drawings may not be drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 An example of a playback environment with a Dolby Surround 5.1 configuration is shown.

[0035] Figure 2 An example of a playback environment with a Dolby Surround 7.1 configuration is shown.

[0036] Figure 3A and 3B Two examples of home theater playback environments including height speaker configurations are shown.

[0037] Figure 4A An example of a graphical user interface (GUI) depicting speaker zones of varying elevation angles in a virtual playback environment is shown.

[0038] Figure 4B Another example of a playback environment is shown.

[0039] Figure 5 is a flow chart providing an example of audio processing for spatially large audio objects.

[0040] Figures 6A-6F is a block diagram illustrating an example of components of an audio processing device capable of processing large audio objects.

[0041] Figure 7 is a block diagram illustrating an example of a system capable of performing a clustering process.

[0042] Figure 8 is a block diagram illustrating an example of a system capable of clustering objects and / or beds in an adaptive audio processing system.

[0043] Fig. 9 is a block diagram providing an example of a clustering process followed by a decorrelation process for a large audio object.

[0044] Fig. 10A An example of a virtual source position relative to a playback environment is shown.

[0045] Fig. 10B Alternative examples of virtual source positions relative to the playback environment are shown.

[0046] Fig.11 is a block diagram providing an example of components of an audio processing device.

[0047] Like reference numbers and symbols refer to like elements throughout the various drawings. DETAILED DESCRIPTION

[0048] The following description is directed to certain embodiments for the purpose of describing some innovative aspects of the present disclosure, and examples of situations in which these innovative aspects can be implemented. However, the teachings of this article can be applied in various different ways. For example, although various embodiments are described with respect to a specific playback environment, the teachings of this article are widely applicable to other known playback environments, and playback environments that may be introduced in the future. Moreover, the described embodiments can be implemented as hardware, software, firmware, cloud-based systems, etc. in various devices and systems, at least in part. Therefore, the teachings of the present disclosure are not intended to be limited to the embodiments shown in the drawings and / or described herein, but have wide applicability.

[0049] Figure 1 An example of a playback environment with a Dolby Surround 5.1 configuration is shown. In this example, the playback environment is a movie theater playback environment. Dolby Surround 5.1 was developed in the 1990s, but this configuration is still widely deployed in home and movie theater playback environments. In a movie theater playback environment, a projector 105 can be configured to project a video image, such as a movie, onto a screen 150. Audio data can be synchronized with the video image and processed by a sound processor 110. A power amplifier 115 can provide a speaker feed signal to a speaker of the playback environment 100.

[0050] The Dolby Surround 5.1 configuration includes a left surround channel 120 for a left surround array 122 and a right surround channel 125 for a right surround array 127. The Dolby Surround 5.1 configuration also includes a left channel 130 for a left speaker array 132, a center channel 135 for a center speaker array 137, and a right channel 140 for a right speaker array 142. In a movie theater environment, these channels may be referred to as a left screen channel, a center screen channel, and a right screen channel, respectively. An independent low frequency effects (LFE) channel 144 is provided for a subwoofer 145.

[0051] In 2010, Dolby provided an enhancement to digital cinema sound with the introduction of Dolby Surround 7.1. Figure 2An example of a playback environment having a Dolby Surround 7.1 configuration is shown. Digital projector 205 may be configured to receive digital video data and project a video image on screen 150. Audio data may be processed by sound processor 210. Power amplifier 215 may provide speaker feed signals to speakers of playback environment 200.

[0052] Like Dolby Surround 5.1, the Dolby Surround 7.1 configuration includes a left channel 130 for a left speaker array 132, a center channel 135 for a center speaker array 137, a right channel 140 for a right speaker array 142, and an LFE channel 144 for a subwoofer 145. The Dolby Surround 7.1 configuration includes a left surround (Lss) array 220 and a right surround (Rss) array 225, each of which can be driven by a single channel.

[0053] However, Dolby Surround 7.1 increases the number of surround channels by dividing the left and right surround channels of Dolby Surround 5.1 into four zones: separate channels for left rear surround (Lrs) speaker 224 and right rear surround (Rrs) speaker 226 in addition to left surround array 220 and right surround array 225. Increasing the number of surround zones in playback environment 200 can significantly improve the localization of sound.

[0054] In an effort to create a more immersive environment, some playback environments may be configured with an increased number of speakers driven by an increased number of channels. Also, some playback environments may include speakers deployed at various elevation angles, some of which may be "height speakers" configured to produce sound from an area above the seating area of ​​the playback environment.

[0055] Figure 3A and 3B Two examples of home theater playback environments including height speaker configurations are shown. In these examples, playback environments 300a and 300b include the main features of a Dolby Surround 5.1 configuration, including left surround speaker 322, right surround speaker 327, left speaker 332, right speaker 342, center speaker 337, and subwoofer 145. However, playback environment 300 includes an extension of the Dolby Surround 5.1 configuration to height speakers, which may be referred to as a Dolby Surround 5.1.2 configuration.

[0056] Figure 3A An example of a home theater playback environment with height speakers mounted on a ceiling 360 is shown. In this example, the playback environment 300a includes a height speaker 352 located at the left top center (Ltm) position and a height speaker 357 located at the right top center (Rtm) position. Figure 3BIn the example shown, the left speaker 332 and the right speaker 342 are Dolby elevation speakers configured to reflect sound from the ceiling 360. If configured correctly, the reflected sound can be perceived by the listener 365 as if the sound source originated from the ceiling 360. However, the number and configuration of speakers are provided as examples only. Some current home theater implementations provide up to 34 speaker positions, and contemplated home theater implementations may allow for more speaker positions.

[0057] Therefore, the modern trend is to include not only more speakers and more channels, but also speakers at different heights. As the number of channels increases and the speaker layout moves from 2D to 3D, the task of positioning and rendering the sound becomes increasingly difficult.

[0058] Therefore, Dolby has developed various tools to increase functionality and / or reduce creation complexity for 3D audio sound systems, including but not limited to user interfaces. Some of these tools can be used to create audio objects and / or metadata for audio objects.

[0059] Figure 4A An example of a graphical user interface (GUI) depicting speaker zones of varying elevation angles in a virtual playback environment is shown. GUI 400 may be displayed on a display device, for example, based on instructions from a logic system, based on signals received from a user input device, etc. Some such devices will be described below with reference to Fig.11 to describe.

[0060] As used herein with reference to a virtual playback environment such as virtual playback environment 404, the term "speaker zone" generally refers to a logical construct that may or may not have a one-to-one correspondence with speakers of an actual playback environment. For example, a "speaker zone location" may or may not correspond to a particular speaker location of a movie theater playback environment. Instead, the term "speaker zone location" may generally refer to a zone of a virtual playback environment. In some implementations, a speaker zone of a virtual playback environment may correspond to virtual speakers, for example, via the use of virtualization technology, such as a Dolby Headphones headset that creates a virtual surround sound environment in real time using a collection of two-channel stereo headphones. TM (Sometimes called Mobile Surround TM ). In the GUI 400, there are seven speaker zones 402a at a first elevation angle and two speaker zones 402b at a second elevation angle, making a total of nine speaker zones in the virtual playback environment 404. In this example, speaker zones 1-3 are in a front region 405 of the virtual playback environment 404. The front region 405 may, for example, correspond to the region where the screen 150 is located in a movie theater playback environment, to the region where a television screen is located at home, and so on.

[0061] Here, speaker zone 4 generally corresponds to speakers in left area 410 of virtual playback environment 404 and speaker zone 5 corresponds to speakers in right area 415. Speaker zone 6 corresponds to left rear area 412 of virtual playback environment 404 and speaker zone 7 corresponds to right rear area 414. Speaker zone 8 corresponds to speakers in upper area 420a and speaker zone 9 corresponds to speakers in upper area 420b (which may be a virtual ceiling area). Figure 4A The positions of the speaker zones 1-9 shown in FIG. 1 may or may not correspond to the speaker positions of an actual playback environment. Moreover, other embodiments may include more or fewer speaker zones and / or elevation angles.

[0062] In various embodiments described herein, a user interface such as GUI 400 may be used as part of an authoring tool and / or a rendering tool. In some embodiments, the authoring tool and / or the rendering tool may be implemented via software stored on one or more non-transitory media. The authoring tool and / or the rendering tool may be implemented (at least in part) by hardware, firmware (such as described below with reference to Fig.11 The invention can be implemented by using a rendering tool (such as a rendering engine, a rendering engine, or a rendering engine) and the like. In some creative implementations, an associated creative tool can be used to create metadata for associated audio data. The metadata can, for example, include data indicating the position and / or trajectory of an audio object in three-dimensional space, speaker zone constraint data, and the like. The metadata can be created with respect to a speaker zone 402 of a virtual playback environment 404, rather than with respect to a specific speaker layout of an actual playback environment. A rendering tool can receive the audio data and the associated metadata, and can calculate audio gains and speaker feed signals for the playback environment. Such audio gains and speaker feed signals can be calculated based on an amplitude panning process, which can produce a sense that the sound is coming from a position P in the playback environment. For example, a speaker feed signal can be provided to speakers 1 to N of the playback environment according to the following formula:

[0063] x i (t) = g i x(t),i=1,...N(Formula 1).

[0064] In formula 1, x i (t) represents the loudspeaker feed signal to be applied to loudspeaker i, g irepresents the gain factor of the corresponding channel, x(t) represents the audio signal, and t represents time. The gain factor can be determined, for example, according to the amplitude panning method described in "V. Pulkki, Compensating Displacement of Amplitude-Panned Virtual Sources (Audio Engineering Society (AES) International Conference on Virtual, Synthetic and Entertainment Audio)", Part 2, pages 3-4, which is incorporated herein by reference. In some embodiments, the gain can be frequency-dependent. In some embodiments, a time delay can be introduced by replacing x(t) with x(t-Δt).

[0065] In some rendering embodiments, audio reproduction data created with reference to speaker zones 402 may be mapped to speaker locations for a wide range of playback environments, such as a Dolby Surround 5.1 configuration, a Dolby Surround 7.1 configuration, a Hamasaki 22.2 configuration, or other configurations. Figure 2 , the rendering tool may map audio reproduction data for speaker zones 4 and 5 to a left surround array 220 and a right surround array 225 of a playback environment having a Dolby Surround 7.1 configuration. Audio reproduction data for speaker zones 1, 2, and 3 may be mapped to a left screen channel 230, a right screen channel 240, and a center screen channel 235, respectively. Audio reproduction data for speaker zones 6 and 7 may be mapped to a left rear surround speaker 224 and a right rear surround speaker 226.

[0066] Figure 4B An example of another playback environment is shown. In some implementations, the rendering tool may map audio reproduction data for speaker zones 1, 2, and 3 to corresponding screen speakers 455 of the playback environment 450. The rendering tool may map audio reproduction data for speaker zones 4 and 5 to left surround arrays 460 and right surround arrays 465 and may map audio reproduction data for speaker zones 8 and 9 to left overhead speakers 470a and right overhead speakers 470b. Audio reproduction data for speaker zones 6 and 7 may be mapped to left rear surround speakers 480a and right rear surround speakers 480b.

[0067] In some authoring implementations, an authoring tool may be used to create metadata for audio objects. The metadata may indicate the object's 3D position, rendering constraints, content type (e.g., dialogue, effects, etc.), and / or other information. Depending on the implementation, the metadata may include other types of data, such as width data, gain data, trajectory data, etc. Some audio objects may be static, while others may move.

[0068] The audio object is rendered according to its associated metadata, which generally includes position metadata indicating the position of the audio object in three-dimensional space at a given point in time. When the audio object is monitored or played back in a playback environment, the audio object is rendered according to the position metadata using speakers present in the playback environment, rather than being output to predetermined physical channels as with traditional channel-based systems (such as Dolby 5.1 and Dolby 7.1).

[0069] In addition to position metadata, other types of metadata may also be necessary to produce the desired audio effect. For example, in some embodiments, metadata associated with an audio object may indicate the audio object size, which may also be referred to as "width". Size metadata may be used to indicate the spatial area or volume occupied by the audio object. Spatially large audio objects should be perceived as covering a large spatial area, rather than just as point sound sources with positions defined only by the audio object position metadata. In some cases, for example, large audio objects should be perceived as occupying a significant portion of the playback environment, potentially even surrounding the listener.

[0070] The human auditory system is very sensitive to changes in the correlation or coherence of the signals arriving at the two ears, and maps this correlation into a perceived object size attribute if the normalized correlation is less than a value of +1. Therefore, in order to create convincing spatial object sizes, or spatial diffuseness, a significant proportion of the loudspeaker signals in the playback environment should be independent of each other, or at least uncorrelated (e.g., independent in terms of first-order cross-correlations or covariances). Satisfactory decorrelation processes are usually quite complex, often involving time-varying filters.

[0071] A cinema soundtrack may include hundreds of objects, each with its associated position metadata / size metadata and possibly other spatial metadata. Furthermore, a cinema sound system may include hundreds of speakers that may be individually controlled to provide a satisfactory perception of audio object position and size. Thus, in a cinema, hundreds of objects may be reproduced by hundreds of speakers, and the mapping of objects to speaker signals consists of a very large matrix of panning coefficients. When the number of objects is given by M and the number of speakers is given by N, this matrix has up to M*N elements. This implies implications for the reproduction of diffuse or large-sized objects. In order to create convincing spatial object sizes, or spatial diffuseness, a significant proportion of the N speaker signals should be independent of each other, or at least uncorrelated. This generally involves the use of many (up to N) independent decorrelation processes, thereby placing a significant processing load on the rendering process. Moreover, the amount of decorrelation may be different for each object, which further complicates the rendering process. Sufficiently complex rendering systems, such as those used in commercial theaters, may be able to provide such decorrelation.

[0072] However, less complex rendering systems, such as those used in home theater systems, may not be able to provide adequate decorrelation. Some such rendering systems cannot provide decorrelation at all. Decorrelation procedures that are simple enough to be performed on home theater systems can introduce artifacts. For example, if a low-complexity decorrelation process is followed by a downmix process, comb-filter artifacts may be introduced.

[0073] Another potential problem is that in some applications, object-based audio is sent in the form of a backward-compatible mix (such as Dolby Digital or Dolby Digital Plus), enhanced with additional information for retrieving one or more objects from the backward-compatible mix. Backward-compatible mixes typically do not include decorrelation effects. In some such systems, reconstruction of objects can only work reliably when the backward-compatible mix is ​​created using a simple panning process. Using a decorrelator in such a process can harm the audio object reconstruction process, sometimes severely. In the past, this meant choosing not to apply decorrelation in a backward-compatible mix, thereby degrading the artistic intent of the mix, or accepting degradation in the object reconstruction process.

[0074] To address these potential issues, some embodiments described herein involve identifying diffuse or spatially large audio objects for special processing. Such methods and apparatus may be particularly suitable for audio data to be rendered in a home theater. However, these methods and apparatus are not limited to home theater use, but have broad applicability.

[0075] Due to their spatially diffuse nature, objects with large sizes are not perceived as point sources with compact and concise locations. Therefore, multiple speakers are used to reproduce such spatially diffuse objects. However, the exact locations of the speakers used to reproduce large audio objects in a playback environment are not as critical as the locations of the speakers used to reproduce compact, small-sized audio objects. Therefore, high-quality reproduction of large audio objects is possible without a priori knowledge of the actual playback speaker configuration used to ultimately render decorrelated large audio object signals to the actual speakers of the playback environment. Therefore, the decorrelation process for large audio objects can be performed "upstream" before the process of rendering audio data for reproduction in a playback environment (e.g., a home theater system) for a listener. In some examples, the decorrelation process for large audio objects is performed before encoding audio data for transmission to such a playback environment.

[0076] This implementation does not require the renderer of the playback environment to be capable of high complexity decorrelation, thereby allowing the rendering process to be relatively simpler, more efficient and less expensive. A backwards-compatible downmix can include the effect of decorrelation to maintain the best possible artistic intent without the need to reconstruct objects for decorrelation on the rendering side. A high-quality decorrelator can be applied to large audio objects upstream of the final rendering process, for example, during the creation or post-production process in a recording studio. Such a decorrelator can be robust with respect to downmixing and / or other downstream audio processing.

[0077] Figure 5 5 is a flow chart providing an example of audio processing for a spatially large audio object. As with other methods described herein, the operations of method 500 need not be performed in the order indicated. Moreover, these methods may include more or fewer blocks than shown and / or described. These methods may be implemented at least in part by a logic system, such as in Fig.11 The method may be performed by the logic system 1110 shown in and described below. Such a logic system may be a component of an audio processing system. Alternatively, or in addition, such a method may be implemented via a non-transitory medium having software stored thereon. The software may include instructions for controlling one or more devices to at least partially perform the methods described herein.

[0078] In this example, method 500 begins at block 505, which involves receiving audio data including an audio object. The audio data may be received by an audio processing system. In this example, the audio object includes an audio object signal and associated metadata. Here, the associated metadata includes audio object size data. The associated metadata may also include audio object position data indicating the position of the audio object in three-dimensional space, decorrelation metadata, audio object gain information, and the like. The audio data may also include one or more audio bed signals corresponding to speaker positions.

[0079] In such an embodiment, block 510 involves, based on the audio object size data, determining a large audio object having an audio object size greater than a threshold size. For example, block 510 may involve determining whether a numerical audio object size value exceeds a predetermined level. The numerical audio object size value may, for example, correspond to a portion of a playback environment occupied by the audio object. Alternatively, or in addition, block 510 may involve determining whether another type of indication, such as a tag, decorrelation metadata, etc., indicates that the audio object has an audio object size greater than a threshold size. Although much of the discussion of method 500 involves processing a single large audio object, it should be appreciated that the same (or similar) process may be applied to multiple large audio objects.

[0080] In this example, block 515 involves performing a decorrelation process on the audio signal of the large audio object, thereby generating a decorrelated large audio object audio signal. In some embodiments, the decorrelation process can be performed at least in part based on the received decorrelation metadata. The decorrelation process can involve delays, all-pass filters, pseudo-random filters, and / or reverberation algorithms.

[0081] Here, in box 520, the decorrelated large audio object audio signal is associated with the object position. In this example, the association process is independent of the actual playback speaker configuration that can be used to ultimately render the decorrelated large audio object audio signal to the actual playback speakers of the playback environment. However, in some alternative embodiments, the object position may correspond to the actual playback speaker position. For example, according to some such alternative embodiments, the object position may correspond to the playback speaker position of a commonly used playback speaker configuration. If an audio bed signal is received in box 505, the object position may correspond to the playback speaker position corresponding to at least some of the audio bed signals. Alternatively, or in addition, the object position may be a position corresponding to at least some of the audio object position data of the received audio object. Therefore, at least some of the object positions may be stationary, while at least some of the object positions may change over time. In some embodiments, box 520 may involve mixing the decorrelated large audio object audio signal with an audio signal for an audio object that is spatially separated from the large audio object by a threshold distance.

[0082] In some embodiments, block 520 may involve rendering decorrelated large audio object audio signals according to virtual speaker positions. Some such embodiments may involve calculating contributions from virtual sources within an audio object area or volume defined by large audio object position data and large audio object size data. Such embodiments may involve determining a set of audio object gain values ​​for each of a plurality of output channels based at least in part on the calculated contributions. Some examples are described below.

[0083] Some embodiments may involve encoding audio data output from the association process. According to some such embodiments, the encoding process involves encoding audio object signals and associated metadata. In some embodiments, the encoding process includes a data compression process. The data compression process can be lossless or lossy. In some embodiments, the data compression process involves a quantization process. According to some examples, the encoding process does not involve encoding decorrelation metadata for large audio objects.

[0084] Some embodiments involve performing an audio object clustering process, also referred to herein as a "scene simplification" process. For example, the audio object clustering process can be part of block 520. For embodiments involving encoding, the encoding process can involve encoding audio data output from the audio object clustering process. In some such embodiments, the audio object clustering process can be performed after a decorrelation process. More examples of processes corresponding to the blocks of method 500, including scene simplification processes, are provided below.

[0085] Figures 6A-6F is a block diagram illustrating an example of components of an audio processing system capable of processing large audio objects as described herein. These components may, for example, correspond to modules of a logic system of the audio processing system, which may be implemented via hardware, firmware, software stored in one or more non-transitory media, or a combination thereof. The logic system may include one or more processors, such as general-purpose single-chip or multi-chip processors. The logic system may include a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, and / or combinations thereof.

[0086] exist Fig. 6A , the audio processing system 600 is capable of detecting large audio objects, such as large audio object 605. The detection process may be substantially similar to that of reference Figure 5 In this example, the audio signal of the large audio object 605 is decorrelated by the decorrelation system 610 to produce a decorrelated large audio object signal 611. The decorrelation system 610 can perform the decorrelation process based at least in part on the received decorrelation metadata for the large audio object 605. The decorrelation process can involve one or more of a delay, an all-pass filter, a pseudo-random filter, or a reverberation algorithm.

[0087] The audio processing system 600 can also receive other audio signals, in this example other audio objects and / or beds 615. Here, other audio objects are audio objects having a size below a threshold size for characterizing an audio object as a large audio object.

[0088] In this example, the audio processing system 600 is able to associate the decorrelated large audio object audio signal 611 with other object positions. The object positions may be fixed or may vary over time. The association process may be similar to that described above with reference to Figure 5 One or more of the processes described in block 520 .

[0089] The association process may involve a mixing process. The mixing process may be based at least in part on the distance between the large audio object position and the other object position. Fig. 6A In the embodiment shown in , the audio processing system 600 is capable of mixing at least some audio signals corresponding to audio objects and / or beds 615 with the decorrelated large audio object signal 611. For example, the audio processing system 600 may be capable of mixing the decorrelated large audio object audio signal 611 with audio signals for other audio objects that are spatially separated from the large audio object by a threshold amount of distance.

[0090] In some embodiments, the association process may involve a rendering process. For example, the association process may involve rendering a decorrelated large audio object audio signal according to the virtual speaker position. Some examples are described below. After the rendering process, it may not be necessary to retain the audio signal corresponding to the large audio object received by the decorrelation system 610. Therefore, the audio processing system 600 can be configured to attenuate or delete the audio signal of the large audio object 605 after the decorrelation process is performed by the decorrelation system 610. Alternatively, the audio processing system 600 can be configured to retain at least a portion of the audio signal of the large audio object 605 (e.g., an audio signal corresponding to a point source contribution of the large audio object 605) after performing the decorrelation process.

[0091] In this example, the audio processing system 600 includes an encoder 620 capable of encoding audio data. Here, the encoder 620 is configured to encode the audio data after the association process. In this embodiment, the encoder 620 can apply a data compression process to the audio data. The encoded audio data 622 can be stored and / or sent to other audio processing systems for downstream processing, playback, etc.

[0092] exist Figure 6BIn the illustrated embodiment, the audio processing system 600 is capable of level adjustment. In this example, the level adjustment system 612 is configured to adjust the level of the output of the decorrelation system 610. The level adjustment process can rely on metadata of the audio objects in the original content. In this example, the level adjustment process relies at least in part on the audio object size metadata and the audio object position metadata of the large audio object 605. This level adjustment can be used to optimize the allocation of the decorrelator output to other audio objects (such as audio objects and / or bed 615). The decorrelator output can be optionally mixed to other object signals that are spatially far away so as to improve the spatial diffuseness of the resulting rendering.

[0093] Alternatively, or in addition, a level adjustment process may be used to ensure that the sound corresponding to the decorrelated large audio object 605 is reproduced only by speakers from a certain direction. This may be achieved by only adding the decorrelator output to objects near the desired direction or location. In such an embodiment, the position metadata of the large audio object 605 is included in the level adjustment process in order to preserve information about the direction from which the perceived sound is coming. Such an embodiment may be appropriate for medium-sized objects, such as audio objects that are considered large, but not so large that their size encompasses the entire reproduction / playback environment.

[0094] exist Figure 6C In the illustrated embodiment, the audio processing system 600 is capable of creating additional object or bed channels during the decorrelation process. Such functionality may be desirable, for example, if the other audio objects and / or beds 615 are not suitable or optimal. For example, in some embodiments, the decorrelated large audio object signal 611 may correspond to a virtual speaker position. If the other audio objects and / or beds 615 do not correspond to a position close enough to the desired virtual speaker position, the decorrelated large audio object signal 611 may correspond to a new virtual speaker position.

[0095] In this example, the large audio object 605 is first processed by a decorrelation system 610. Subsequently, additional objects or bed channels corresponding to the decorrelated large audio object signal 611 are provided to the encoder 620. In this example, the decorrelated large audio object signal 611 is level adjusted before being sent to the encoder 620. The decorrelated large audio object signal 611 can be a bed channel signal and / or an audio object signal, the latter of which can correspond to a stationary or moving object.

[0096] In some embodiments, the audio signal output to the encoder 620 may also include at least some of the original large audio object signal. As noted above, the audio processing system 600 may be able to retain the audio signal corresponding to the point source contribution of the large audio object after performing decorrelation processing. This may be beneficial, for example, because different signals may be correlated to each other to different degrees. Therefore, it may be helpful to render at least a portion of the original audio signal corresponding to the large audio object 605 (e.g., the point source contribution) and separately. In such an embodiment, it may be beneficial to level the decorrelated signal corresponding to the large audio object 605 with the original signal.

[0097] An example of this is Fig.6D 6. In this example, at least some of the original large audio object signals 613 are subjected to a first leveling process by the level adjustment system 612a and the decorrelated large audio object signals 611 are subjected to a leveling process by the level adjustment system 612b. Here, the level adjustment system 612a and the level adjustment system 612b provide output audio signals to the encoder 620. In this example, the output of the level adjustment system 612b is also mixed with other audio objects and / or the bed 615.

[0098] In some embodiments, the audio processing system 600 may be capable of evaluating the input audio data to determine (or at least estimate) the content type. The decorrelation process may be based at least in part on the content type. In some embodiments, the decorrelation process may be selectively performed based on the content type. For example, the amount of decorrelation to be performed on the input audio data may be performed at least in part dependent on the content type. For example, it is generally desirable to reduce the amount of decorrelation used for speech.

[0099] An example is Fig. 6E In this example, media intelligence system 625 is capable of evaluating the audio signal and estimating the content type. For example, media intelligence system 625 may be capable of evaluating the audio signal corresponding to large audio object 605 and estimating the content type to be speech, music, sound effects, etc. Fig. 6E In the example shown, media intelligence system 625 can send control signal 627 to control the amount of decorrelation or size processing of objects based on an estimate of the content type.

[0100] For example, if the media intelligence system 625 estimates that the audio signals of the large audio object 605 correspond to speech, the media intelligence system 625 can send a control signal 627 indicating that the amount of decorrelation for these signals should be reduced or that these signals should not be decorrelated. Various methods of automatically determining the likelihood that a signal is a speech signal can be used. According to one embodiment, the media intelligence system 625 can include a speech likelihood estimator capable of generating a speech likelihood value based at least in part on audio information in the center channel. Some examples are described by Robinson and Vinton in "Automated Speech / Other Discrimination for Loudness Monitoring" (Audio Engineering Society, Preprint number 6437 of Convention 118, May 2005).

[0101] In some implementations, the control signal 627 can indicate an amount of level adjustment and / or can indicate parameters for mixing the decorrelated large audio object signal 611 with the audio signal for the audio object and / or bed 615 .

[0102] Alternatively, or in addition, the amount of decorrelation for large audio objects can be based on "stems," "flags," or other explicit indications of the content type. Such explicit indications of the content type can, for example, be created by a content creator (e.g., during a post-production process) and sent as metadata with the corresponding audio signal. In some embodiments, such metadata can be human-readable. For example, in practice, human-readable stems or flags can explicitly indicate "this is dialogue," "this is a special effect," "this is music," and so on.

[0103] Some embodiments may involve a clustering process that groups objects that are similar in some way, such as in terms of spatial location, spatial size, or content type. Some examples of clustering are described below. Figure 7 and 8 To describe. Fig. 6F In the example shown, objects and / or beds 615a are input to a clustering process 630. A smaller number of objects and / or beds 615b are output from the clustering process 630. The audio data corresponding to the objects and / or beds 615b are mixed with the flattened decorrelated large audio object signal 611. In some alternative embodiments, the clustering process may follow the decorrelation process. An example is provided below with reference to Fig. 9 Such an implementation may, for example, prevent dialogue from being mixed into clusters with undesirable metadata, such as locations not near the center speaker, or large cluster sizes.

[0104] Scene simplification via object clustering

[0105] For ease of the following description, the terms "clustering" and "grouping" or "combining" may be used interchangeably to describe the grouping of objects and / or beds (channels) to reduce the amount of data in a unit of adaptive audio content to be transmitted and rendered in an adaptive audio playback system; and the term "reduction" may be used to refer to the act of performing scene simplification for adaptive audio through such clustering of objects and beds. The terms "clustering", "grouping" or "combining" throughout this description are not limited to a strictly unique assignment of objects or bed channels to a single cluster, but rather, objects or bed channels may be distributed over more than one output bed or cluster using weights or gain vectors that determine the relative contribution of an object or bed signal to an output cluster or output bed signal.

[0106] In one embodiment, an adaptive audio system includes at least one component configured to reduce the bandwidth of object-oriented audio content by object clustering and sensory transparent simplification of a spatial scene created by a combination of a channel bed and objects. The object clustering process performed by the component(s) uses certain information about the objects, which may include spatial location, object content type, temporal attributes, object size, etc., to reduce the complexity of the spatial scene by grouping similar objects into object clusters that replace the original objects.

[0107] Additional audio processing used in standard audio encoding to distribute and render a compelling user experience based on the original complex beds and tracks is generally referred to as scene simplification and / or object clustering. The main purpose of this processing is to reduce the spatial scene through clustering or grouping techniques that reduce the number of individual audio elements (beds and objects) to be delivered to the reproduction device, but still retain enough spatial information so that the perceived difference between the originally authored and the rendered output is minimized.

[0108] The scene simplification process can utilize information about objects, such as spatial location, temporal attributes, content type, size, and / or other appropriate characteristics, to facilitate rendering of objects plus bed content in a reduced bandwidth channel or encoding system in order to dynamically cluster objects to a reduced number. The process can reduce the number of objects by performing one or more of the following clustering operations: (1) clustering objects to objects; (2) clustering objects and beds; and (3) clustering objects and / or beds to objects. In addition, objects can be distributed across two or more clusters. The process can use temporal information about objects to control clustering and declustering of objects.

[0109] In some embodiments, an object cluster replaces the individual waveforms and metadata elements of the constituent objects with a single equivalent waveform and set of metadata, such that the data for N objects is replaced with data for a single object, thus essentially compressing the object data from N to 1. Alternatively, or additionally, an object or bed channel can be distributed across more than one cluster (e.g., using amplitude shifting techniques), thereby reducing the object data from N to M, where M < N. The clustering process can use an error metric based on distortion due to changes in the position, loudness, or other characteristics of the clustered objects to determine the trade-off between clustering compression and the sound degradation of the clustered objects. In some embodiments, the clustering process can be performed synchronously. Alternatively, or additionally, the clustering process can be event-driven, such as by using auditory scene analysis (ASA) and / or event boundary detection, to simplify object control through clustering.

[0110] In some embodiments, the process can utilize an endpoint rendering algorithm and / or knowledge of the device to control clustering. In this way, certain characteristics or attributes of the playback device can be used to inform the clustering process. For example, different clustering schemes can be used for speakers versus headphones or other audio drivers, or different clustering schemes can be used for lossless versus lossy coding, etc.

[0111] Figure 7 is a block diagram showing an example of a system capable of performing a clustering process. As Figure 7 shown, system 700 includes an encoder 704 and a decoder 706 stage that processes an input audio signal to produce an output audio signal at a reduced bandwidth. In some embodiments, part 720 and part 730 can be in different locations. For example, part 720 can correspond to a post-production authoring system and part 730 can correspond to a playback environment, such as a home theater system. In Figure 7 the example shown, part 709 of the input signal is processed by known compression techniques to produce a compressed audio bitstream 705. The compressed audio bitstream 705 can be decoded by decoder stage 706 to produce at least a portion of output 707. Such known compression techniques can involve analyzing the input audio content 709, quantizing the audio data, and then performing compression techniques on the audio data itself, such as masking, etc. The compression technique can be lossy or lossless and can be implemented in a system that allows the user to select the compression bandwidth, such as 192 kbps, 256 kbps, 512 kbps, etc.

[0112] In the adaptive audio system, at least a portion of the input audio includes an input signal 701 containing audio objects, which in turn include audio object signals and associated metadata. The metadata defines certain characteristics of the associated audio content, such as object spatial location, object size, content type, loudness, etc. Any practical number of audio objects (e.g., hundreds of objects) can be processed by the system for playback. In order to facilitate accurate playback of numerous objects in various playback systems and transmission media, the system 700 includes a clustering process or component 702 that reduces the number of objects to a smaller, more manageable number of objects by combining the original objects into smaller groups of objects.

[0113] Therefore, the clustering process builds groups of objects from the original set of individual input objects 701 to produce a smaller number of output groups 703. The clustering process 702 basically processes the metadata of the object and the audio data itself to produce a reduced number of object groups. The metadata can be analyzed to determine which objects are most suitable for combination with other objects at any point in time, and the corresponding audio waveforms for the combined objects can be summed together to produce an alternative or combined object. In this example, the combined object group is then input to an encoder 704, which is configured to generate a bitstream 705 containing audio and metadata for sending to a decoder 706.

[0114] In general, an adaptive audio system incorporating object clustering process 702 includes a component that generates metadata from a raw spatial audio format. System 700 includes a portion of an audio processing system configured to process one or more bitstreams containing both conventional channel-based audio elements and audio object coding elements. Extension layers containing audio object coding elements can be added to a channel-based audio codec bitstream or to an audio object bitstream. Thus, in this example, bitstream 705 includes an extension layer to be processed by a renderer for use with existing speaker and driver designs or with next-generation speakers that utilize individually addressable drivers and driver definitions.

[0115] Spatial audio content from a spatial audio processor may include audio objects, channels, and position metadata. When an object is rendered, it may be assigned to one or more speakers based on the position metadata and the location of the playback speakers. Additionally metadata, such as size metadata, may be associated with an object to change the playback location or otherwise limit the speakers to be used for playback. Metadata may be generated in an audio workstation in response to an engineer's mixing input to provide rendering cues that control spatial parameters (e.g., position, size, speed, intensity, timbre, etc.) and specify which driver(s) or speaker(s) in the listening environment play their respective sounds during a presentation. Metadata may be associated with corresponding audio data in a workstation for packaging and shipping by the spatial audio processor.

[0116] Figure 8 is a block diagram illustrating an example of a system capable of clustering objects and / or beds in an adaptive audio processing system. Figure 8 In the example shown, an object processing component 806 capable of performing scene simplification tasks reads in any number of input audio files and metadata. The input audio files include input objects 802 and associated object metadata, and may include beds 804 and associated bed metadata. Thus, such input files / metadata correspond to "beds" or "objects" tracks.

[0117] In this example, the object processing component 806 can combine media intelligence / content classification, spatial distortion analysis, and object selection / clustering information to create a smaller number of output objects and bed tracks. In particular, objects can be clustered together to create new equivalent objects or object clusters 808, with associated object / cluster metadata. Objects can also be selected for downmixing into the bed. This is Figure 8 The output, shown as a downmix object 810, is input to a renderer 816 for combination with a bed 812 to form an output bed object and associated metadata 820. The output bed configuration 820 (e.g., a Dolby 5.1 configuration) does not necessarily need to match the input bed configuration, for example it could be 9.1 for Atmos cinemas. In this example, new metadata for the output track is generated by combining the metadata from the input tracks, and new audio data for the output track is generated by combining the audio from the input tracks.

[0118] In this embodiment, object processing component 806 can use certain processing configuration information 822. Such processing configuration information 822 can include the number of output objects, frame size and certain media intelligence settings. Media intelligence can be related to determining (or associated with) parameters or characteristics of an object, such as content type (i.e., dialogue / music / effect / etc.), region (segment / category), pre-processing results, auditory scene analysis results, and other similar information. For example, object processing component 806 can determine which audio signals correspond to voice, music and / or special effect sounds. In some embodiments, object processing component 806 can determine at least some of these characteristics by analyzing audio signals. As an alternative, or in addition, object processing component 806 can determine at least some of these characteristics according to associated metadata, such as flags, tags, etc.

[0119] In an alternative embodiment, audio generation can be deferred by keeping references to all original tracks as well as reduction metadata (e.g., which objects belong to which cluster, which objects are to be rendered to the bed, etc.) Such information can be useful, for example, for distributing the functionality of the scene reduction process between a studio and an encoding house, or other similar scenarios.

[0120] Fig. 9 600 is a block diagram that provides an example of a clustering process performed after a decorrelation process for a large audio object. The blocks of the audio processing system 600 may be implemented via any suitable combination of hardware, firmware, software stored in a non-transitory medium, etc. For example, the blocks of the audio processing system 600 may be implemented via a logic system and / or other elements, such as described below with reference to Fig.11 Those described are implemented.

[0121] In this embodiment, the audio processing system 600 receives an audio object O 1 To M Here, the audio object includes an audio object signal and associated metadata, including at least audio object size metadata. The associated metadata may also include audio object position metadata. In this example, the large object detection module 905 is capable of determining a large audio object 605 whose size is greater than a threshold size based at least in part on the audio object size metadata. The large object detection module 905 may, for example, be as described above with reference to Figure 5 The process operates as described in block 510 of .

[0122] In this embodiment, the module 910 can perform a decorrelation process on the audio signal of the large audio object 605 to generate a decorrelated large audio object audio signal 611. In this example, the module 910 can also render the audio signal of the large audio object 605 to the virtual speaker position. Therefore, in this example, the decorrelated large audio object audio signal 611 output by the module 910 corresponds to the virtual speaker position. Fig. 10A and 10B To describe some examples of rendering audio object signals.

[0123] Fig. 10A An example of a virtual source position relative to a playback environment is shown. The playback environment can be an actual playback environment or a virtual playback environment. Virtual source position 1005 and speaker position 1025 are merely examples. However, in this example, the playback environment is a virtual playback environment and speaker position 1025 corresponds to a virtual speaker position.

[0124] In some embodiments, virtual source locations 1005 may be evenly spaced in all directions. Fig. 10A In the example shown, the virtual source locations 1005 are evenly spaced along the x, y, and z axes. The virtual source locations 1005 may be configured as N x Multiply by N y Multiply by N zA rectangular grid of virtual source locations 1005. In some implementations, the value of N may be in the range of 5 to 100. The value of N may depend, at least in part, on the number of speakers in the playback environment (or anticipated to be in the playback environment): it may be desirable to include two or more virtual source locations 1005 between each speaker position.

[0125] However, in alternative embodiments, the virtual source locations 1005 may be spaced differently. For example, in some embodiments, the virtual source locations 1005 may have a first uniform spacing along the x and y axes and a second uniform spacing along the z axis. In other embodiments, the virtual source locations 1005 may be spaced non-uniformly.

[0126] In this example, the audio object volume 1020a corresponds to the size of the audio object. The audio object 1010 can be rendered based on the virtual source position 1005 surrounded by the audio object volume 1020a. Fig. 10A In the example shown, audio object volume 1020a occupies part, but not all, of playback environment 1000a. Larger audio objects may occupy most (or all) of playback environment 1000a. In some examples, if audio object 1010 corresponds to a point source, audio object 1010 may have a size of zero and audio object volume 1020a may be set to zero.

[0127] According to some such embodiments, the authoring tool may link audio object size with decorrelation (e.g., via a decorrelation flag included in the associated metadata) by indicating that decorrelation should be turned on when the audio object size is greater than or equal to a size threshold and that decorrelation should be turned off if the audio object size is below the size threshold. In some embodiments, decorrelation may be controlled (e.g., increased, decreased, or disabled) based on user input regarding the size threshold and / or other input values.

[0128] In this example, a virtual source position 1005 is defined in a virtual source volume 1002. In some implementations, a virtual source volume can correspond to a volume within which an audio object can move. Fig. 10A In the example shown, playback environment 1000a and virtual source volume 1002a are coextensive such that each virtual source location 1005 corresponds to a location in playback environment 1000a. However, in alternative embodiments, playback environment 1000a and virtual source volume 1002a may not be coextensive.

[0129] For example, at least some of virtual source locations 1005 may correspond to locations outside of the playback environment. Fig. 10BAn alternative example of virtual source locations relative to a playback environment is shown. In this example, a virtual source volume 1002b extends outside of the playback environment 1000b. Some virtual source locations 1005 in the audio object volume 1020b are located within the playback environment 1000b and other virtual source locations 1005 located within the audio object volume 1020b are located outside the playback environment 1000b.

[0130] In other embodiments, the virtual source positions 1005 may have a first uniform spacing along the x and y axes and a second uniform spacing along the z axis. The virtual source positions 1005 may be configured as N x Multiply by N y Multiply M z A rectangular grid of virtual source locations 1005. For example, in some embodiments, there may be fewer virtual source locations 1005 along the z-axis than along the x or y-axis. In some such embodiments, the value of N may be in the range of 10 to 100, and the value of M may be in the range of 5 to 10.

[0131] Some embodiments involve calculating a gain value for each virtual source position 1005 in an audio object volume 1020. In some embodiments, a gain value for each of a plurality of output channels of a playback environment (which may be an actual playback environment or a virtual playback environment) is calculated for each virtual source position 1005 in the audio object volume 1020. In some embodiments, the gain values ​​may be calculated by applying a vector-based amplitude panning (“VBAP”) algorithm, a pair-wise panning algorithm, or a similar algorithm to calculate a gain value for a point source located at each virtual source position 1005 in the audio target volume 1020. In other embodiments, a separable algorithm may be applied to calculate a gain value for a point source located at each virtual source position 1005 in the audio target volume 1020. As used herein, a “separate” algorithm is one in which the gain for a given speaker can be expressed as a product of multiple factors (e.g., three factors), where each factor depends on only one of the coordinates of the virtual source position 1005. Examples include algorithms implemented in various existing mixing console panners, including but not limited to the Pro Tool 1000 implemented in digital cinema consoles provided by AMS Neve. TM Software and panner.

[0132] Back again Fig. 9 In this example, the audio processing system 600 also receives the bed channel B 1 To B N , and a low frequency effects (LFE) channel. The audio objects and bed channels are processed according to a scene simplification or "clustering" process, for example, as described above with reference to Figure 7 and 8However, in this example, the LFE channel is not input to the clustering process, but is passed to encoder 620.

[0133] In this embodiment, bed channel B 1 To B N is transformed into a static audio object 917 by module 915. In addition to the audio objects that the large object detection module 905 has determined are not large audio objects, module 920 also receives the static audio objects 917. Here, module 920 also receives decorrelated large audio object signals 611, which in this example correspond to virtual speaker positions.

[0134] In this embodiment, module 920 can render the static object 91, the received audio object and the decorrelated large audio object signal 611 into cluster C 1 To C P In general, module 920 will output a smaller number of clusters than the number of received audio objects. In such an embodiment, module 920 is able to associate the decorrelated large audio object signal 611 with the location of the appropriate clusters, for example, as described above with reference to Figure 5 As described in block 520 of FIG.

[0135] In this example, cluster C 1 To C P The audio data of the LFE channel and the LFE channel are encoded by the encoder 620 and sent to the playback environment 925. In some embodiments, the playback environment 925 may include a home theater system. The audio processing system 930 is capable of receiving and decoding the encoded audio data, and rendering the decoded audio data according to the actual playback speaker configuration of the actual playback speakers of the playback environment 925, such as speaker positions, speaker capabilities (e.g., bass reproduction capabilities), etc.

[0136] Fig.11 1 is a block diagram providing an example of components of an audio processing device. In this example, the audio processing system 1100 includes an interface system 1105. The interface system 1105 may include a network interface, such as a wireless network interface. Alternatively, or in addition, the interface system 1105 may include a universal serial bus (USB) interface or another such interface.

[0137] The audio processing system 1100 includes a logic system 1110. The logic system 1110 may include a processor, such as a general-purpose single-chip or multi-chip processor. The logic system 1110 may include a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, or discrete hardware components, or a combination thereof. The logic system 1110 may be configured to control other components of the audio processing system 1100. Although in Fig.11 1100, but the logic system 1110 may be configured to have an interface for communicating with other components. Other components may or may not be configured to communicate with each other as appropriate.

[0138] The logic system 1110 may be configured to perform audio processing functions, including but not limited to functions of the type described herein. In some such embodiments, the logic system 1110 may be configured to operate (at least in part) according to software stored on one or more non-transitory media. The non-transitory media may include memory associated with the logic system 1110, such as random access memory (RAM) and / or read-only memory (ROM). The non-transitory media may include the memory of the memory system 1115. The memory system 1115 may include one or more suitable types of non-transitory storage media, such as flash memory, a hard drive, and the like.

[0139] The display system 1130 may include one or more suitable types of displays, depending on the form factor of the audio processing system 1100. For example, the display system 1130 may include a liquid crystal display, a plasma display, a bi-stable display, or the like.

[0140] The user input system 1135 may include one or more devices configured to accept input from a user. In some embodiments, the user input system 1135 may include a touch screen covering a display of the display system 1130. The user input system 1135 may include a mouse, a trackball, a gesture detection system, a joystick, one or more GUIs and / or menus presented on the display system 1130, buttons, keyboards, switches, and the like. In some embodiments, the user input system 1135 may include a microphone 1125: the user may provide voice commands to the audio processing system 1100 via the microphone 1125. The logic system may be configured for voice recognition and for controlling at least some operations of the audio processing system 1100 according to such voice commands. In some embodiments, the user input system 1135 may be considered a user interface and therefore as part of the interface system 1105.

[0141] The power system 1140 may include one or more suitable energy storage devices, such as nickel-cadmium batteries or lithium-ion batteries. The power system 1140 may be configured to receive power from an electrical outlet.

[0142] Various modifications to the implementations described in the present disclosure will be readily apparent to those of ordinary skill in the art. The general principles defined herein may be applied to other embodiments without departing from the spirit or scope of the present disclosure. Therefore, rather than limiting the claims to the embodiments shown herein, the claims are to be interpreted as conforming to the broadest scope consistent with the present disclosure, principles, and novel features disclosed herein.

Claims

1. An audio processing method, comprising: receiving audio data comprising at least one low frequency (LFE) channel, at least one audio object, and audio object metadata, wherein the audio object metadata is associated with the at least one audio object, wherein the audio object metadata comprises a size of the at least one audio object and an indicia indicating whether the at least one audio object is spatially diffuse; performing decorrelation filtering on the at least one audio object to determine decorrelated audio object audio signals based on determining that the at least one audio object is spatially diffuse in the playback environment, which indicates that the at least one audio object has a perceived size greater than a threshold, wherein each of the decorrelated audio object audio signals corresponds to at least one reproduction speaker of a plurality of reproduction speakers; Outputting the decorrelated audio object audio signal; as well as Output LFE channel. 2 . The method of claim 1 , wherein the at least one audio object is associated with at least one object position, wherein at least one of the at least one object position is fixed. 3 . The method of claim 1 , wherein the at least one audio object is associated with at least one object position, wherein at least one of the at least one object position changes over time.

4. The method of claim 1, further comprising applying a level adjustment process to the decorrelated audio object audio signals. The method of claim 1 , wherein performing decorrelation filtering comprises at least one of delaying and filtering.

6. The method of claim 1, wherein performing decorrelation filtering comprises at least one of all-pass filtering and pseudo-random filtering. The method of claim 1 , wherein performing decorrelation filtering comprises a reverberation process.

8. A computer program product comprising a physical, non-transitory computer readable medium storing instructions for executing the method according to any one of claims 1 to 7.

9. An audio processing device, comprising: a receiver configured to receive audio data comprising at least one low frequency (LFE) channel, at least one audio object, and audio object metadata, wherein the audio object metadata is associated with the at least one audio object, and wherein the audio object metadata comprises a size of the at least one audio object and a flag indicating whether the at least one audio object is spatially diffuse; a decorrelator configured to, based on determining that the at least one audio object is spatially diffuse in a playback environment, indicating that the at least one audio object has a perceived size greater than a threshold, perform decorrelation filtering on the at least one audio object to determine decorrelated audio object audio signals, wherein each of the decorrelated audio object audio signals corresponds to at least one reproduction speaker of a plurality of reproduction speakers; A first output unit, configured to output the decorrelated audio object audio signal; as well as The second output unit is used to output the LFE channel.

10. A non-transitory storage medium having stored thereon software comprising instructions which, when executed by at least one processor, cause the method of any one of claims 1 to 7 to be performed.

11. A device comprising: processor, and A non-transitory storage medium having software containing instructions stored thereon, wherein when the instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

12. An apparatus comprising means for performing the method of any one of claims 1 to 7.

13. A computer program product having instructions which, when executed by a computing device or system, cause the computing device or system to perform the method of any one of claims 1 to 7.

14. A method for processing audio data, comprising: receiving audio data comprising an audio object, the audio object comprising an audio object signal and associated metadata, the metadata comprising at least audio object size data, and the audio data further comprising one or more audio bed signals corresponding to speaker positions; Based on the audio object size data, determining a large audio object whose audio object size is greater than a threshold size; performing a decorrelation process on the audio object signal of the large audio object to generate a decorrelated large audio object audio signal; Associating the decorrelated large audio object audio signal with the object position, the associating process being independent of the actual playback speaker configuration and comprising mixing any of the following with the decorrelated large audio object audio signal: at least some of the audio bed signals, at least some of the audio object signals, or at least some of both audio bed signals and audio object signals; as well as The audio data output from the association process is encoded, wherein the encoding process includes a data compression process and does not involve encoding decorrelation metadata for the large audio object.

15. A system capable of clustering objects and / or beds in an adaptive audio processing system, comprising: an object processing component that receives an input audio file and metadata, wherein the input audio file includes an input object and associated object metadata, and also includes a bed and associated bed metadata, and performs object processing based on the input audio file and metadata, wherein The object processing includes at least one of media intelligence / content classification, spatial distortion analysis and object selection / clustering, metadata generation, and A renderer capable of receiving and combining a downmix object and a bed output from the object processing component, outputting a bed object and associated metadata.

Images