Separation and rendering of height objects

Abstract

The present disclosure relates to a method and system for processing audio, and a computer program product comprising instructions that, when the program is executed by a computer, cause the computer to execute the method. The method includes obtaining an input audio signal and processing the input audio signal to extract a height audio object from the input audio signal, where the height audio object is extracted using a source separation module configured to extract an audio object of a predetermined height audio source type. The method further includes rendering the input audio signal to a multi-channel presentation such that the at least one altitude audio object is included in at least one altitude channel of the multi-channel presentation.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application Serial No. 63 / 509,232 filed on June 20, 2023, U.S. Provisional Application Serial No. 63 / 495,515 filed on April 11, 2023, and International Application No. PCT / CN2022 / 101568 filed on June 27, 2022, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The invention relates to a method for processing an audio signal. Background Art

[0004] Today, audio content is captured using a wide variety of devices ranging from smart devices (such as smartphones, smart watches, and tablet computers) to professional recording devices with expensive and high-quality microphone elements. The vast majority of all audio content captured today is so-called user-generated content (UGC), and most UGC is captured using limited-performance microphones and distributed (e.g., uploaded to streaming services) with minimal or no post-processing, and because the process of capturing and distributing UGC is so simple, it has quickly become widely adopted.

[0005] For professionally generated content (PGC), the source audio content typically includes many individual channels (e.g., one or more channels for dialogue, one or more channels for music, and one or more channels for effects), and mixing engineers perform complex audio processing to form a well-balanced audio presentation suitable for rendering to, for example, a 5.1 surround sound system or binaural headphones.

[0006] UGC usually does not include or only includes a single captured channel, and is usually not subjected to complex manual post-processing by a mixing engineer. For example, in UGC, a target audio source (e.g., speech or music) may be recorded with an integrated microphone in a smartphone, which is held at a certain distance from the target audio source, wherein the microphone also captures background audio and noise, which degrade the capture quality of the target audio source. In addition, in many cases, the captured microphone signal is a mono audio signal, which means that the characteristics of the signal are very limited or have no spatial properties.

[0007] To this end, various automatic post-processing techniques for enhancing UGC have been proposed to, for example, improve intelligibility and reduce noise. Examples of such automatic post-processing include using a speech separator to extract speech content from the UGC signal and suppressing background audio and noise to make the speech more intelligible. Further examples of automatic post-processing that can be used to enhance UGC include EQ processing, volume adjustment, and reverberation processing. Summary of the invention

[0008] The disadvantage of existing solutions for processing audio content (especially UGC) is that, although, for example, speech separation, noise suppression, EQ processing, equalization, reverberation processing, etc. can enhance the perceived intelligibility and quality of sound during playback, the resulting audio content lacks spatial acoustic properties (e.g., temporal and spectral cues for sound source localization) and produces a bland or non-immersive impression when reproduced in a spatial format. As an example, a mono audio signal recorded by a capture device does not carry spatial information indicating the position of an audio source relative to the capture device used to capture the audio source. As a further example, a stereo audio signal captured by a pair of microphones of a smart device or a binaural capture device may carry spatial information indicating the position of an audio source in a horizontal plane, but the stereo audio signal still does not carry information about the height (elevation) of the audio source. For this reason, when rendering UGC into an immersive multi-channel presentation format (e.g., 2.0.2, 5.1.2, 5.1.4, 7.1.2, or 7.1.4 format), the resulting presentation lacks the spatial acoustic properties present in the environment in which the UGC was recorded.

[0009] An object of the present disclosure is to provide a method and system for processing and rendering audio content (especially UGC content), which can restore or enhance spatial properties when these audio contents are reproduced in a multi-channel format.

[0010] According to a first aspect of the present invention, a method for processing audio is provided. The method comprises obtaining an input audio signal and processing the input audio signal to extract a height audio object from the input audio signal. The height audio object is extracted using a source separation module, the source separation module being configured to extract audio objects of a predetermined height audio source type. The method further comprises rendering the input audio signal to a multi-channel presentation such that the at least one height audio object is at least included in at least one height channel of the multi-channel presentation.

[0011] The at least one height audio object is at least included in at least one height channel, which means that the height audio object is rendered to at least one height channel of the multi-channel presentation, and the height audio object is optionally also rendered to one or more non-height channels. In some embodiments, in addition to the at least one height channel, the at least one height audio object is also rendered to at least one non-height channel. Alternatively, in some embodiments, the at least one height audio object is only rendered to one or more height channels of the multi-channel presentation.

[0012] That is, even for a simple input audio signal (e.g., a mono audio signal, a stereo audio signal, or a binaural audio signal), at least one height audio object of a height audio source type is extracted and rendered to at least one height channel of a multi-channel presentation to form a presentation with enhanced spatiality. UGC content in the form of a mono audio signal captured with a single microphone or a stereo or binaural audio signal captured with a pair of horizontally separated microphones (e.g., a pair of wireless earbuds) does not carry information about the height of the captured audio source. However, through the above method, a predetermined type of height audio object present in such an audio signal can be automatically moved (translated) to at least one height channel of a multi-channel presentation. In this way, the audio objects are distributed between the height channels and the horizontal channels, which can enhance the listener's sense of immersion.

[0013] Another benefit of the method according to the first aspect of the present invention is that the method can be computationally lightweight, making it very suitable for implementation on computationally constrained devices. Some recording devices rely entirely on multiple microphones to estimate the height of an audio object by using a time-based or intensity-based analysis between corresponding microphone signals, however, this height estimation process can be computationally intensive and can be challenging to implement on portable and power-constrained devices (such as smartphones or wireless headsets). In addition, the reliability of such height estimation techniques is sensitive to the orientation of the recording device when recording content, and for some orientations (e.g., when the recording microphones are roughly aligned on a horizontal plane), height estimation will typically become unreliable or even impossible. However, the method according to the first aspect of the present invention can be very efficient and can be said to be input-independent, which means that height audio objects can be efficiently extracted regardless of the orientation of the recording device, and even height audio objects can be applied to monophonic audio signals. The method according to the first aspect of the present invention can be used instead of or in addition to time-based or intensity-based analysis.

[0014] The height audio source type refers to audio content associated with a source that is typically or primarily located above the user recording the audio signal. In other words, any audio content that the listener associates with an incident direction from above the listener is a candidate for the height audio source type. As an illustrative example, most audio content is recorded at ground level (e.g., on the street), which means that any sound characteristics of flying objects (birds, airplanes, drones) are candidates for the height audio source type.

[0015] The extracted audio object can be represented by an audio object signal comprising an isolated sound of a height audio source type. In certain embodiments, each source separation module is configured to extract the audio content associated with the audio object type for each time period of the input audio signal. If there is no height audio source type in one or more time periods of the input audio signal, the audio object will be substantially silent (that is, not comprising audio content) in these time periods, and if there is a height audio source type in the input audio signal, the sound of the audio source type will be included in the audio object signal. Alternatively, it is envisioned that the source separation module is configured to generate an audio object signal only when there is audio associated with a height audio source type (for example, exceeding a predetermined energy or level threshold).

[0016] In some embodiments, the height audio source type includes at least one of the following: man-made sounds related to height (e.g., the sound of blades rotating in the air, the vibration sound caused by a mechanical propulsion system, and the sound of explosion and combustion), sounds made by living things related to height (e.g., sounds associated with animals using aerial locomotion and sounds associated with arboreal animals), and natural sounds related to height (e.g., sounds of weather phenomena). By extracting audio objects associated with any of these types of height audio sources, a credible and convincing distribution of height audio objects in height can be achieved, thereby further enhancing the sense of immersion.

[0017] Artificial sounds related to height can also include music or songs. For example, users recording an orchestra or opera are often in the audience, who are located lower than the stage where the orchestra or singers are performing, which means that the music is perceived as coming from above.

[0018] In some embodiments, the method further includes: processing the input audio signal so as to also extract non-height audio objects from the input audio signal. The non-height audio objects can be extracted using a source separation module, the source separation module being configured to extract audio objects of a predetermined non-height audio source type, wherein rendering the input audio signal further includes rendering the non-height audio objects to at least one non-height channel of the multi-channel presentation. Accordingly, the non-height audio objects will be rendered at least to one or more horizontally distributed channels in the multi-channel presentation. Therefore, height audio objects and non-height audio objects are extracted separately and rendered in different ways to obtain a rendered multi-channel presentation with a credible distribution of audio objects in height, thereby enhancing immersion.

[0019] Examples of non-altitude audio source types include speech, sounds produced by man-made objects that are not related to altitude (e.g., the sound of a diesel or gasoline engine or the sound of tires rolling on the ground), sounds produced by living things that are not related to altitude (e.g., sounds associated with surface creatures such as cats, dogs, or cows), or natural sounds that are not related to altitude (e.g., the sound of waves lapping on the shore).

[0020] Non-height audio object types can be implicitly extracted. For example, any audio content not included in one or more height audio object types can be defined as a non-height audio object.

[0021] According to a second aspect of the present invention, there is provided a computer program product, the computer program product comprising instructions which, when the program is executed by a computer, cause the computer to perform the method according to the first aspect.

[0022] According to a third aspect of the present invention, there is provided a system, comprising one or more processors, wherein the one or more processors are configured to execute the method according to the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Aspects of the invention will be described in more detail with reference to the accompanying drawings, which show exemplary embodiments.

[0024] Figure 1 is a block diagram illustrating an audio processing system according to some embodiments.

[0025] Figure 2 is a flow chart describing a method for processing audio according to some embodiments.

[0026] Figure 3 is a block diagram illustrating an audio processing system with two source separation blocks according to some embodiments.

[0027] Figure 4A source separation block is shown in accordance with some embodiments.

[0028] Figure 5 A neural network based source separation block is shown in accordance with some embodiments.

[0029] Figure 6 A filter-based source separation block is shown in accordance with some embodiments.

[0030] Figure 7 Schematically depicted a 2.0.2 speaker layout for a media system. DETAILED DESCRIPTION

[0031] The systems and methods disclosed in this application can be implemented as software, firmware, hardware or a combination thereof. In hardware implementations, the task division does not necessarily correspond to the division of physical units; on the contrary, a physical component can have multiple functions, and a task can be performed by several physical components in collaboration.

[0032] The computer hardware may be, for example, a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a smart phone, an AR / VR wearable automotive infotainment system, a web device, a network router, a switch or a bridge, or any machine capable of (sequentially or otherwise) executing instructions specifying actions to be taken by the computer hardware. Further, the present disclosure will relate to any collection of computer hardware that executes instructions, alone or in combination, to perform any one or more of the concepts discussed herein.

[0033] Some or all of the components may be implemented by one or more processors that accept a computer-readable (also referred to as machine-readable) code containing a set of instructions that, when executed by one or more processors, perform at least one method described herein. Any processor that is capable of executing (sequentially or otherwise) a set of instructions that specifies an action to be taken is included. Thus, an example is a typical processing system (e.g., computer hardware) that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem that includes a hard drive, an SSD, a RAM, and / or a ROM. A bus subsystem may be included for communication between components. The software may reside in the memory subsystem and / or the processor during its execution by the computer system.

[0034] The one or more processors may operate as standalone devices or may be connected to other processor(s), such as via a network. Such a network may be constructed on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0035] The software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, various forms of physical (non-transitory) storage media, such as EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage devices, cassettes, tapes, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Further, it is well known to those skilled in the art that (transitory) communication media typically embodies computer-readable instructions, data structures, program modules or other data in the form of modulated data signals or other transmission mechanisms such as carrier waves, and includes any information transfer medium.

[0036] Figure 1 is a block diagram schematically illustrating an audio processing system 100 according to some embodiments. Figure 2 , the audio processing system 100 will now be described in detail. At step S1, an input audio signal is obtained by a source separation block 1 including at least one source separation module (e.g., 10a, 10b). The input audio signal may include one or more channels. Typically, the input audio signal includes one or more horizontal channels (e.g., two or more horizontal channels or three or more horizontal channels), but does not include a height channel. For example, the input audio signal is a mono audio signal (with one channel), a stereo or binaural audio signal (with two channels), or a surround sound audio signal (with more than two channels) (such as a 5.1 signal with six channels including an LFE channel or a 7.1 signal with eight channels including an LFE channel).

[0037] For example, a mono input audio signal or a stereo input signal may be captured using a recording device (eg, a smartphone or a tablet computer) having a mono or stereo microphone configuration.

[0038] The binaural audio signals may be acquired using a virtual head microphone or, as is more common for UGC, using headphones (eg, wireless headphones or earbuds) equipped with separate microphones for recording the binaural audio signals.

[0039] Regardless of the number of channels in the input audio signal, the input audio signal will in most cases contain audio content that is a mixture of many types of audio sources. As an example, if the audio signal is recorded during an outdoor interview, it may include a mixture of the interviewer's voice, the interviewee's voice, voices from other people nearby, traffic noise, birdsong, the sound of airplanes flying overhead, and different types of background noise (such as static white noise).

[0040] The source separation block 1 is configured to process the input audio signal at step S2 to extract at least one height audio object of a predetermined height audio source type from the input audio signal. More specifically, the source separation block 1 includes at least one source separation module (e.g., 10a, 10b), wherein each source separation module (e.g., 10a, 10b) is configured to extract a corresponding height audio object of a corresponding predetermined height audio source type. Each source separation module (e.g., 10a, 10b) outputs an object audio signal carrying audio content associated with the predetermined height audio source type.

[0041] In some embodiments, each source separation module (e.g., 10a, 10b) is configured to extract a height audio object of a predetermined height audio source type, wherein the height audio source type covers or includes one or more of the following: man-made sounds related to height, sounds emitted by living things related to height, or natural sounds related to height. Exemplary subcategories under man-made sounds related to height are the sound of blades rotating in the air (e.g., the sound of a helicopter, drone, propeller engine, or jet engine), the sound caused by man-made objects flying in the air (the sound of an airplane or balloon moving in the air), and the sound of combustion (e.g., the sound of fireworks exploding). Exemplary subcategories under living things related to height are sounds associated with animals that use aerial locomotion (e.g., the calls of bats, birds, or insects or the sounds of bats, birds, or insects moving in the air) and sounds associated with arboreal animals (e.g., the calls of sloths and / or monkeys / primates and the sounds of sloths and / or monkeys / primates moving in trees). Exemplary subcategories under altitude-related natural sounds are weather sounds (eg, thunder, rain, wind, and hail) and altitude-related landscape feature sounds (waterfalls and leaves rustling).

[0042] As an example, the source separation module 10a is configured to extract a height audio object type that contains (if present in the input audio signal) artificial sounds related to height, and the source separation module 10b is configured to extract height audio objects of a more specific height audio source type, such as only extracting the sound of blades rotating in the air and / or the sound of insects.

[0043] In some embodiments, there are more than one source separation modules 10a, 10b in the source separation block 1, wherein each source separation module 10a, 10b extracts the height audio objects of different corresponding height audio source types. For example, one source separation module 10a extracts the height audio objects including any artificial sounds related to height, and another source separation module 10b extracts the height audio objects including birdsong. Generally, it may be challenging to design a single source separation module 10a, 10b that reliably extracts various different height audio object types with different acoustic properties. For example, as described below, each source separation module 10a, 10b may include a filter for extracting different types of height audio objects, and designing a single filter that covers all conceivable height audio object types (artificial and natural sounds related to height and sounds associated with living things) without covering one or more non-height audio object types (such as speech or traffic sounds), which may be difficult. Thus, using multiple source separation modules 10a, 10b, where each source separation module extracts a single specific type of height audio object (e.g., bird song) or a group of specific types of height audio objects (e.g., all man-made sounds related to height) having similar frequency characteristics, can facilitate more reliable and accurate extraction of various types of height audio objects with little or no erroneous extraction of non-height object types.

[0044] The source separation modules 10a, 10b extract the corresponding height audio object types and represent each type of extracted height audio object with an audio object signal including sounds of the predetermined type of audio object. The audio object signals (each carrying the corresponding type of audio object) are optionally provided to a height object processor 3, which in an optional step mixes the audio object signal S3 into a mixed height signal and / or applies a predetermined gain or an adaptive gain to each audio object signal. Therefore, the mixed height signal may include a mixture of at least two different types of audio objects.

[0045] In some embodiments, where only a single type of height audio object is extracted, the mixing at S3 can be skipped. Alternatively, the height object processor 3 only adjusts the gain of the single type of height audio object (e.g., based on the identified scene type) without performing any mixing.

[0046] Typically, the source separation block 1 is configured to separate the input audio signal into N types of height audio objects, where N is equal to or greater than one. The source separation block 1 can be implemented in different versions, for example, it can rely on filtering to perform separation and / or rely on a neural network to perform separation, which will be discussed below in conjunction with Figure 5 and Figure 6 Give a description.

[0047] The mixed height signal is provided to a multi-channel renderer 6, which renders the mixed height signal to a multi-channel presentation at step S6, so that the height audio object type of the mixed height signal is rendered to at least one height channel of the multi-channel presentation. The rendering performed by the multi-channel renderer 6 involves rendering the mixed height signal to one or more height channels of a multi-channel format. For example, if the multi-channel format is a 2.0.2 format, a 5.1.2 format, or a 7.1.2 format, the mixed height signal (comprising a mixture of two or more types of height audio objects) may be rendered to a 0.0.2 height channel.

[0048] In some embodiments, the height audio object signals extracted by the source separation modules 10a, 10b are provided directly to a renderer 6 which renders the height audio object signals to one or more height channels of a multi-channel presentation.

[0049] The renderer 6 may be configured to assign a predetermined spatial position to each height audio object signal. For example, an object-based spatial renderer renders the spatial audio object to a presentation channel based on the position of the spatial audio object. As an example, each height audio object signal may be rendered as a zenith elevation or a 45° elevation so as to be perceived as coming from above the listener. In some embodiments, the height audio object signal includes two channels, whereby the renderer 6 is configured to render the two channels of the height audio object signal as two height channels having a predetermined elevation angle and channel separation angle.

[0050] Optionally, the mixed height signal or (multiple) audio object signals are provided to a crosstalk cancellation module 5, which performs crosstalk cancellation processing on the mixed height signal or (multiple) audio object signals at step S5 to reduce or remove crosstalk between at least two height channels in the rendered presentation. Crosstalk cancellation is particularly useful for binaural input audio signals, as this can ensure that some binaural properties are maintained when the audio objects are presented to the user. Figure 7 The crosstalk cancellation process is described in more detail.

[0051] Thus, even if the input audio signal itself does not include separate height audio channels (e.g., the input audio signal is a mono audio signal), one or more types of height audio objects have been extracted from the signal and used to create a multi-channel presentation, wherein the height channels carry the extracted height audio objects to improve the spatial immersion of a user consuming the content.

[0052] In some embodiments, the source separation block 1 is further configured to extract one or more types of non-height audio objects. For example, the source separation module 10c of the source separation block 1 can be configured to extract non-height audio objects of predetermined non-height audio source types. Non-height audio source types can be, for example, voices, sounds emitted by living things related to non-height (the sound of a cat or a dog), or artificial sounds related to non-height (for example, traffic noise, background voice). (Multiple) non-height audio objects are represented by corresponding (multiple) non-height audio object signals, and the (multiple) non-height audio object signals are optionally provided to a non-height object processor 8, which mixes (multiple) non-height audio object signals and / or applies a predetermined gain or adaptive gain. Similar to the discussion of multiple source separation modules 10a, 10b, the source separation block 1 can be equipped with two or more source separation modules, each of which is configured to extract audio objects of corresponding predetermined non-height audio source types.

[0053] While extracting non-height audio object types separately has some benefits (such as allowing the relative reproduction levels of non-height audio objects to be adjusted based on, for example, the detected acoustic scene type), this feature is optional. Therefore, in some embodiments, only one or more height audio object types are extracted, and the (multiple) non-height audio object types are implicitly determined as any residual audio content that is present in the input audio signal but not in any of the extracted height audio object types. The residual audio content can be provided directly to the multi-channel renderer 6 for rendering to the non-height channels in the multi-channel presentation.

[0054] Optionally, obtain auxiliary height audio channel at S31 place, and also provide to height object processor 3, described height object processor mixes it with mixed height audio signal.Auxiliary height audio channel can be extracted from source audio signal comprising non-height channel and height channel.Non-height channel is used as input audio signal (from which one or more height audio objects can be extracted), and height audio channel is used as auxiliary height audio channel.For example, in some cases, there may be preliminary height audio channel, and this channel has included one or more types of height audio objects.However, some height audio object types may still remain in non-height channel, and according to present method, these types can be extracted and mixed with auxiliary height audio channel.For this reason, audio processing system 10 can not only be used for extracting height audio object type when no height channel is available, but also can be used for further extracting height audio object type outside some preliminary height audio object types existing in the height channel of source audio signal.

[0055] In some embodiments, the source separation block 1, the height object processor 3 and / or the non-height object mixer 8 are further configured to obtain non-audio context data associated with the input audio signal and / or associated with a video or image captured simultaneously with the input audio signal. The non-audio context data indicates the context of the input audio signal. For example, the non-audio context data includes the results of a semantic or keyword analysis of the input audio signal, geographic location data associated with the recording location of the input audio signal, information about an identified object in an image or video captured simultaneously with the input audio signal (e.g., using the same user device), or capture attributes associated with a video or image captured simultaneously.

[0056] The source separation block 1 may use the non-audio context data to select which source separation modules 10a to 10c should be used in the source separation block 1. Alternatively, the height object processor 3 and / or the non-height object mixer 8 may use the non-audio context data to adjust the gain of some height / non-height audio objects (e.g., complete silence).

[0057] like Figure 1As shown, non-audio context data is provided to a source separation block 1, which selects at least one source separation module 10a from a group of source separation modules based on the non-audio context data. The group of source separation modules includes a plurality of separation modules, each of which is configured to extract a height audio object of a predetermined height audio source type, and each separation module is associated with a corresponding type of non-audio context data. For example, if the non-audio context data includes location data indicating that the input audio signal is recorded indoors, or includes data indicating that a video or image captured simultaneously with the input audio signal includes an object associated with an indoor environment (e.g., a TV, a kitchen, or a table), the source separation block 1 can select a source separation module 10a associated with an indoor non-audio context data type, and optionally avoid selecting a source separation module 10b associated with an outdoor non-audio context data type.

[0058] In this way, according to some embodiments, the audio processing system 100 can always avoid using all source separation modules 10a to 10c and only use selected source separation modules 10a to 10c, which extract potentially valid high-frequency audio source types based on non-audio context data.

[0059] It should be understood that "selecting" a source separation module includes activating / deactivating one or more source separation modules 10a to 10c, which actually constitutes a selection of which modules to use (activate). Deactivation can be implemented in the source separation block, where only the processing of the deactivated source separation modules 10a to 10c is omitted. However, deactivation can also be performed in the height object processor 3 and / or the non-height object mixer 8, which mutes the associated extracted height audio objects to be deactivated.

[0060] Additionally or alternatively, the selection of the above-mentioned source separation modules 10a to 10c may be based on the identified acoustic scene, which in turn may be based on non-audio context data.

[0061] exist Figure 2 In the embodiment of the present invention, the auxiliary height channels are mixed with a mixed height signal, and the mixed height signal includes a mixture of the extracted audio object signals. However, it should be understood that optional steps S3, S31 and S4 can be performed in a different order. For example, it is envisaged that in a single step, the auxiliary height channels are mixed with the extracted (multiple) audio object signals to form a mixed height signal including a mixture of the audio object signals and the auxiliary height audio channels.

[0062] Auxiliary height audio channels may be acquired using a recording device having two or more microphones capable of vertical source separation.

[0063] For example, smartphones typically have two or more microphones, and when the smartphone is in portrait mode, typically at least one microphone is pointed upwards and another microphone is pointed downwards. With this microphone arrangement, elevation-related information can be extracted from the audio signals recorded by these microphones (using, for example, beam steering), and the elevation-related information can be used to extract auxiliary height audio channels from the two microphone signals.

[0064] For example, in a frequency range where the upward-facing and downward-facing microphones have omnidirectional microphone patterns, the microphone pair can form a fixed or adaptive beamformer with which sound events from non-horizontal elevation angles can be extracted and used as auxiliary height audio channels.

[0065] As another example, in frequency ranges where the capture device includes a microphone with a polar microphone pattern, the microphone may be used to directly extract auxiliary height audio channels.

[0066] In addition, in some embodiments, auxiliary height audio channels, whether height channels of a source audio signal or height channels extracted from a recording device having multiple microphones, are processed with a height object classifier that separates the auxiliary height audio channels into height audio object types and non-height audio object types. Wherein, the height audio object types are used in the auxiliary height audio channels, and the non-height audio object types are discarded or combined with the non-height audio object types extracted from the input audio signal. That is, even when the auxiliary height audio channels exist and include actual height audio objects, the audio processing system 100 can reposition these preliminary height audio objects to non-height channels in a multi-channel presentation.

[0067] In some embodiments, the auxiliary height audio channel is not a binaural audio signal (e.g., the auxiliary height audio channel is mono), while the input audio signal and the different types of height audio objects extracted therefrom are binaural. In such an embodiment, the auxiliary height audio channel can be binauralized using the HRTF processor 4 before being mixed with the extracted height audio object type in the height object processor 3. The HRTF processor 4 processes the auxiliary height audio channel using HRTFs known in the art. For example, if the auxiliary height audio channel is mono, the HRTF processor 4 can process the mono with the HRTF to generate two binaural channels, wherein the mono audio channel is treated as an audio object, placed at a predetermined elevation and azimuth angle.

[0068] exist Figure 3 In FIG. 1 , an audio processing system 110 with two source separation blocks 1 a , 1 b is shown. Figure 3 The audio processing system 110 and Figure 1The audio processing system 100 differs in that it comprises a speech-music separator 7, two source separation blocks 1a, 1b, one for speech-dominated content and one for music-dominated content, and it comprises a scene classifier for controlling the gain(s) applied by the height object processor 3.

[0069] The speech-music separator 7 obtains an input audio signal and separates the input audio signal into two separate signals, namely a speech signal including speech content and a music signal including music content. The speech-music separator 7 can be configured to divide the input signal so that all contents are retained and exist in the speech signal or the music signal in a limited overlapping or non-overlapping manner. For example, the speech-music separator 7 can be configured to include any music content in the music signal and include any other content in the speech signal (including some contents that are neither speech nor music, such as the sound of a helicopter). The speech-music separator 7 can include a trained neural network for performing speech-music separation.

[0070] The speech signal and the music signal are provided to the speech separation block 1a and the music separation block 1b, respectively. Each separation block 1a, 1b includes at least one source separation module, and the at least one source separation module is configured to extract the height audio object of the corresponding predetermined height audio source type that may be present in the speech signal and the music signal, respectively. For example, the speech separation block 1a includes at least one source separation module, and the at least one source separation module is configured to extract the height object belonging to at least one of the following types: artificial sounds related to height, sounds emitted by living things related to height, and natural sounds related to height. The music separation block 1b includes one or more source separation modules, and the one or more source separation modules are configured to extract the corresponding type of musical instrument (string instrument, woodwind instrument, brass instrument, percussion instrument) as a height audio object and / or a specific instrument (e.g., violin, trombone or trumpet) as a height audio object. Although music does not contain an audio object related to height itself (compared to, for example, the sound of an airplane, in most cases, the sound of an airplane will be perceived as a sound related to height), it is expected that, according to the present disclosure, by identifying any audio object in a music signal and treating it as a height audio object, the listener's sense of space and immersion can be enhanced. In some embodiments, as mentioned above, all music content may be treated as a type of height audio object providing enhanced immersion, in which case the music appears to the listener to be coming from a stage slightly above the listener.

[0071] Optionally, each source separation block 1a, 1b may also include one or more non-height source separation modules, which are configured to extract at least one non-height audio object type for rendering to a multi-channel presentation. For example, the speech source separation block 1a may include a source separation module, which is configured to extract speech as a type of non-height audio object.

[0072] By dividing the input audio signal into two parts based on content, it is easier to achieve accurate separation blocks 1a, 1b. Preferably, the input audio signal is divided into at least two content types, wherein the audio objects extracted from each content type contain similar harmonics or specific time-frequency patterns, as this can improve separation performance.

[0073] Any type of height or non-height audio object extracted by each source separation block 1a, 1b is provided to the height object processor 3 or the non-height object processor 8 and combined into a mixed height audio signal or a mixed non-height audio signal with a predetermined or adaptive gain for each type of audio object, respectively.

[0074] The audio processing system 110 also includes a scene classifier 2. The scene classifier 2 is configured to analyze the input audio signal and determine the acoustic scene type of the input audio signal. In some embodiments, the scene classifier 2 is configured to determine whether the input audio signal is an indoor scene type or an outdoor scene type. In addition, the scene classifier can be configured to determine whether the input audio signal is an additional acoustic scene type, such as a traffic (commuting) scene, a natural scene, or an urban scene.

[0075] The term "acoustic scene classification" means the identification of high-level semantic properties of the recorded audio content, which are indicative of the environment in which the content was recorded. For example, a speech recorded outdoors sounds different from the same speech recorded indoors due to, for example, different reverberation effects and echoes. The acoustic scene classifier 2 analyzes the audio content of the input audio signal and determines the acoustic scene of the audio content. The acoustic scene classifier 2 may, for example, be configured to analyze an energy spectrogram of the input audio signal, or to use a trained neural network to determine the acoustic scene of the input audio signal.

[0076] It is also contemplated that the scene classifier 2 may analyze one or more extracted height or non-height audio objects to determine the acoustic scene type. Although each extracted audio object type includes only a subset of the information available in the input audio signal, the scene classifier 2 may still accurately determine the acoustic scene type based on the one or more extracted audio object types. For example, the scene classifier 2 may be trained to classify a scene as an outdoor scene given an input of one or more extracted audio objects including the rustling of leaves, the sound of wind, or the sound of raindrops hitting the ground.

[0077] Additionally or alternatively, the scene classifier 2 obtains non-audio context data associated with the input audio signal, and determines the acoustic scene type based on the non-audio context data. The non-audio context data may indicate a media capture mode associated with the input audio signal, a result of a semantic or keyword analysis of the input audio signal, and geographic location data associated with the recording location of the input audio signal. For example, if the input audio signal includes speech, various keywords may be identified, wherein the keywords will help the scene classifier 2 make a more informed decision about the current acoustic scene. For example, if keywords associated with objects, spaces, or events that are usually indoors (e.g., "living room," "sofa," "kitchen," "stove," or) are detected, the acoustic scene is likely to be an indoor scene. Similarly, if keywords associated with objects, spaces, or events that are usually outdoors (e.g., "tree," "hiking," "bird," "lake," "ocean") are detected, the acoustic scene is likely to be an outdoor scene. It should be understood that any number of acoustic scenes may be detected, not just indoor scenes or outdoor scenes, wherein each scene is associated with one or more keywords.

[0078] Non-audio context data can be context data extracted from a video or image captured simultaneously with the input audio signal. The input audio signal can be, for example, a recorded audio track of a video file, which means that the input audio signal is included in the same media file or media stream as the video. For example, the non-audio context data can indicate one or more detected objects (e.g., birds, trees, people) in the video or image or capture attributes associated with the video or image (e.g., scene brightness, shutter speed, ISO level). The scene classifier 2 can use this information to classify the scene more accurately. For example, if one or more trees, the sky, or a lake are identified in the video or image, the acoustic scene is likely to be an outdoor scene. As another example, if the scene brightness is low, the shutter speed is long, or the ISO level is high, the video or image is likely to be captured in a dim indoor environment, which means that the acoustic scene is likely to be an indoor scene.

[0079] The above-mentioned exemplary information indicated by the non-audio context data is merely exemplary and may also include many other types of information. For example, some capture devices allow users to select a specific capture mode (e.g., indoor capture mode, food capture mode, landscape capture mode, starry sky capture mode, etc.) for a video or image captured simultaneously with an input audio signal. The non-audio context data indicating the capture mode of the associated image or video can be used to determine the acoustic scene. For example, if the capture mode is an indoor capture mode or a food capture mode, the acoustic scene is likely to be an indoor capture scene, and if the capture mode is a landscape capture mode or a starry sky capture mode, the acoustic scene is likely to be an outdoor scene.

[0080] Although certain examples of non-audio context data are presented in conjunction with scene classifier 2, it should be understood that the same examples of non-audio context data may be applied to other components in embodiments of the present invention that use non-audio context data (e.g., non-audio context data may be used to select an appropriate source separation block of a source separation module).

[0081] The scene classifier 2 notifies the determined acoustic scene to the height object processor 3 and / or the non-height object processor 8. The height object processor 3 and the non-height object processor 8 can be configured accordingly to adaptively control the gain applied to at least one height object signal based on the determined acoustic scene. In some embodiments, each height and / or non-height audio object type is associated with at least one acoustic scene type. Therefore, the first (height or non-height) audio object type is associated with the first acoustic scene type (e.g., birdsong is associated with an outdoor scene), and the second (height or non-height) height object type is associated with the second acoustic scene type (e.g., ceiling fan sound is associated with an indoor scene). The height object processor 3 and the non-height object processor 8 are configured to increase the mixing gain of the audio object signal (carrying the corresponding object type) associated with the current acoustic scene and / or reduce the (multiple) audio objects associated with different acoustic scenes (e.g., complete silence). For example, if the acoustic scene is an indoor scene, the gain of the bird song audio object type is reduced (or muted completely) because birds are not expected to sing indoors, and the gain of the ceiling fan audio object type is reduced because ceiling fans are likely to be used indoors.

[0082] Steering Figure 4 , shows a source separation block 1 according to some embodiments. The source separation block 1 includes a plurality of source separation modules 10a to 10d. The source separation module 10a is configured to extract height audio object types, while the source separation modules 10b to 10d are configured to extract corresponding non-height audio object types, sometimes also referred to as "residual" audio object types. The extracted height audio object types are optionally provided to a height object processor (see Figure 1 or Figure 3 ), and rendered to at least one height channel in a multi-channel presentation. Optionally, the non-height audio object type is also subjected to some mixing before being rendered to at least one non-height channel in a multi-channel presentation.

[0083] In an exemplary embodiment, source separation module 10a is configured to extract sounds made by living things related to height (e.g., birdsong), source separation module 10b is configured to extract speech, source separation module 10c is configured to extract sounds associated with living things without height (e.g., cats or dogs), and source separation module 10d is configured to extract background audio, i.e., any audio content not extracted by source separation modules 10a to 10d. The audio extracted by source separation module 10a is used as a height audio object type, and the remaining audio objects extracted by source separation modules 10b to 10d are used as corresponding non-height audio object types.

[0084] It should be understood that Figure 4 The source separation block shown is only exemplary, and many variations are possible. For example, more or less height object separation modules 10a can be used, and / or more or less non-height separation modules 10c to 10d can be used. In addition, it should be noted that the source separation module 10d for clearly determining that any one of the other source separation modules 10a to 10c does not cover the background audio can be omitted, because the background audio can be implicitly determined by other source separation modules 10a to 10c. For example, the basic operation of the input audio signal and the extracted audio object type (for example, subtracting the extracted audio object type from the input audio signal) can be used to determine the background audio content.

[0085] In the above, combined with Figure 1 It has been described that non-audio context data may be provided to the source separation block 1, wherein the source separation block 1 selects the source separation modules 10a to 10d based on the non-audio context data. Figure 4 As shown, it is also contemplated that non-audio context data can be provided directly to one or more source separation modules 10a to assist the separation processing of the source separation module 10a. It is contemplated that the processing of the source separation module 10a can be fine-tuned based on the non-audio context data, or that the source separation module 10a can operate in different modes based on the non-audio context data.

[0086] As an example, the source separation module 10a is configured to extract a height audio object of a height audio source type corresponding to sounds associated with the movement of blades in the air (e.g., rotors from a drone, helicopter, turbojet engine, or ceiling fan). If the non-audio context data indicates that the context of the input audio signal is an outdoor scene, the source separation module 10a can operate in a first mode, which is configured to extract sounds associated with the rotors of a drone, helicopter, or turbojet engine (sounds that can be heard primarily outdoors). On the other hand, if the non-audio context data indicates that the context of the input audio signal is an indoor scene, the source separation module 10a can operate in a second mode, which is configured to extract sounds specifically associated with a ceiling fan (sounds that can be heard primarily indoors).

[0087] In addition to or alternatively to the non-audio context data, the identified acoustic scene can be provided to one or more of the source separation modules 10a to 10d, wherein the source separation modules 10a to 10d are configured to perform extraction based on the acoustic scene. The acoustic scene determined by the scene classifier can in turn be based on the non-audio context data and / or attributes of the input audio signal.

[0088] exist Figure 5 , an example implementation of a neural network-based source separation block 1 is depicted. The source separation block 1 obtains an input audio signal and outputs N types of separated audio objects (height object types and / or non-height audio object types). The source separation block 1 includes a common feature extraction block 11, which is trained to extract a feature set based on the input audio signal. For example, the feature extraction block 11 extracts a feature set for each time period or time-frequency tile of the input audio signal. The feature extraction block 11 may include one or more neural network layers, such as one or more dense neural network layers, one or more convolutional neural network layers, one or more recurrent neural network layers, or a combination thereof. The features extracted by the feature extraction block 11 may be latent features.

[0089] The same feature set extracted by the feature extraction block 11 is provided to each of the N gain mask predictors 12a, 12b, 12c. Each of the gain mask predictors 12a, 12b, 12c is configured to generate a gain mask (soft mask) for isolating the corresponding (height or non-height) audio object. The gain mask includes a plurality of gain values, one for each time-frequency slice, which is configured to be applied to the input signal so as to remove almost all audio objects except the target audio object (e.g., birdsong or speech). Conversely, the gain mask may include a plurality of gain values, which are configured to be applied to the input signal to remove only the target audio object. In either case, each gain mask is configured to separate the target audio object from all other audio objects present in the input audio signal. Each gain mask may include a binary gain value (i.e., one or zero) to completely mute the time-frequency slice or to pass the time-frequency slice unprocessed. Each gain mask may also include a "softer" gain value, i.e., a gain value of a value range between zero and one may also be used.

[0090] The feature extraction block 11 and / or the gain mask predictors 12a, 12b, 12c may be provided with non-audio context data and / or information about the acoustic scene as auxiliary information, allowing these networks to be trained specifically for different types of contexts and scenes.

[0091] At least one of the gain mask predictors 12a, 12b, 12c is configured to predict a gain mask for extracting an audio object associated with a type of height audio source, such as man-made sounds related to height (e.g., sounds of blades rotating in the air, sounds caused by man-made objects moving in the air, and sounds of explosions and combustion), sounds made by living things related to height (sounds associated with the movement of animals using aerial locomotion and sounds associated with arboreal animals), or natural sounds related to height (e.g., weather sounds or sounds associated with landscape features). Each gain mask predictor 12a, 12b, 12c may include one or more neural network layers that are trained to predict a gain mask for isolating a corresponding audio object based on a common feature set.

[0092] For example, the gain mask predictor 12a is configured to predict a gain mask for extracting bird songs based on the same set of features extracted by the feature extraction block 11, and the gain mask predictor 12b is configured to predict a gain mask for extracting helicopter sounds based on the same set of features.

[0093] Each gain mask is provided to a respective mask applicator unit 13a, 13b, 13c which applies the gain mask to the input signal to form an extracted audio object. Continuing with the example above, the audio object extracted by the gain mask predictor 12a and mask applicator 13a would be an isolated bird song, and the audio object extracted by the gain mask predictor 12b and mask applicator 13b would be an isolated helicopter sound.

[0094] Therefore, the source separation block 1 extracts one or more different types of audio objects related to height, which can be rendered to the height channels of the multi-channel presentation to enhance the sense of space. It is worth noting that the input audio signal can be a mono signal, a binaural signal, a stereo signal or a multi-channel signal, which means that even for an input audio signal without any height information (for example, a mono signal captured with a smartphone), the audio object types related to height can be extracted. The extracted audio object types are then optionally provided to a non-height or height object processor, or directly to a multi-channel renderer, as described above in conjunction with Figure 1 discussed.

[0095] Note that the common feature extraction block 11, the gain mask predictor 12a and the mask applicator 13a together may form a first source separation module configured to extract a first audio object type. Similarly, the common feature extraction block 11, the gain mask predictor 12b and the mask applicator 13b may form a second source separation module configured to extract a second audio object type different from the first audio object type. Additional mask predictors 12c and mask applicators 13c may be added to extract N types of audio objects, where N is greater than or equal to one.

[0096] In order to train the feature extraction block 11 and the audio object separators 12a, 12b, 12c, a test audio signal can be provided, wherein the test audio signal includes a mixture of audio object types. The test audio signal is provided to the feature extraction block 11, which extracts features and provides the features to the audio object separators 12a, 12b, 12c, which predict gain masks for separating audio objects of corresponding audio object types. In some embodiments, the predicted gain mask is compared with a manually extracted basic fact gain mask to separate audio objects of corresponding types. Based on the difference between the basic fact gain mask and the predicted gain mask, the internal weights of the audio object separators 12a, 12b, 12c and / or the feature extraction block 11 can be updated to reduce the difference, and the audio object separators 12a, 12b, 12c and / or the feature extraction block 11 are trained to produce a gain mask that is closer to the basic fact gain mask.

[0097] It is also contemplated that the prediction gain mask may be applied to the test audio signal to obtain a predicted audio object signal that should capture the corresponding type of audio object. The internal weights of the audio object separator 12a, 12b, 12c and / or the feature extraction block 11 may be updated by comparing the audio object signal with a ground truth audio object signal comprising manually separated audio objects of the corresponding type.

[0098] It is also possible to use a separate feature extraction block 11 for each gain mask predictor 12a, 12b, 12c. The benefit of such an embodiment is that each pair of feature extraction block 11 and gain mask predictor 12a, 12b, 12c forms an independent extractor model, which can be trained separately and then combined with other independent models. A disadvantage of this technical solution is that the overall complexity of the model also grows linearly with the increase in the number of audio object types. For this reason, it is beneficial to use a common feature extraction 11 for at least two gain mask predictors 12a, 12b, 12c for reasons of computational complexity.

[0099] Further references Figure 3 It has been described that the audio processing system 110 according to some embodiments may include two or more source separation blocks 1a, 1b. Each source separation block 1a, 1b may include a separate common feature extraction block 11 before one or more gain mask predictors 12a, 12b, 12c, such as Figure 5 As shown. Advantageously, the audio object types to be extracted are grouped into the same source separation block 1a or 1b, so that spectrally similar audio object types are extracted by the same source separation block 1a or 1b. This facilitates the training of the common feature extraction block 11, and / or enables the use of a simpler feature extraction block 11 (e.g., with fewer neural network layers) while maintaining performance, because the common feature extraction block 11 can operate on spectrally similar audio content.

[0100] exist Figure 3 In the audio processing system 110 of FIG. 1 , the scene classifier 2 is used together with the speech-music separator 7. However, it should be understood that the scene classifier 2 can be used without the speech-music separator 7, and vice versa. For example, the scene classifier 2 can be used to adjust the level of the height audio object type extracted by the single source separation block (e.g., when the scene is identified as an indoor scene, the gain level of the bird song audio object type is reduced).

[0101] exist Figure 6, an alternative version of the source separation block 1 is depicted. Instead of utilizing a neural network-based feature extractor and an audio object separator, the source separation block 1 may include a plurality of filters 15a, 15b, 15c configured to isolate corresponding height / non-height audio object types. Each filter 15a, 15b, 15c may be a time domain and / or frequency domain filter or filter bank (e.g., a QMF filter bank). This version of the source separation block 1 is well suited for separating audio object types that do not overlap or only overlap to a limited extent in the frequency domain.

[0102] A method for determining the appropriate passband for each filter 15a, 15b, 15c includes first collecting sample audio content of the associated audio object (e.g., birdsong or helicopter sound). The more sample content that can be collected, the better, but typically a few hours of sample audio content is sufficient. Secondly, the sample audio content is analyzed to determine over which frequencies the sample audio content is distributed. Using information indicating the frequencies of the sample audio content distribution, a suitable filter for separating the audio objects can be designed. The resulting filter can be referred to as a soft mask over a set of frequency bands. Preferably, the soft mask does not have a sharp cutoff frequency, but is rather a smooth version of the EQ characteristics similar to the audio object type.

[0103] Optionally, an equalization characteristic of energy within a frequency band of the audio object is also determined.In some embodiments, filtering the input signal using the audio object specific filter further comprises performing equalization processing using the equalization characteristic of the target audio object.

[0104] The filter-based implementation of the source separation block 1 can be computationally very efficient and can be applied to both time-domain and frequency-domain representations of the input audio signal. However, it works best for audio objects whose frequencies are clearly distinguishable.

[0105] The filter based source separation block 1 may be provided with non-audio context data and / or information about the acoustic scene, allowing, for example, fine-tuning of the filters (in terms of passband, roll-off, center frequency, etc.) based on the context or scene.

[0106] refer to Figure 7 , a process for rendering at least one height audio object type and a non-height audio object type to a 2.0.2 presentation will now be described, however, it should be understood that a similar process may be performed to render audio objects to a different presentation format with height channels, such as a 5.1.2 or 7.1.2 presentation.

[0107] The 2.0.2 presentation format includes four channels, two horizontal channels referred to as 2.0.0 channels, and two height channels referred to as 0.0.2 channels. The 2.0.2 format is suitable, for example, for playback on a media system 200 having four speakers 201a to 201d and a display, wherein two speakers 201a, 201b are located above the center line of the display 202, and two speakers 201c, 201d are located below the center line of the display 202, such as Figure 7 Schematically shown. The media system 200 can be, for example, a tablet computer 210 placed horizontally (which is often the case when the tablet computer 210 is used to consume media content). However, other types of media systems are also possible, such as a laptop computer or even a TV, and one or more speakers having a corresponding spatial relationship with the display, just like the speakers 201a to 201d and the display 202.

[0108] Rendering at least one height audio object type and a non-height audio object type to the 2.0.2 format may include rendering the height audio object type only to two speakers 201a, 201b located above the screen 202 and rendering the non-height audio object type to all four speakers 201a to 201d. In this way, the height audio object type will be perceived by the user as originating from a position above the display 202 or above the user, while the non-height audio object type is rendered more uniformly in the horizontal plane and has more rendering energy. Therefore, the non-height audio object type will be perceived by the user as originating from the center of the display 202.

[0109] Optionally, a crosstalk cancellation processor (see Figure 1 and Figure 3 ) processes height audio object types, and the crosstalk cancellation processor is configured to process height audio object types so that when these height audio object types are rendered to the two top speakers 201a, 201b, the crosstalk between these speakers 201a, 201b and the user's ears is reduced or removed. Crosstalk means that the audio output by the top right speaker 201b reaches the left ear of the user located in front of the media system 200, and the audio output by the top left speaker 201a reaches the right ear of the user. Using crosstalk cancellation, the audio object types to be rendered are adjusted so that virtual channels are formed between the top left speaker and the left ear and between the top right speaker and the right ear, respectively. In fact, this allows the user to perceive an acoustic image that is very similar to listening to binaural headphones.

[0110] In some embodiments of crosstalk cancellation, the method involves estimating four impulse responses, one being the impulse response H between the left speaker 201a and the left ear. LL, one is the impulse response H between the right speaker 201b and the left ear RL , one is the impulse response H between the left speaker 201a and the right ear LR , one is the impulse response H between the right speaker 201b and the right ear RR Using these impulse responses H LL , H RL , H LR , H RR Its inverse, the crosstalk cancellation processor processes the audio signal to be rendered to the overhead speakers to achieve crosstalk cancellation.

[0111] When the input audio signal is a recorded binaural audio signal, crosstalk cancellation is particularly beneficial. For binaural input audio signals, there may already be some spatial properties included in the binaural audio signal, and the extracted height / non-height audio object type can also maintain these properties. For example, although the height of the audio object may not be distinguished from the binaural audio signal, the binaural audio signal includes some information indicating whether the audio object is located on the left or right side. For this reason, when extracting and rendering the binaural height audio object type, in addition to rendering the binaural audio object to the height channel multi-channel presentation, the left and right information can also be retained. Through crosstalk cancellation, even if the user is listening to the rendered presentation via a speaker rather than via headphones, the binaural properties of the audio object type can also be at least partially maintained.

[0112] In order to more accurately cancel crosstalk, it may be beneficial to determine the orientation of the user in front of the media system 200 in substantially real time (e.g., continuously update the estimate of the impulse response function). To accomplish this, a motion tracker may be used to monitor the movement of the user's head to adjust the crosstalk cancellation of the height channels rendered to be perceived by the user accordingly. As an example, video-based head tracking using, for example, a camera connected to or integrated in the media system 200 may be used to perform motion tracking and adjust the crosstalk cancellation accordingly.

[0113] Alternatively, crosstalk cancellation may be performed by assuming that the user is located in a predetermined position (sometimes referred to as an optimal position) in front of the media system 200, facing the display 202. While this assumption may not always be accurate (e.g., due to the user moving his or her head), it is contemplated that it may still provide a convincing binaural effect under typical viewing scenarios.

[0114] Unless otherwise specifically stated, it will be apparent from the following discussion that it should be understood that throughout the disclosed discussion, terms such as "processing," "computing," "calculating," "determining," "analyzing," etc. are utilized to refer to the actions and / or processes of computer hardware or computing systems or similar electronic computing devices to manipulate and / or transform data represented as physical (e.g., electronic) quantities into other data similarly represented as physical quantities.

[0115] It should be understood that in the above description of the exemplary embodiments of the present invention, various features are sometimes grouped together in a single embodiment, figure or its description to simplify the present disclosure and help understand one or more of the various creative aspects. However, the method of the present disclosure should not be interpreted as reflecting the intention that the claimed invention requires more features than the features explicitly stated in each claim. On the contrary, as reflected in the following claims, the creative aspects are less than all the features of the single aforementioned disclosed embodiment. Therefore, the claims following the specific embodiments are hereby expressly incorporated into the specific embodiments, wherein each claim is independently a separate embodiment of the present invention. In addition, although some embodiments described herein include some features included in other embodiments without including other features included in other embodiments, as will be understood by those skilled in the art, the combination of features of different embodiments is intended to be within the scope of the present invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0116] In addition, some embodiments are described herein as a combination of methods or method elements that can be implemented by a processor of a computer system or by other devices that perform functions. Therefore, a processor with an instruction for performing such a method or method element forms a device for performing a method or method element. It should be noted that when the method includes multiple elements (e.g., several steps), unless otherwise stated, any order of these elements is not implied. In addition, the elements of the device embodiments described herein are examples of devices that perform functions performed by elements in order to perform embodiments of the present invention. In the description provided herein, many specific details are set forth. However, it should be understood that embodiments of the present invention can be practiced without these specific details. In other examples, well-known methods, structures, and techniques are not shown in detail to avoid blurring the understanding of this specification.

[0117] Therefore, while particular embodiments of the present invention have been described, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the invention, and it is intended to claim all such changes and modifications that fall within the scope of the invention.

[0118] Various aspects of the present invention may be understood from the following enumerated example embodiments (EEE):

[0119] EEE 1. A method of processing audio, the method comprising: receiving an input audio signal; extracting an audio object associated with height information from the input audio signal; and rendering the audio object according to the height information using multi-speaker rendering.

[0120] EEE 2. The method as described in EEE 1, wherein the input audio signal is a binaural signal.

[0121] EEE 3. The method as described in any one of EEE 1 or 2, wherein extracting the audio object includes using audio scene classification to guide object extraction.

[0122] EEE 4. The method as described in any one of EEE 1 to 3, wherein the extracting of the audio object is performed based on a frequency band distribution characteristic.

[0123] EEE 5. The method as described in any one of EEE 1 to 4, wherein extracting the audio object is performed based on deep learning.

[0124] EEE 6. The method of EEE 1, wherein the input audio signal includes height channels provided by multi-device capture. ...

[0125] EEE 7. The method of any one of EEE 1 to 6, wherein rendering the audio object comprises performing crosstalk cancellation of speakers on the 0.0.2 channel without upmixing the binaural signal. ...

[0126] EEE 8. A system comprising one or more processors configured to perform the operations as described in any one of EEE 1 to 7.

[0127] EEE 9. A computer program product configured to cause one or more processors to perform the operations described in any one of EEE 1 to 7.

Claims

1. A method for processing audio, the method comprising: Obtaining an input audio signal; processing the input audio signal to extract at least one height audio object from the input audio signal, wherein the at least one height audio object is extracted using a source separation module configured to extract audio objects of a predetermined height audio source type; as well as The input audio signal is rendered to a multi-channel presentation such that the at least one height audio object is included in at least one height channel of the multi-channel presentation.

2. The method according to claim 1, wherein: The height audio source type includes at least one of the following: sounds emitted by man-made objects related to height, sounds emitted by living things related to height, and natural sounds related to height.

3. The method according to claim 2, wherein: The man-made sounds associated with altitude include at least one of the following: the sound of blades rotating in the air, the sound caused by man-made objects moving in the air, and the sound of combustion, and / or wherein, The altitude-related sounds emitted by living things include at least one of: sounds associated with animals using aerial locomotion and sounds associated with arboreal animals, and / or wherein the altitude-related natural sounds include at least one of sounds associated with weather and sounds associated with landscape features.

4. The method according to any one of the preceding claims, further comprising: processing the input audio signal to extract at least two height audio objects, wherein the at least two height audio objects are extracted using a respective one of at least two source separation modules, each source separation module being configured to extract a respective height audio object of a respective height audio source type; The input audio signal is rendered to the multi-channel presentation such that the at least two height audio objects are included in at least one height channel of the multi-channel presentation.

5. A method according to any one of the preceding claims, wherein: The input audio signal is a binaural audio signal.

6. The method according to claim 5, wherein: The at least one height audio object is a binaural audio signal, and wherein rendering the input audio signal to a multi-channel presentation comprises: The input audio signal is rendered such that the at least one height audio object is presented in at least two height channels of the multi-channel presentation.

7. A method according to any one of the preceding claims, wherein: Each source separation module is configured to process the input audio signal using a respective frequency filter, wherein a passband of each respective frequency filter corresponds to a characteristic frequency range of an associated height audio source type.

8. The method according to any one of claims 1 to 7, wherein: Each source separation module comprises a neural network trained to extract height audio objects of a corresponding predetermined height audio source type.

9. The method according to claim 8, wherein: Processing the input audio signal includes Processing the input audio signal using a feature extractor to obtain a feature representation of the input audio signal; as well as The feature representation of the input audio signal is processed with each source separation module, wherein each source separation module is trained to extract the predetermined height audio object based on the feature representation.

10. The method according to any one of the preceding claims, further comprising: The acoustic scene type of the input audio signal is identified by processing the input audio signal using a scene classifier.

11. The method according to any one of the preceding claims, further comprising: A scene classifier is used to identify the acoustic scene type of the input audio signal by processing non-audio context data, wherein the non-audio context data includes at least one of the following: data indicating a media capture mode, results of a semantic or keyword analysis of the input audio signal, geographic location data associated with the input audio signal, and data associated with an image or video captured simultaneously with the input audio signal.

12. The method according to claim 10 or claim 11, further comprising: A gain of the at least one height audio object in the multi-channel presentation is controlled based on the acoustic scene type.

13. The method according to any one of claims 10 to 12, wherein: The scene classifier is configured to identify whether the acoustic scene type of the input audio signal is an indoor scene or an outdoor scene.

14. The method according to any one of claims 10 to 13, further comprising: processing the input audio signal using at least two source separation modules to extract at least a first height audio object associated with a first acoustic scene type and a second height audio object associated with a second acoustic scene type different from the first acoustic scene type, wherein each of the at least two source separation modules is configured to extract a respective height audio object of a respective predetermined height audio source type; determining whether the first acoustic scene type matches the identified acoustic scene type; and Based on determining that the first acoustic scene type matches the identified acoustic scene type, the input audio signal is rendered to a multi-channel presentation so that only the first height audio object of the first height audio object and the second height audio object is included in the at least one height channel of the multi-channel presentation.

15. The method according to claim 14, further comprising: Based on determining that the first acoustic scene type does not match the identified acoustic scene type, rendering the input audio signal to a multi-channel presentation such that only the first height audio object and the second height audio object are included in the at least one height channel of the multi-channel presentation.

16. The method according to any one of the preceding claims, further comprising: processing the input audio signal using a content separator to extract audio signals associated with a first content type and a second content type, respectively; processing the audio signal associated with the first content type to extract a first height audio object, wherein the first height audio object is extracted using a first source separation module associated with the first content type; processing the audio signal associated with the second content type to extract a second height audio object, wherein the second height audio object is extracted using a second source separation module associated with the second content type; and The input audio signal is rendered to the multi-channel presentation such that the first height audio object and the second height audio object are included in at least one height channel of the multi-channel presentation.

17. The method according to claim 16, wherein: The content separator is configured to separate at least music content and voice content.

18. The method according to any one of the preceding claims, further comprising: processing the input audio signal to extract non-height audio objects from the input audio signal, wherein the non-height audio objects are extracted using a source separation module configured to extract audio objects of a predetermined non-height audio source type; and Wherein, rendering the input audio signal further comprises: The non-height audio object is rendered to at least one non-height channel of the multi-channel presentation.

19. The method according to claim 18, further comprising: The non-height audio objects are rendered to all channels of the multi-channel presentation.

20. The method according to any one of the preceding claims, further comprising: obtaining a multi-channel source audio signal, the source audio signal comprising at least one non-height channel and height channels, wherein the at least one non-height channel is used as the input audio signal; and The input audio signal and the height channels are rendered to a multi-channel presentation such that the at least one height audio object extracted from the input audio signal and the height channels of the source audio signal are included in the at least one height channel of the multi-channel presentation.

21. A method according to any one of the preceding claims, wherein: The multi-channel presentation includes at least two height channels, the method further comprising: The height audio object is processed using a crosstalk cancellation module to reduce or remove crosstalk of the height audio object between the at least two height channels when rendering the height audio object.

22. A method according to any one of the preceding claims, wherein: The multi-channel presentation includes at least two non-height channels, the method further comprising: At least one non-height audio object of the input audio signal is processed using a crosstalk cancellation module to reduce or remove crosstalk of the at least one non-height audio object between the at least two non-height channels when rendering the at least one non-height audio object.

23. The method according to any one of the preceding claims, further comprising: Obtaining non-audio context data, wherein the non-audio context data includes at least one of the following: data indicating a media capture mode, results of a semantic or keyword analysis of the input audio signal, geographic location data associated with the input audio signal, data associated with an image or video captured simultaneously with the input audio signal; and Wherein, the source separation module is configured to extract the audio object of the predetermined high-level audio source type based on the non-audio context data.

24. The method according to any one of the preceding claims, further comprising: Obtaining non-audio context data, wherein the non-audio context data includes at least one of the following: data indicating a media capture mode, results of a semantic or keyword analysis of the input audio signal, geographic location data associated with the input audio signal, and data associated with an image or video captured simultaneously with the input audio signal; The source separation module is selected from a group of source separation modules based on the non-audio context data, wherein the group of source separation modules includes a plurality of source separation modules, each source separation module is associated with a height audio object of a predetermined height audio source type, and each source separation module is associated with a type of non-audio context data.

25. A computer program product comprising instructions which, when said program is executed by a computer, cause said computer to perform the method according to any one of claims 1 to 24.

26. A computer-readable storage medium storing the computer program according to claim 25.

27. A system comprising one or more processors configured to perform the method according to any one of claims 1 to 24.

Images