Method and system for stereo source elimination

Abstract

The present disclosure relates to a method for processing audio content to suppress a target audio signal and an apparatus implementing the method. The method comprises obtaining an input audio signal, and determining, for a specific segment associated with a segment specific source activity metric indicating that the target audio source is inactive, a processing application coefficient by smoothing a segment specific source activity metric across at least two segments. The method further comprises extracting a difference audio signal based on a difference between a side audio signal and the input audio signal and extracting, from the difference audio signal, a source suppressed difference. The method further comprises forming a modified difference audio signal as a weighed sum of the difference audio signal and the source suppressed difference audio signal based on the processing application coefficient.

Description

METHOD AND SYSTEM FOR STEREO SOURCE ELIMINATION CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 583,198, filed on September 15, 2023, all of which is incorporated herein by reference in its entirety. FIELD

[0002] The present invention relates to a method and system for processing audio signals, and in particular for performing source suppression processing. BACKGROUND

[0003] In some audio processing contexts it is often desirable to suppress certain types of audio sources in an audio signal comprising a mix of multiple audio sources. For example, an audio signal captured at a concert may comprise a mix of music and noise from the crowd wherein it may be desirable to process the audio signal to suppress the crowd noise while leaving the music.

[0004] As another example, dubbing of an audio signal associated with a film or video involves replacing original speech of a first language with replacement speech of a second language. In most modern professionally generated content (PGC), dialogue is often present in a separate audio channel with other audio content (music, effects, ambience) being comprised in other audio channels. Dubbing this type of content in which the speech is present in a separate channel could simply involve replacing the speech channel with a different speech channel, recorded and / or processed so as to be in sync with the original video content and the other audio content (music, effects, ambience).

[0005] However, for many types of audio content the speech is mixed with other audio content in one or more channels, with each channel carrying a mix of speech with other audio content. This is often the case for user generated content (UGC) where e.g. a user has recorded a scene with his / her smartphone capturing a stereo audio signal comprising a mix of speech, music, effects and ambience. To dub an audio signal which comprises the original speech mixed with other audio content the audio signal is often processed to remove or at least suppress the original speech prior to adding the replacement speech, by e.g. mixing an audio signal comprising only the replacement speech with the processed audio signal which has the original speech suppressed.

[0006] In a simple implementation, the volume of the original audio signal carrying the mix of the original speech and other audio content is turned down, e.g. silenced completely, when the original speech is active. This ensures that the original speech is suppressed and, after the replacement speech has been mixed in at corresponding points in time, the fact that other audio content also is attenuated silenced is less noticeable for listeners. SUMMARY

[0007] A problem with the above described solutions for dubbing audio signals is that the audio processing results in some acoustic artifacts which are noticeable and disruptive to some listeners. It is therefore a purpose of the present disclosure to provide an audio processing method and audio processing system that offers enhanced performance for suppressing a designated target audio source while still preserving the spatial and / or timbral qualities of the original audio signal which results in a source suppressed output audio signal with reduced or completely removed acoustic artifacts.

[0008] According to a first inventive concept of the present invention there is provided a method for processing audio comprising obtaining an input audio signal comprising two channels, the input audio signal comprising a plurality of consecutive segments, determining, for each segment of the input audio signal, a segment specific source activity metric indicating whether a predetermined target audio source is active in the segment, determining, for a specific segment associated with a segment specific source activity metric indicating that the target audio source is inactive, a processing application coefficient by smoothing the segment specific source activity metric across a group of segments associated with the specific segment, wherein the group of segments comprises a segment associated with a segment specific source activity metric indicating that the target audio source is inactive and a segment associated with a segment specific source activity metric indicating that the target audio source is inactive. The method further comprising for the specific segment extracting, from the input audio signal, a side audio signal, extracting a difference audio signal based on a difference between the side audio signal and the input audio signal, extracting, from the difference audio signal, a source suppressed difference audio signal by using a source separator configured to determine to what extent the specific segment contains audio content associated with the target audio source or residual audio content, and forming the source suppressed difference audio signal based on the extent to which the specific segment contains audio content associated with the target audio source. The method further comprising forming a modified difference audio signal as a weighed sum of the difference audio signal and the source suppressed difference audio signal based on theprocessing application coefficient and combining the modified difference audio signal with the side audio signal to form an output audio signal in which the target audio source is suppressed.

[0009] The group of segments comprises at least two segments. With the group of segments being associated with specific segment (which is inactive) it is meant that the specific segment is included in the group of segments along with at least one active segment or that the segment specific source activity metric of group of segments is used to extrapolate or interpolate the processing application coefficient for the specific segment which e.g. lies temporally between, earlier or later than the segments of the group. Since the group comprises an active segment and an inactive segment the smoothing is applied when the target source transitions from being active to inactive or vice versa. In some implementations the smoothed processing application coefficient is only determined for inactive segments wherein active segments use a processing application coefficient equal to the segment specific source activity metric.

[0010] In some implementations, the segment specific source activity metric is a binary value assuming a high value (e.g.1) when the target source is active and a low value (e.g.0) when the target source is inactive. Accordingly, when target audio source becomes active or inactive there is an abrupt step in segment specific source activity metric. The processing application coefficient may be a smoothed version of the segment specific source activity metric which fulfills e.g. a minimum ramping up rate or ramping down rate. For example, the processing application coefficient can assume a value of 0, a value of 1 or a fractional value between 0 and 1 when the segment specific source activity metric is a binary value of 0 or 1. As a further example, the processing application coefficient for a specific segment is found by running a smoothing window across the segment specific source activity metric, wherein the smoothing window contains one or more look-ahead and / or one or more look-back segments for the specific segment.

[0011] The method according the first aspect of the present invention generates an output audio signal comprising the audio content of the side audio signal and a weighted combination of the source suppressed difference audio signal and the difference audio signal wherein the combination is based the processing application coefficient. As most types of target audio sources (e.g. speech, music, music from a specific instrument) is present in both channels of the input audio signal, and therefore not present in a side audio signal being based on the difference between the channels, the side audio signal is expected to contain audio content associated with background audio content. With the term “background audio content” or “residual audio content” it is meant any audio content which is not associated with the target audio source.

[0012] Furthermore, the difference audio signal being based on the difference between the side audio signal and the input audio signal comprises all audio content present in the input audio signal but not present in the side audio signal. The difference audio signal will likely contain the target audio source, but also some background audio content. To this end, the source separator is used to process the difference audio signal to extract any residual audio content from the difference audio signal. The residual content from the side audio signal and from the difference audio signal are then combined to form the output audio signal with the target audio source removed but most or all residual audio content intact.

[0013] For most types of audio content, the target audio source will not be active at all times (i.e. during all segments of the audio signal) meaning that at some instances (when the target audio source is inactive) there is no target audio source to eliminate. To maintain as much content and spatial properties as possible from the input audio signal the input audio signal is used as the output audio signal for inactive segments. For example, this may entail using the difference audio signal as the modified difference audio signal whereby the input audio signal is reconstructed perfectly to form the output audio signal.

[0014] On the other hand, during segments when the target audio source is active, the target audio source should be fully suppressed to only include background audio content in the output audio signal. For example, this is achieved by using the source suppressed difference audio signal as the modified difference audio signal whereby the input audio signal is the combination of the side audio signal and the source suppressed difference audio signal.

[0015] It is however preferable that the change between using the difference audio signal and the source suppressed difference audio signal as the modified difference audio signal does not occur to rapidly as this causes undesirable acoustic artifacts. To this end, when transitioning from segments in which the target audio source is active to segments in which the target audio is inactive, or vice versa, the modified difference audio signal for a specific segment is formed as a weighed sum of the difference audio signal and the source suppressed difference audio signal. That is, for the specific segment, the modified difference audio signal is a combination of the difference audio signal and the source suppressed difference audio signal with a non-zero contribution from each of these audio signals. The specific segment for which the weighted combination is formed is a segment in which the target audio source is inactive, this ensures that the target audio source is not reintroduced by the weighted sum. For example, the modified difference audio signal based on the weighted combination is only formed for segments associated with a target source activity metric indicating that the target audio source is inactive.

[0016] In some implementations, smoothing the segment specific target source activity metric to form the processing application coefficient, PAC, may comprise at least one of: enforcing a maximum ramp-up and / or ramp-down rate for the PAC and determining sufficiently short sequences of segments associated with an inactive target audio source and setting the PAC to a high value associated with an active source for these segments.

[0017] According to a second inventive concept there is provided a method for processing audio comprising obtaining an input audio signal comprising two channels, the input audio signal comprising a plurality of consecutive segments and determining, for each segment of the input audio signal, a segment specific source activity metric indicating whether a predetermined target audio source is active in the segment. The method further comprises, for each segment, extracting, from the input audio signal, a side audio signal, extracting a difference audio signal based on a difference between the side audio signal and the input audio signal and extracting, from the difference audio signal, a source suppressed difference audio signal by processing the difference audio signal with a source separator. The method further comprises determining an inactive energy level based on the spectral energy of the difference audio signal for at least one segment associated with a segment specific activity metric indicating that the target audio source is inactive, determining an active energy level based on the spectral energy of the source suppressed difference audio signal for at least one segment associated with a segment specific activity metric indicating that the target audio source is active, applying a gain to at least one of the side audio signal and the source suppressed difference audio signal, based on the difference between the inactive and active energy level for the at least one segment associated with a segment specific activity metric indicating that the target audio source is active and combining the source suppressed difference audio signal with the side audio signal, wherein at least one of the source suppressed difference audio signal and the side audio signal has the gain applied, to form an output audio signal in which the target audio source is suppressed.

[0018] As a portion of the audio content is removed when the target audio source is eliminated there may be a perceptible difference in spectral energy (i.e. perceived volume) between segments in the output audio signal where the input audio signal is active and inactive. By determining and applying a compensation gain using the method of the second inventive concept this difference in spectral energy can be reduced or removed.

[0019] According to some implementations of the second inventive concept, the active and / or inactive energy is determined as a mean and / or average over multiple active and inactive segments respectively.

[0020] According to a third inventive concept there is provided a method for processing audio comprising obtaining an input audio signal comprising two channels, the input audio signal comprising a plurality of consecutive segments and, for each segment, extracting, from the input audio signal, a side audio signal and extracting a difference audio signal based on a difference between the side audio signal and the input audio signal. The method further comprises extracting, from the difference audio signal, a source suppressed difference audio signal by processing the difference audio signal with a source separator, combining the source suppressed difference audio signal with the side audio signal to form an output audio signal in which the target audio source is suppressed and determining, for a plurality of frequency bands of each segment of the output audio signal a panning parameter and / or a phase parameter. The method further comprises scaling the panning parameter of each frequency band and segment with a panning scaling function, the panning scaling function being based on a deviation between the panning parameter and center panning, to obtain an adjusted panning parameter with greater deviation from center panning compared to the panning parameter and / or determining a reference phase for each frequency band and segment and scaling the phase parameter of each frequency band and segment with a phase scaling function, the phase scaling function being based on a deviation between the phase parameter and the reference phase, to obtain an adjusted phase parameter with greater deviation from reference phase compared to the phase parameter. The method further comprises forming a modified output audio signal using the adjusted panning parameter and / or adjusted panning parameter.

[0021] For many types of audio content the target audio source to be eliminated dominates the audio content and is center panned and / or features a near-zero phase difference. After elimination of the target audio source, the remaining background audio content will have inherited the same spatial properties (panning and / or phase) from the target audio source. However, background audio content is perceived as much more convincing if the spatial features are more broadly distributed.

[0022] This is achieved by determining the spatial properties of the output audio signal and scaling (and optionally smoothing) these spatial properties to cover a larger range of panning and / or phase parameters and using the adjusted spatial parameters for forming a modified output audio signal.

[0023] The source separator in the second and / or third inventive concept may be the same source separator as the one used in the first inventive concept. However, it is understood that the source separator of the second and / or third inventive concept may be different. For example, it is envisaged that the source separator merely attenuates (e.g. completely silences) the differenceaudio signal for segments in which the segment specific source activity metric indicates that the target audio source is inactive.

[0024] The second and third inventive concepts may be used in addition to or as an alternative to the method of the first aspect of the invention. Each inventive concepts relates to an audio processing technique forming a source eliminated audio signal with enhanced perceived quality of the resulting output audio signal having the target source suppressed. The inventive concepts listed above can be used separately or combined as desired.

[0025] According to another aspect of the present invention there is provided an apparatus comprising one or more processors, and a memory storing instructions that when executed by the one or more processors, cause the one or more processor to perform the method of one or more of the inventive concepts.

[0026] According to a further aspect of the present invention there is provided a non- transitory, computer readable storage medium having stored thereon instructions, that when executed by one or more processors, cause the one or more processors to perform the method of any of the inventive concepts. DESCRIPTION OF THE DRAWINGS

[0027] Aspects of the present invention will be described in more detail with reference to the appended drawings, showing exemplary embodiments.

[0028] Figure 1 is a block diagram showing an audio processing system according to some implementations.

[0029] Figure 2 is a flow chart describing a method for processing audio signals according to some implementations.

[0030] Figure 3A is a graph showing a segment specific source activity metric that varies over time according to some implementations.

[0031] Figure 3B is a graph of a processing application coefficient, PAC, obtained by smoothing the segment specific source activity metric of fig.3A, according to some implementations.

[0032] Figure 4A is block diagram illustrating how the source suppressed difference audio signal can be combined with the difference audio signal to form the modified difference audio signal according to some implementations.

[0033] Figure 4B is a block diagram illustrating an alternative arrangement for determining the modified difference audio signal, according to some implementations.

[0034] Figure 5 is a block diagram showing the details of the source suppressor according to some implementations.

[0035] Figure 6A shows an example of a softmask determined by a source separator according to some implementations.

[0036] Figure 6B shows an example of a more inclusive modified softmask that has been generated from the softmask in fig.6A, according to some implementations.

[0037] Figure 7a is graph illustrating the spectral energy difference between the input audio signal and the side audio signal for a segment in which the target audio source is inactive according to some implementations.

[0038] Figure 7b is a graph illustrating the spectral energy difference constituted by the source suppressed difference audio signal for a segment in which the target audio source is active, according to some implementations.

[0039] Figure 8 is a block diagram showing a brick-wall filtering system that can be used together with audio processing system according to some implementations.

[0040] Figure 9 is a block diagram showing how the output audio signal can be mixed with a replacement audio signal according to some implementations. DETAILED DESCRIPTION

[0041] Systems and methods disclosed in the present application may be implemented as software, firmware, hardware or a combination thereof. In a hardware implementation, the division of tasks does not necessarily correspond to the division into physical units; to the contrary, one physical component may have multiple functionalities, and one task may be carried out by several physical components in cooperation.

[0042] The computer hardware may for example be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a smartphone, an AR / VR wearable, automotive infotainment system, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that computer hardware. Further, the present disclosure shall relate to any collection of computer hardware that individually or jointly execute instructions to perform any one or more of the concepts discussed herein.

[0043] Certain or all components may be implemented by one or more processors that accept computer-readable (also called machine-readable) code containing a set of instructions that when executed by one or more of the processors carry out at least one of the methods described herein. Any processor capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken are included. Thus, one example is a typical processing system (e.g., computer hardware) that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system further may include a memory subsystem including a hard drive, SSD, RAM and / or ROM. A bus subsystem may be included for communicating between the components. The software may reside in the memory subsystem and / or within the processor during execution thereof by the computer system.

[0044] The one or more processors may operate as a standalone device or may be connected, e.g., networked to other processor(s). Such a network may be built on various different network protocols, and may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

[0045] The software may be distributed on computer readable media, which may comprise computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to a person skilled in the art, the term computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, physical (non-transitory) storage media in various forms, such as EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, it is well known to the skilled person that communication media (transitory) typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

[0046] Fig.1 is a block chart showing schematically an audio processing system 1 according to some implementations.

[0047] The audio processing system 1 is configured to perform audio processing that removes a target audio source (e.g. speech, music, or a specific music instrument, such as guitar) from the two channel input audio signal IN to generate an output audio signal OUT having the target audio source removed but otherwise the having spatial and timbral qualities of the input audio signal IN preserved. Preserving the spatial and timbral qualities of the input audio signal IN even after the target source has been removed is a challenging task and the present disclosure presents various audio processing techniques that can be used individually, or combinedarbitrarily, to produce an output audio signal OUT having the spatial and timbral qualities of the input audio signal IN even when the target audio source has been suppressed or eliminated completely.

[0048] The input audio signal IN comprises two channels and may for example be a stereo audio signal or a binaural audio signal. However, the audio processing system 1 can process audio content of other formats as well. For example, a mono audio signal or an audio signal with more than two channels can be converted to a two channel format using any suitable known umixing or downmixng process whereby the two channel input audio signal is processed with the audio processing system and optionally converted back to the original format after processing. As a further alternative, input audio signals IN with three or more channels can be processed with the audio processing system 1 operating on pairwise combinations of the three or more channels.

[0049] Common for all audio processing techniques is that the audio processing system 1 operates using a difference audio signal D and a side audio signal S extracted from the input audio signal IN. The operation of the audio processing system 1 will now be described with further reference to the flowchart in fig.2.

[0050] The input audio signal IN comprises two channels (e.g. a stereo input audio signal comprising a left channel L and a right channel R) and is obtained by the audio processing system 1 at step S1. The input audio signal is provided to the difference-side extractor 11 which extracts, at step S3 a difference audio signal D and a side audio signal S from the input audio signal IN. For some audio processing techniques, a source activity metric α is extracted from the input audio signal at step S2. The source activity metric α is described in further detail below and may be used to control the audio processing techniques.

[0051] In general, extraction of a mid audio signal and side signal M, S from a left channel L and right channel R of the input audio signal IN can be achieved by determining the sum and difference, respectively, of the left and right audio channel L, R. In short, the mid and side audio signal M, S can be determined as ^ = ^^^ + ^^ (eq.1) ^ = ^^^ − ^^ (eq.2) wherein the coefficient β is equal to 1⁄2 or β is equal to 1⁄√2. With β equal to 1⁄√2 the energy is preserved, meaning that the total spectral energy of the mid and side audio signal M, S is equal to the spectral energy of the left and right audio signal L, R.

[0052] Using equations 1 and 2 above, a center panned mid and side audio signal M, S are extracted. For many types of audio content, a center panned mid-side pair is appropriate sincemany types of target audio sources (such as speech) are often center panned meaning that the target audio source will be included in the mid audio signal and any background audio content left in the side audio signal. However, more generally, the mid and side audio signal M, S audio signal are extracted with an arbitrary panning parameter θ as: ^ = L cos ^ + R sin ^ (eq.3) ^ = L sin ^ − R cos ^ (eq.4) wherein 0 ≤ θ ≤ π / 2, and θ = π / 4 denotes center panning. The panning parameter θ can be fixed, for e.g. specified by a user or it can be dynamic over time and frequency, for example determined as a detected panning parameter as explained in “TARGET MID-SIDE SIGNALS FOR AUDIO APPLICATIONS” filed as International Application No. PCT / US23 / 63717, filed 03 March 2023, hereby incorporated by reference in its entirety. In brief, this reference discloses determining, for each time-frequency tile, a detected panning parameter and averaging the detected panning parameter across multiple tiles (over time and / or frequency) to form a target panning parameter. The target panning parameter is then used to form the target mid audio signal. For example, this dynamic mid audio signal is capable of capturing multiple sources with difference panning, provided they are separated in frequency and / or time.

[0053] In the following, the audio signals will be assumed to be represented in Short Time Fourier Transform (STFT) domain wherein any audio signal is represented with a plurality of time-frequency tiles, each tile comprising a plurality of STFT coefficients representing a specific frequency band of a specific segment.

[0054] When extracting the difference audio signal D and side audio signal S, the difference-side extractor 11 extracts the side audio signal S using equation 4 above. For each time-frequency tile the difference-side extractor 11 also determines at least the phase parameter φ and magnitude U of the side audio signal S, and optionally also the panning parameter θ. Especially, the difference-side extractor 11 determines the phase and magnitude for the side signal S as U = ^|S|^(eq.5) ^ = ^^ ^^^ (eq.6) However, the panning parameter θ of the side audio signal S is that of the input audio signal IN. Accordingly, the difference-side extractor 11 also determines the panning parameter θ for each time frequency-tile of the side audio signal S using the input audio signal IN as θ = arctan%|&||'|((eq.7)for corresponding time frequency tiles of the input audio signal IN. The panning, phase and magnitude parameters are collectively referred to as spatial parameters or Spatio-Level Filtering (SLF) parameters.

[0055] The side audio signal S is a currently a mono audio signal, however the signal may be converted to a stereo side audio signal S using the SLF parameters as ^)= |^| cos^^^*+,(eq.8) ^^= |^| sin^^^*+,(eq.9) with S1, S2 denoting the two channels of a stereo representation of the side channel S. The stereo side audio signal S is subtracted from the input audio signal IN (also comprising two channels) by the difference-side extractor 11 at step S3 to form the difference audio signal D as: -.), .^0= -^, ^0− -^), ^^0(eq.10)

[0056] with D1and D2representing the two channels of the difference audio signal D. The difference audio signal D is referred to as the “difference” audio signal since it comprises any audio content of the input audio signal IN not included in the side audio signal S. Accordingly, combining the difference audio signal D with the side audio signal S in equal proportions will perfectly reconstruct the input audio signal IN.

[0057] In many situations, the target audio source is center-panned. For most input audio signals IN comprising speech, the speech (when active) is the dominating audio source which governs the panning of the input audio signal IN. Since speech often is center-panned the input audio signal IN will be center-panned meaning that the side audio signal S is substantially free from speech while all speech is present in the difference audio signal D.

[0058] In some cases, the dominating audio source is not center-panned but panned off- center or panned differently over time. For these cases, an identified panning parameter θ can be used to form adaptive side audio signal S as described in equation 4. The adaptive side audio signal is converted to a two-channel representation using equations 8 and 9 whereby the difference audio signal D is extracted using equation 10. With an adaptive side audio signal S the dominating audio source can be assumed to be included in the difference audio signal D and excluded from the side audio signal S even if the dominating audio source has off-center panning and / or time varying panning.

[0059] However, this approximation (the difference audio signal D containing only the target audio source and no other audio content) is accurate only to a certain extent and in some cases the difference audio signal D will also comprise audio content not associated with the target audio source in addition to the target audio source. For example, if the target audio source is any speech content, noise or background audio may be present in the difference audio signal Din addition to speech. To this end, the difference audio signal D is passed to a target source separator 12 which at step S4 extracts a source suppressed difference audio signal Dsuppressedfrom the difference audio signal D. The source suppressed difference audio signal Dsuppressed is effectively a processed version of the difference audio signal D with any content associated with the target audio source removed and / or attenuated.

[0060] In some implementations, the target source separator 12 comprises a denoising module (e.g. implemented as a trained neural network) configured to determine a softmask for suppressing any audio content that does not belong to the target audio source. The determined softmask is applied to the difference audio signal D to generate a source audio signal Dsource carrying audio content associated with target audio source. The source suppressed difference audio signal Dsuppressed can then be formed by subtracting the source audio signal Dsource from the difference audio signal D as .1233451156= . − .172485. (eq.11) audio content that doesnot belong to the target audio can be inverted using trivial operations to form an inverted softmask that can be applied to the difference audio signal D to form the source suppressed difference audio signal Dsuppressed directly. Alternatively, the target source separator 12 may be configured to determine the inverted softmask directly.

[0062] Further details of the target source separator 12 will be described below, in connection to fig.5.

[0063] The source suppressed difference audio signal Dsuppressed and side audio signal S are passed to subsequent modules in the audio processing system 1 wherein one or more of the following audio processing techniques will be performed to form an output audio signal OUT, or modified output audio signal OUTMwith spatial and timbral qualities restored.

[0064] Audio processing technique 1 involves smoothly transitioning from perfect reconstruction of the input audio signal IN at times when the target audio source is inactive to reconstruction using the source suppressed difference signal Dsuppressedwhen the target audio source is active, or vice versa. This audio processing technique involves the target source identifier 18, the target source separator 12 and the combiner module 14. Audio processing technique 1 comprises step S5 of forming a modified difference audio signal Dmodified weighted combination of the difference audio signal D and the source suppressed difference audio signal Dsuppressed such that for time segments where the target source is active the weighted combination is equal to the source suppressed difference audio signal Dsuppressed, for time segments where the target source is inactive the weighed combination is equal to the difference audio signal and fortime segments in between, where the target source transitions from being active to inactive, or vice versa, the weighted combination is a interpolation of the source suppressed difference audio signal Dsuppressed and the difference audio signal D. The modified difference audio signal Dmodified is then combined with the side audio signal S at step S6 to form the output audio signal OUT.

[0065] Audio processing technique 2 involves biasing the extraction of the source suppressed audio signal in the target source separator 12 at step S4 to exclude the target audio source with an extra safety margin by also excluding some audio content that does not belong to the target audio source. At the cost of suppressing some audio content that is not associated with the target audio source the audio processing system 1 will mitigate or completely remove the risk of the target audio source leaking through to the output audio signal OUT. This is especially beneficial when the target audio source is speech, since any speech “leftover” the output audio signal OUT has been found to be easily distinguishable by users which perceive the “leftover” speech as a disturbance.

[0066] Audio processing technique 3 involves determining and applying a compensation gain Cgainto the side audio signal S and / or the source suppressed difference audio signal Dsuppressed of segments in which the target audio source is active to more closely resemble the balance between side audio signal S and difference audio signal D of segments in which the target audio source is inactive. This audio processing technique uses the level analyzer 17 and gain compensator module 15. By suppressing the target audio source during active segments and reintroducing some additional background audio content in form of the source suppressed difference audio signal Dsuppressed there may be a change in perceived spectral energy for background audio content between active and inactive segments. To minimize the perceptual effect of this, a compensation gain Cgain is determined and applied to the side audio signal or source suppressed difference audio signal to compensate for this difference.

[0067] Audio processing technique 4 involves reconstructing the spatial properties of the output audio signal OUT with the spatial parameter adjuster 13. As the extracted difference and side audio signals D, S will be biased towards center panning and / or near-zero phase difference, and it has been found that by stretching the panning and / or phase difference to form a wider distribution significantly enhances the spatial fidelity of the audio signal.

[0068] Audio processing technique 5 involves performing any of the above audio processing techniques for a predetermined frequency band of an original audio signal INorg and leaving the remaining frequency bands unprocessed. Using e.g., brick-wall filters, an original input audio signal INorgcan be divided into two portions in frequency, a first portion which is used as the input audio signal IN and processed with the audio processing system 1 and a secondportion which is left unprocessed and mixed with the output audio signal OUT to form a final output audio signal OUTfinal. The frequency bands of the first portion are selected to cover the expected frequency range of the target audio source and the frequency bands of the second portion are expected to be substantially free from audio content associated with the target audio signal.

[0069] In the following, audio processing techniques 1 - 5 outlined above will now be described in detail.

[0070] Audio processing technique 1 involves utilizing the combiner 14 which forms a weighted combination of the suppressed difference audio signal Dsuppressed and difference audio signal D based on a segment specific target source activity metric α obtained from the target source identifier 18.

[0071] The segment specific target source activity metric α is extracted by the target source identifier 18 for each segment of the input audio signal IN, or the difference audio signal D, and indicates whether the target audio source is active or not in the current segment. The target source identifier 18 may be any type of identifier configured to identify any type of audio source. For example, the target source identifier 18 may be configured to determine whether or not speech or music content is comprised in the input audio signal IN or the difference audio signal D. The target source identifier 18 may be / comprise a voice activity detection (VAD) module.

[0072] The segment specific target source activity metric α may be a binary value (e.g. indicating either that the target audio source is active or inactive). The segment specific target source activity metric α may vary between subsequent time segments as the target audio source becomes active or inactive. For example, speech may be active for some segments but not for other segments.

[0073] The segment specific target source metric α may be extracted from a continuous value (e.g. indicating the likelihood that the target audio source is active) by comparing the continuous value to a threshold αT. For example, a continuous value indicating the likelihood that the source is active is determined for each segment wherein the likelihood is a value between 0 and 1, i.e. a continuous value ranging from 0% to 100%. The segment specific target source activity metric is then the result of comparing the continuous value to the threshold αT. If the continuous value exceeds the threshold αT, the segment specific source activity metric assumes high value α2 (e.g. one) and if the continuous value is below the threshold, the segment specific source activity metric assumes the low value α1(e.g. zero). For example, the threshold αT is 0.5 or 50%.

[0074] With further reference to fig.3A there is shown an example of a segment specific source activity metric α that takes a high value α2(e.g. one) when the target audio source is active and a low value α1 (e.g. zero) when the target audio source is inactive. Of course, the audio processing system 1 could be designed to operate with a segment specific source activity metric α that is defined differently.

[0075] The segment specific target source activity metric α is provided to the combiner 14 which may combine the source suppressed difference audio signal Dsuppressed with the difference audio signal to form a modified difference audio signal Dmodified according to .976:;:56= <.1233451156+ ^1 − <^. (eq.12) Dmodified is equal to Dsuppressedα zero, source Dmodifiedis equal to D. This enables the difference audio signal to be perfectly reconstructed when the target audio source is inactive and some of the audio content of the difference audio signal D that does not belong to the target audio source to be included in the modified difference audio signal Dmodified when the target audio source is active.

[0076] However, in some implementations the segment source activity metric α may vary rapidly which may cause noticeable acoustic artifacts as the weighting of the difference audio signal D and the source suppressed difference audio signal Dsuppressedchanges quickly. To this end, the segment specific source activity metric α may be smoothed across time over a group of segments comprising at least two segments to form a processing application coefficient, PAC, that controls the combination of the difference audio signal D and source suppressed difference audio signal Dsuppressed. The extraction of the PAC from the segment source activity metric α is performed by the PAC extractor 19.

[0077] Smoothing, by the PAC extractor 19, the segment specific source activity metric α across time to form the PAC may comprise smoothing the segment specific source activity metric α across at least two segment, comprising at least one active segment and at least one inactive segment. For example, smoothing may be performed over one or more look-back segments and / or one or more look-ahead segments relative a specific segment for which the PAC is to be determined. For some segment sequences (e.g. between t1and t2in fig.3A) where the target audio source is constantly active or inactive the PAC is equal to the segment specific source activity metric α. However, the PAC will deviate from the segment specific source activity metric α in the area surrounding each transition from active to inactive or vice versa. Importantly, the PAC is equal to the specific source activity metric α for all active segments with any smoothing to form smooth transitions occurring for segments in which the target audiosource is inactive. This will prevent audio content associated with the target audio source from leaking through to the modified difference audio signal Dmodified.

[0078] Smoothing, by the PAC extractor 19, the segment specific source activity metric α may additionally or alternatively comprise processing the segment specific source activity metric α to exclude short periods of the segment specific source activity metric α that are below a predetermined threshold value and / or processing the segment specific source activity metric to exhibit a predetermined minimum attack and / or release time. The smoothing may result in a processing application coefficient PAC as shown in fig.3B.

[0079] Processing the segment specific source activity metric α to exclude short segment sequences being below a predetermined threshold value may comprise determining a number of consecutive segments (or a time portion of the input or difference audio signal D, IN) that is associated with a segment specific source activity metric α that is below a predetermined threshold αT(e.g. αT= α1 / 2). If the number of segments and / or the duration is below a predetermined maximum number of segments Nmax or maximum duration the PAC for these segments is set to the maximum value α2. In effect, this means that short periods of inactive source will be ignored by the combiner 14 which is controlled by the PAC which avoids performing perfect reconstruction for very short periods of inactivity.

[0080] An example of a PAC obtained by this type of smoothing is depicted in fig.3A and 3B between times t2and t3, where the segment specific source activity metric α during t2to t3is below the predetermined maximum number of segments Nmax or maximum duration. The segment specific source activity metric α in fig.3A drops briefly to α1 (e.g. since the target audio source is inactive for a brief period of time) but to avoid changing the weighting of the difference audio signal D and the source suppressed audio signal Dsuppressed too often this short drop in segment specific source activity metric α is ignored as can be seen in fig.3B wherein the PAC remains at α2 from t2 to t3. The PAC is thereby biased towards retaining a state indicating that the target audio source is active.

[0081] Processing the segment specific source activity metric α to exhibit a predetermined minimum attack and release time comprises obtaining a minimum time for transition from the low value α1 to the high value α2 (minimum attack time) and a minimum time for transition from the high value α2to the low value α1(minimum release time). The minimum attack and release times may be different or may be the same. For example, the minimum attack and release times are both at least one second, at least two seconds or at least three seconds.

[0082] For example, by comparing fig.3A with fig.3B, it is seen that the PAC features a less rapid transition from the low value α1 to the high value α2 between times t0 and t1 comparedto the segment specific source activity metric α which increases substantially instantaneously from the low value α1to the high value α2at time t1. This is due to the minimum attack time putting a limit on the maximum rate of change for the PAC. Similarly, the minimum release time means that it takes much longer for the PAC to transition from the high value α2to the low value α1 between times t4 and t5 compared to the segment specific source activity metric α which drops from α2to α1substantially instantaneously at time t4.

[0083] A further requirement may be put on the PAC, namely that the PAC should be equal to a high state indicating active target source for at least all segments for which the segment specific source activity metric is equal to the high value α2. In other words, this means that any ramping or smoothing occurs for segments for which the target audio source is inactive.

[0084] The PAC may be used to replace the segment specific source activity metric α in equation 12 above to achieve smoothly varying weighting of the source suppressed difference audio signal Dsuppressedand the difference audio signal D. This is shown in figure 4a, showing additional details of the combiner 14 which forms the modified difference audio signal as .976:;:56= =^> × .1233451156+ ^1 − =^>^. (eq.13) replaced by a combiner 14’, shownin fig.4B, which forms the modified difference audio signal Dmodified by weighting the source suppressed difference audio signal Dsuppressedwith the source audio signal Dsourceusing the PAC. Combiner 14’ is used in combination with target source separator 12’ (see fig.5) which outputs both the target source difference audio signal Dsourceand the source suppressed difference audio signal Dsuppressed.

[0086] More specifically, combiner 14’ forms the modified difference audio signal as .976:;:56= .1233451156+ ^1 − =^>^.172485. (eq.14) That is, the source suppressed difference audio signal Dsuppressed is always included in the modified difference audio signal Dmodifedwherein the source difference audio signal Dsourceis weighted in smoothly when the target audio source is not active to approach a modified difference audio signal comprising a combination of Dsuppressedand Dsourcewhich forms the full difference audio signal D (i.e. perfect reconstruction).

[0087] While the details of the weighting in combiner 14 and combiner 14´ from fig.4A and fig.4B differs, it is understood that the overall processing performed by the block charts from fig.4A and fig.4B is equivalent. For example, since all audio content of the difference audio signal D is divided between the source difference audio signal Dsourceand the source suppressed difference audio signal Dsuppressed it holds that Dsource = D – Dsuppressed. If Dsource inequation 14 is replaced with D – Dsuppressed the result is the weighting formulation from equation 13, meaning that while the equations may appear different the weighting process is the same.

[0088] Additionally, the target source separator 12 from fig.4A may be identical to the target source separator 12’ with the only difference being that Dsourceis not provided as an output from the target source separator 12.

[0089] The weighting provided to the difference audio signal D and the source suppressed difference audio signal Dsuppressed is controlled by the PAC which in turn may be a smoothed version of the segment specific target source activity metric α. The smoothing is biased towards retaining an indication that target audio source is active. For example, the modified difference audio signal Dmodified based on the weighted combination is only formed for segments associated with a target source activity metric indicating that the target audio source is inactive. For any segment associated with a target source activity metric indicating that the target audio source is active the difference audio signal is source suppressed difference audio signal with no influence from the difference audio signal. In this way, the influence of the source suppressor 12, 12’ is smoothly faded out during inactive segments by introducing more and more of the difference audio signal.

[0090] Turning to fig.5, further details of the target source separator 12’ will now be described. The difference audio signal D is provided to a background suppressor 121 that determines a softmask having a gain value for suppressing background audio content in the difference audio signal D. The softmask has one gain value for each of a plurality of frequency bands for each segment. For example, if the target audio source is speech, the background audio content could comprise one or more of music, stationary and / or non-stationary noise, the sound of rain or wind etc. On the other hand, if the target audio source is music the background audio content may comprise speech but not music.

[0091] The background suppressor 121 may comprise a trained neural network that has been trained to predict a gain mask for suppressing background audio content for the target audio source. For example, the gain mask may be a softmask.

[0092] A softmask 125 is shown schematically in fig.6A wherein the difference audio signal has been divided into a plurality of consecutive segments (columns in fig.6A) with a respective timestamp T wherein each segment comprises a plurality of frequency bands F (rows in fig.6A). For each tile (individual table element in fig.6A) the background suppressor 121 determines a gain for suppressing any audio content not associated with the target audio source. In fig.6A, the gain values are values between zero and one where zero results in a complete silencing of a tile and one results in maintaining the tile as is. In the depicted example, the firstand last column of the softmask 125 has generally low gain values. This indicates that the background suppressor 121 has indicated that in these tiles there is no target audio source. On the other hand, in the third and fourth column some tiles are associated with much higher gains, even the maximum gain of one is assigned for some tiles. This indicates that the background suppressor 121 has indicated that these tiles comprise content associated with the target audio source. The softmask gains 125 are therefore indicative of the extent to which each frequency band comprises audio content associated with the target audio source or residual audio content.

[0093] In some implementations, a simplified softmask can be used which comprises a single value for each segment indicating to what extent the segment comprises audio content associated with the target audio source or residual audio content. That is, the division of each segment into a plurality of frequency bands, with separate gain values for each frequency bands, may enhance performance but is not necessary for all implementations.

[0094] The softmask 125 may be smoothed across time to produce a smoothed softmask. For example, the softmask is averaged across ten or more segments such as averaged across about 35 segments. The smoothing may be performed by softmask processor 122.

[0095] Audio processing technique 2 involves further processing of the softmask 125 or smoothed softmask to make the softmask “more inclusive”, i.e. biased towards identifying more content as associated with target source. This further processing may be performed by the softmask processor 122. Most background suppressors 121 are configured (e.g. trained if they utilize neural networks) to identify content associated with the target source with a balanced false positive and false negative rate. However, for the purposes of the present audio processing method it is beneficial if the background suppressor has very few, or no, false negatives (i.e. providing a low gain to a tile which in actuality is associated with the target source). False negatives will mean that some of the target source is identified as background, and as the background content may be present in the output audio signal users may hear the misclassified target source. This is especially noticeable when the target audio source is speech, since users are generally very sensitive to any speech content, even if the speech content is limited in frequency and time duration.

[0096] To circumvent these issues, the softmask 125 or smoothed softmask is processed to form a modified softmask 125’ shown in fig.6B. Modifying the softmask 125 or smoothed softmask may comprise, for each tile, scaling all gain values with a positive scaling factor C1 larger than one and / or increasing all gain values with an adjustment factor C2 larger than zero. That is, a gain value g may be modified to form a modified gain value gmodifiedas gmodified= C1*g+ C2. As an example C1 = 1.5 and C2 = 0.15. Optionally, after modifying the gain values the gain values are capped, such that any gain value exceeding one is set to one.

[0097] As seen in fig.6B the modified softmask 125’ is now “more inclusive” compared to the softmask 125 from fig.6A in terms of more tiles being associated with higher gain values indicating that the content belongs to the target audio source. For example, columns three, four and five in the modified softmask 125’ comprises two, three and three tiles, respectively, with the maximum gain 1.000 compared to the corresponding columns of the softmask 125 which has zero, two and two tiles, respectively, with the maximum gain 1.000. Similarly, the other tiles are also associated with higher gain values in the modified softmask 125’ compared to the gain mask 125, indicating that more of the content is associated with the target source for the modified softmask 125’.

[0098] In general, the softmask gains may be defined on the interval [g1, g2] where the minimum value g1indicates the associated segment or frequency band comprises only, or at least a majority, of residual audio content and a value of g2 indicating that the associated segment or frequency band comprises fully, or at least a majority, of target source audio content, or vice versa. The proportion of a segment or frequency band that constitutes the residual audio content or target source content may be defined as the spectral energy of the residual audio content or target source content with respect to the total spectral energy of the segment or frequency band.

[0099] The modified softmask 125’ is provided to the softmask application which applies the modified softmask 125’ to the difference audio signal D to form the source difference audio signal Dsource containing the isolated target audio source. The source difference audio signal Dsourceis provided to a difference calculator 124 alongside the difference audio signal D wherein the difference calculator 124 determines the source suppressed difference audio signal Dsuppressed simply as the difference D – Dsource. The skilled person will further realize that different operations can be used to obtain the source suppressed difference audio signal Dsuppressed. For example, the modified softmask 125’ can be inverted and applied to the difference audio signal D to obtain the source suppressed difference audio signal Dsuppressed directly.

[0100] In the embodiment shown in fig.5, both the source suppressed difference audio signal Dsuppressed and the source difference audio signal Dsource are provided as output, however in some implementations only one of the signals Dsuppressed, Dsourceis provided as the output.

[0101] Furthermore, it is also possible to use a background suppressor 121 that directly determines a softmask that captures the residual or background audio content. In such implementations, the softmask is made less inclusive of the background (to avoid accidentally capturing the target audio source) by a similar procedure. For example, the gain values of thissoftmask are scaled with a positive scaling factor smaller than one and / or adjusted by subtracting an adjustment factor larger than zero and smaller than one to form an adjusted softmask.

[0102] With audio processing technique 2 an enhanced determination of the source suppressed difference audio signal Dsuppressedis provided which mitigates or reduces the risk of the target audio source leaking through the audio processing system. Audio processing technique 2 can therefore be used in an implementation of the source separator used in any of the concepts described herein.

[0103] Audio processing technique 3 comprises adjusting the gain of at least one of the side audio signal and the source suppressed difference audio signal when the target audio source is active. When the target audio source is eliminated the total amount of spectral energy of the audio signal is reduced. Accordingly, without application of a compensation gain the output audio signal may be perceived as becoming quieter in segments where target audio source has been suppressed which in general is undesirable. Audio processing technique 3 relates to determining and applying an appropriate compensation gain to avoid or mitigate the problem of quieter segments.

[0104] With reference to fig.7A the total spectral energy E is compared between the input audio signal IN and the side audio signal S for a segment in which the target audio source is inactive. The total spectral energy E is lower for the side audio signal S than for the input audio signal IN and the difference ∆1constitutes the spectral energy that is included in the difference audio signal D. For segments with inactive target audio source the difference audio signal D is combined with the side audio signal S to recreate the input audio signal IN.

[0105] Fig.7B shows the spectral energy of the side audio signal S plus the source suppressed difference audio signal Dsuppressed and the spectral energy of the side audio signal S during a segment in which the target audio source is active. The difference ∆2constitutes the spectral energy that is included in the source suppressed difference audio signal Dsuppressed.

[0106] To avoid distracting acoustic artifacts caused by using the original difference audio signal D for non-active segments and the source suppressed difference audio signal Dsuppressed ∆1 and ∆2are compared to determine a gain Cgainthat reduces, or completely eliminates, the difference between ∆1 and ∆2 is determined and applied in the gain compensator 15 to at least one of the side audio signal S and the source suppressed difference audio signal Dsuppressedduring active segments.

[0107] The gain Cgain may be greater or smaller than one in linear terms and therefore positive or negative when expressed in decibel units. Accordingly, while Cgainis referred to as a gain it is understood that the term gain in this context covers both an attenuation andamplification of the related audio signals. For example, if the difference in spectral energy ∆1 is 5 dB and ∆2is 10 dB the difference Cgainis determined as ∆1- ∆2is 5 – 10 = -5 dB is applied to the side audio signal S or source suppressed difference audio signal Dsuppressed to compensate.

[0108] As noted above, the compensation gain to be applied by the gain compensator 15 is determined based on difference ∆1 determined for at least one segment in which the target audio source is inactive and the difference ∆2determined for at least one segment in which the target audio source is active. As the differences ∆1 and ∆2 may vary from inactive segment to inactive and from active segment to active segment, respectively, it is preferable to determine an average or median difference ∆1 and / or ∆2 over multiple inactive / active segments and determine Cgain based on the average or median ∆1 and / or ∆2.

[0109] For example, multiple segments in which the target audio source is inactive are identified and difference ∆1 is determined for each of these segments. A median difference is determined across all instances of the ∆1difference (e.g. the median of each ratio expressed in decibels). Additionally or alternatively, an average ∆1 difference is determined as a sum of squares of the spectral energy of the input audio signal IN divided by a sum of squares of the spectral energy of side audio signal. The average may be expressed in decibels. It has further been found that an average of both the median ∆1and average ∆1provides a good estimation of the D1 difference.

[0110] An average and / or median ∆2difference (or the average of both the average and mean ∆2) can be determined analogously for multiple segments in which the target audio source is active.

[0111] In some implementations, determination of the compensation gain Cgainis performed offline. In offline implementations, the entire input audio signal IN, side audio signal S and source suppressed difference audio signal Dsuppressedmay be available meaning that the compensation gain Cgain can be determined based on average and / or median ∆1 and ∆2 differences covering all inactive and active segments respectively.

[0112] Audio processing technique 3 can also be applied online. For example, the average and / or median ∆1and ∆2differences are determined over a window region covering many segments, wherein the window region follows a current online segment. Preferably, the compensation gain Cgainis processed so as to not allow too rapid changes over time since this may cause noticeable, and disruptive, rapid volume changes. To this end, Cgain may be constrained so as to not change more than 1 dB, or more than 2 dB, over a time period of 5 seconds or 10 second.

[0113] In some implementations, the determination of the differences ∆1 and ∆2 as well as the difference Cgaincorresponding to the compensation gain is performed for multiple frequency bands. The frequency bands may be quasi-octave frequency bands. For example, the differences ∆1and ∆2as well as the difference Cgainare determined and applied independently for the following frequency bands: 0 – 400 Hz, 400 – 800 Hz, 800 – 1600 Hz, 1600 – 3200 Hz, 3200 – 6400 Hz, 6400 – 13200 Hz, and 13200 – 24000 Hz.

[0114] Audio processing technique 4 comprises modifying the spatial properties of the output audio signal OUT. For many types of audio content the target source (speech) is center- panned and, when active, constitutes the majority of the audio content. The side audio signal S extracted by the difference and side extractor 11 will reflect this as the panning of the side audio signal S is the panning of the input audio signal IN which will be close to center panning, i.e. θ = π / 4. It is the now eliminated target source which is responsible for the panning being very center- heavy and there is a need for modifying the output audio signal OUT to achieve a panning which is more suitable for the audio content now that the target source has been removed.

[0115] According to some implementations, the spatial parameter adjuster 13 obtains the output audio signal and determines, for a plurality of frequency bands of each segment of the output audio signal OUT, a panning parameter θ. The spatial parameter adjuster 13 then applies a panning scaling function fpanning(θ) to each panning parameter θ to determine an adjusted panning parameter θ’ = fpanning(θ). The panning scaling function fpanning(θ) being configured generate, for any panning parameter θ, an adjusted panning parameter θ’ which deviates more from center panning than the panning parameter θ input to the panning scaling function fpanning(•).

[0116] In some implementations, the panning scaling function is given by ^@=AA B +B^CD E-^6:1F0∗|^6:1F|3^ (eq.15)defined as ^6:1F= ^^ − H / 4^ / ^H / 4^ (eq.16) and wherein the exponent p fulfills 0 < p < 1, for example p = 0.6.

[0117] It is appreciated that any panning parameter θ processed with the scaling function will result in an adjusted panning parameter θ’ with a greater normalized distance from center panning. As an example, if θ is equal to 0.5 radians, θdist as determined by equation 16 above is approximately -0.36. After processing with the panning scaling function fpanning(•) according to equation 15 the adjusted panning parameter is about 0.36 radians giving a normalized distance from center panning of about 0.54. That is, the panning parameter has been stretched further away from center panning at H / 4, in case by being moved from 0.5 radians to 0.36 radians.

[0118] By providing this stretching of the panning parameter θ from center panning, the adjusted panning parameters θ’ will more realistically resemble the panning parameters of actual background audio for a large variety of audio content types.

[0119] The stretching of the panning parameters may cause some disturbing artifacts due to exaggeration of small differences in θ. To mitigate or remove these artifacts the adjusted panning parameters are smoothed over multiple segments. In one implementation, the panning parameters are smoothed using a hamming window spanning ten or more segments, e.g. using a hamming window spanning 31 segments or 41 segments.

[0120] The, optionally smoothed, adjusted panning parameter θ’ replaces the panning parameter θ to form a modified output audio signal OUTmodified with enhances spatial properties.

[0121] A similar process can be performed the spatial parameter adjuster 13 for stretching out the phase parameter φ of the output audio signal OUT. For many types of target sources, the target source has close to zero phase difference. In the same way as for the panning described above, audio content dominated by the target audio source will carry the phase difference of (mainly) the target audio source in the output audio signal even if the target audio source has been eliminated.

[0122] To this end, the spatial parameter adjuster 13 may determine a reference phase parameter φref for each segment and frequency band of the output audio signal. The reference phase φrefmay be the phase of any one of the channels of the input audio signal. In some implementations, the reference phase φref is the phase of any one of the channels of the input audio signal scaled with a scaling factor based on θ, θ’ or smoothed versions thereof. For example, the reference phase is determined as ^45;= ^K∗ CDE^^θ’^ (eq.17)of the input audio signal respectively. In equations 17 and 18 above θ’ can be replaced with θ or smoothed versions thereof.

[0123] Using the reference phase φrefthe phase parameter φ is determined as the difference between the reference phase φref and the phase of the output audio signal. The adjusted phase parameter φ’ is determined using a phase scaling function fphase(φ) as ^@= H ∗ CD E^^^ ∗ |^ / H|3where p is a value fulfilling 0 < p < 1, for example p = 0.6 or p = 0.4. Accordingly, in a manner similar to the stretching of the panning parameter the adjusted phase parameter is stretched to better represent a background audio signal.

[0124] Optionally, the adjusted phase parameter φ’ is smoothed across two or more segments. For example, the adjusted phase parameter φ’ of each frequency band and segment is smoothed using a hamming window. In some implementations, the hamming window covers ten or more segments, such as 21 segments or 31 segments.

[0125] The (optionally smoothed) adjusted phase parameter φ’ replaces the phase parameter φ of the output audio signal to form a modified output audio signal with enhanced spatial properties.

[0126] It is noted that while the spatial parameter adjuster 13 described above performs stretching of both the panning and phase parameter it is envisaged that the spatial parameter adjuster according to some implementations performs stretching of only one of the parameters.

[0127] In any of audio processing techniques 1 – 4 described above it is understood that the processing may be performed individually and independently for one or more frequency bands. For processing technique 1 this may entail determining a segment specific source activity metric α for each frequency band, a corresponding PAC for each frequency band and forming the weighted combination of the difference audio signal D and source suppressed difference audio signal Dsuppressedindividually for each frequency band using individual PAC. For audio processing technique 2 this may entail that individual softmasks are modified (and optionally smoothed) for each frequency band. For processing technique 3 this may entail determining and applying an individual compensation gain for each frequency band. For processing technique 4 this may entail determining and modifying the panning and / or phase parameter individually, for each of a plurality of frequency bands.

[0128] Turning to the block chart in fig.8 audio processing technique 5 will now be described in further detail.

[0129] As described above, audio processing technique 1 involves performing little to no processing the input audio signal when the target source is inactive and performing full processing when the target source is active with a smooth transitioning between the two states using a weighted combination. This enables the input audio signal to be maintained as it is when the target source is inactive. Audio processing technique 5 has the same general purpose, namely to avoid processing portions of the audio content where the target audio source is inactive but audio processing technique 5 is implemented completely separately from any of the other audio processing techniques.

[0130] An original audio signal INoriginal is obtained and passed to a set of frequency domain brick-wall filters 21, 22. The brick-wall filters 21, 22 are complementary, meaning that any frequencies that are stopped by one filter is allowed to pass through another filter. At least one brick-wall filter 21 is configured to filter an original audio input signal INoriginalto form the input audio signal IN. The brick-wall filter 21 associated with the input audio signal IN is configured to let frequencies associated with the target audio source pass while blocking other frequencies.

[0131] Further, at least one brick-wall filter 22 is provided as the complement to the one or more filters associated with the input audio signal IN. The complementing brick-wall filter 22 is allowed to allow any frequencies stopped by the brick-wall filter 21 associated with the input audio signal IN to pass. The signal passing through brick-wall filter 22 is referred to as an unprocessed audio signal INunprocessed since this audio signal will bypass the audio processing system 1.

[0132] The input signal IN is provided to the audio processing system 1 and processed in accordance to at least one of the other audio processing techniques described above to form a modified output audio signal OUTmodified (or an output audio signal OUT if the spatial stretching processing is skipped). The modified output audio signal OUTmodifiedis provided to a signal combiner 23 which combines the modified output audio signal OUTmodified with the unprocessed audio signal from a final signal.

[0133] Accordingly, it is only frequency bands that are expected to contain the target audio source which are passed to the audio processing system 1 for source elimination. Some frequencies bands that are expected to not contain any audio content associated with the target audio source are instead allowed to bypass the audio processing system 1. This means that a portion of the original input audio signal INoriginalwill be left unprocessed and closely resemble the original input audio signal OUToriginal while the target audio source has been removed from the frequency range in which it is expected to exist.

[0134] As an example, if the target audio source is speech, brick-wall filter 21 forming the input audio signal IN may be configured to allow frequencies in the range of 80 – 14000 Hz (a frequency band covering the most common speech frequencies) to pass through while blocking frequencies below 80 Hz and frequencies above 14000 Hz. The complementing brick-wall filter 22 allows frequencies below 80 Hz and above 14000 Hz to pass so as to form an unprocessed audio signal INunprocessed carrying the very low and very high frequency content of the original input audio signal INoriginal. The input audio signal IN carrying frequencies 80 – 14000 Hz is processed with the audio processing system 1 to form a modified output audio signal OUTmodifiedwith the target audio source suppressed. The modified output audio signal OUTmodified is then combined with the unprocessed audio signal INunprocessed(which is not expected to comprise any audio content associated with the target audio source) to form a final audio signal OUTfinal with frequencies ranging from 0 to 20000 Hz or more.

[0135] The passband of the different brick-wall filters 21, 22 may be set based on the type of target audio source. In the example above, the passbands of the filters 21, 22 are set to capture speech in the input audio signal however many other alternatives are possible. For example, if it is known that the speech is male speech or female speech the 80 – 14000 Hz frequency range may be adjusted such that the higher frequencies are excluded for male speech and the lower frequencies are excluded for female speech. As a further example, if a musical instrument is to be eliminated the passband of the brick-wall filter 21 associated with the input audio signal is set to the typical frequency range of the instrument and brick-wall filter 22 is set to the complement.

[0136] The audio processing system 1, together with the brick-wall filters 21, 22 form a frequency selective audio processing system 1’.

[0137] Fig.9 is a block diagram illustrating how the final output audio signal OUTfinalcan be mixed with a replacement audio signal INreplace to form a replacement output audio signal OUTreplace. The frequency selective audio processing system 1’ has achieved suppression of the target audio source and in some implementations it is desirable to replace the suppressed source with a replacement source. The replacement source is comprised in the replacement audio signal INreplace. The final output audio signal OUTfinal and the replacement input audio signal INreplace are provided to a signal combiner 2 which mixes the signals to obtain a replacement output audio signal OUTreplacewherein the target audio source present in the original input audio signal INorghas been replaced with the replacement audio source present in the INreplace signal.

[0138] In one example implementation the target audio source that is suppressed is speech and the replacement audio signal INreplace is speech of a different language. Effectively, this achieves dubbing wherein the speech content can be replaced with different speech content.

[0139] As another example, the target audio source is a specific instrument, wherein the replacement audio signal INreplacecomprises different music played by the same instrument type or music from a different instrument. In this way, it is possible to replace e.g. a guitar solo with a different guitar solo.

[0140] It is understood that the frequency selective audio processing system 1’ may be replaced with the audio processing system of fig.1 implementing one or more of the audio processing techniques detailed in the above.

[0141] Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the disclosure discussions utilizing terms such as “processing”, “computing”, “calculating”, “determining”, “analyzing” or the like, refer to the action and / or processes of a computer hardware or computing system, or similar electronic computing devices, that manipulate and / or transform data represented as physical, such as electronic, quantities into other data similarly represented as physical quantities.

[0142] It should be appreciated that in the above description of exemplary embodiments of the invention, various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention. Furthermore, while some embodiments described herein include some but not other features included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention, and form different embodiments, as would be understood by those skilled in the art. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0143] Furthermore, some of the embodiments are described herein as a method or combination of elements of a method that can be implemented by a processor of a computer system or by other means of carrying out the function. Thus, a processor with instructions for carrying out such a method or element of a method forms a means for carrying out the method or element of a method. Note that when the method includes several elements, e.g., several steps, no ordering of such elements is implied, unless specifically stated. Furthermore, an element described herein of an apparatus embodiment is an example of a means for carrying out the function performed by the element for the purpose of carrying out the embodiments of the invention. In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0144] Thus, while there has been described specific embodiments of the invention, those skilled in the art will recognize that other and further modifications may be made thereto withoutdeparting from the spirit of the invention, and it is intended to claim all such changes and modifications as falling within the scope of the invention.

[0145] Various aspects of the present invention may be appreciated from the following Enumerated Example Embodiments (EEEs):

[0146] EEE 1. A method for processing audio comprising: obtaining an input audio signal comprising two channels, the input audio signal comprising a plurality of consecutive segments; determining, for each segment of the input audio signal, a segment specific source activity metric indicating whether a predetermined target audio source is active in the segment; for each segment: extracting, from the input audio signal, a side audio signal; extracting a difference audio signal based on a difference between the side audio signal and the input audio signal; extracting, from the difference audio signal, a source suppressed difference audio signal by processing the difference audio signal with a source separator; determining an inactive energy level based on the spectral energy of the difference audio signal for at least one segment associated with a segment specific activity metric indicating that the target audio source is inactive; determining an active energy level based on the spectral energy of the source suppressed difference audio signal for at least one segment associated with a segment specific activity metric indicating that the target audio source is active; applying a gain to at least one of the side audio signal and the source suppressed difference audio signal, based on the difference between the inactive and active energy level for the at least one segment associated with a segment specific activity metric indicating that the target audio source is active; and combining the source suppressed difference audio signal with the side audio signal, wherein at least one has the gain applied, to form an output audio signal in which the target audio source is suppressed.

[0147] EEE 2. The method of EEE 1, further comprising: determining the inactive energy level as a mean spectral energy of the difference audio signal for at least two segments associated with a segment specific activity metric indicating that the target audio source is inactive.

[0148] EEE 3. The method of EEE 1 or EEE 2, further comprisingdetermining the active energy level as a mean spectral energy of the source suppressed audio signal for at least two segments associated with a segment specific activity metric indicating that the target audio source is active.

[0149] EEE 4. The method according to any of the preceding EEEs, wherein the output audio signal comprises, for each segment, a plurality of frequency bands and wherein each frequency band is associated with a panning parameter, the method further comprising: scaling the panning parameter of each frequency band and segment with a panning scaling function, the panning scaling function being based on a deviation between the panning parameter and center panning, to obtain an adjusted panning parameter with greater deviation from center panning compared to the panning parameter; and forming a modified output audio signal using the adjusted panning parameter.

[0150] EEE 5. The method according to EEE 4, further comprising: smoothing the adjusted panning parameter of each frequency band and segment across time.

[0151] EEE 6. The method according to any of the preceding EEEs, wherein the output audio signal comprises, for each segment, a plurality of frequency bands and wherein each frequency band is associated with a phase parameter, the method further comprising: determining a reference phase for each frequency band and segment; scaling the phase parameter of each frequency band and segment with a phase scaling function, the phase scaling function being based on a deviation between the phase parameter and the reference phase, to obtain an adjusted phase parameter with greater deviation from reference phase compared to the phase parameter; and forming a modified output audio signal using the adjusted phase parameter.

[0152] EEE 7. The method according to EEE 6, further comprising: smoothing the adjusted phase parameter of each frequency band and segment across time.

[0153] EEE 8. The method according to EEE 6 or EEE 7 when depending on EEE 4, wherein determining the reference phase for each frequency band and segment comprises: scaling a phase parameter of a corresponding frequency band of one channel of the input audio signal with the adjusted panning parameter.

[0154] EEE 9. A method for processing audio comprising: obtaining an input audio signal comprising two channels, the input audio signal comprising a plurality of consecutive segments;for each segment: extracting, from the input audio signal, a side audio signal; extracting a difference audio signal based on a difference between the side audio signal and the input audio signal; extracting, from the difference audio signal, a source suppressed difference audio signal by processing the difference audio signal with a source separator; combining the source suppressed difference audio signal with the side audio signal to form an output audio signal in which the target audio source is suppressed; determining, for a plurality of frequency bands of each segment of the output audio signal a panning parameter and / or a phase parameter, the method further comprising: scaling the panning parameter of each frequency band and segment with a scaling panning scaling function, the panning scaling function being based on a deviation between the panning parameter and center panning, to obtain an adjusted panning parameter with greater deviation from center panning compared to the panning parameter; and / or determining a reference phase for each frequency band and segment and scaling the phase parameter of each frequency band and segment with a phase scaling function, the phase scaling function being based on a deviation between the phase parameter and the reference phase, to obtain an adjusted phase parameter with greater deviation from reference phase compared to the phase parameter; forming a modified output audio signal using the adjusted panning parameter and / or adjusted panning parameter.

[0155] EEE 10. The method according to EEE 9, further comprising smoothing the adjusted panning parameter of each frequency band and segment across time.

[0156] EEE 11. The method according to EEE 9 or EEE 10, further comprising smoothing the adjusted phase parameter of each frequency band and segment across time.

[0157] EEE 12. The method according to any of EEEs 9 - 11, wherein determining the reference phase for each frequency band and segment comprises: scaling a phase parameter of a corresponding frequency band of one channel of the input audio signal with the adjusted panning parameter.

[0158] EEE 13. The method according to any of the preceding EEEs, wherein extracting a source suppressed difference audio signal by processing the difference audio signal with a source separator comprises: - using a source separator configured to determine to what extent the segment comprises audio content associated with the target audio source or residual audio content, and- forming the source suppressed difference audio signal based on the extent to which each respective segment contains audio content associated with the target audio source.

[0159] EEE 14. The method according to EEE 13, wherein the source separator is configured to determine, for each segment, a gain mask for removing the target audio source from the difference audio signal.

[0160] EEE 15. The method according EEE 14, further comprising: smoothing the gain mask over at least two segments to form a smoothed gain mask; and applying the smoothed gain mask to the difference audio signal to form the source suppressed difference audio signal.

[0161] EEE 16. The method according to EEE 14 or EEE 15, wherein each gain mask comprises a value for each segment, the value being in a range of [g1, g2], wherein g1 and g2 are positive values greater than or equal to zero, wherein g1indicates that a corresponding segment contains audio content associated with the target source and g2 indicates that a corresponding segment contains residual audio content, the method further comprising: scaling each value of the gain mask with a positive scaling factor smaller than one and / or decreasing each value of the gain mask by subtracting a positive adjustment value greater than zero to form a modified gain mask; and applying the modified gain mask to the difference audio signal to form the source suppressed difference audio signal.

[0162] EEE 17. The method according to EEE 16, wherein each gain mask comprises a value for each of a plurality of frequency bands of each segment.

[0163] EEE 18. The method according to EEE 16 or EEE 17, wherein g1 indicates that a corresponding segment or frequency band contains audio content dominated by the target source and g2 indicates that a corresponding segment or frequency band contains audio content dominated by the residual audio content.

[0164] EEE 18. The method according to any of the preceding EEEs, further comprising: determining, for a specific segment associated with a segment specific source activity metric indicating that the target audio source is inactive, a processing application coefficient by smoothing the segment specific source activity metric across a group of segments associated with the specific segment, wherein the group of segments comprises a segment associated with a segment specific source activity metric indicating that the target audio source is inactive and a segment associated with a segment specific source activity metric indicating that the target audio source is active;forming, for the specific segment, a modified difference audio signal as a weighed sum of the difference audio signal and the source suppressed difference audio signal based on the processing application coefficient; and combining, for the specific segment, the modified difference audio signal with the side audio signal to form an output audio signal in which the target audio source is suppressed.

[0165] EEE 18. The method according to EEE 17, wherein smoothing the segment specific source activity metric to determine the processing application coefficient comprises limiting the maximum difference of the processing application coefficient between two consecutive segments.

[0166] EEE 19. The method according to EEE 17, wherein smoothing the segment specific source activity metric to the determine the processing application coefficient comprises: smoothing the segment specific source activity metric across Nsmooth segments, wherein Nsmoothis greater than or equal to two and wherein the total time duration of Nsmoothsegments is at least 1 second, preferably at least 2 seconds, or most preferably at least 2.5 seconds.

[0167] EEE 20. The method according to EEE 17 - 19, wherein the source specific activity metric for each segment assumes a value of either α1or α2, where α2indicates that the target source is active and α1 indicates that the target source is inactive, and wherein smoothing the segment specific source activity metric to the determine the processing application coefficient comprises: determining, that for N number of consecutive segments the segment specific source activity metric is α1, wherein N is between one and a predetermined maximum number Nmax; and using the source suppressed difference audio signal as the modified difference audio signal for each of the N number of consecutive segments.

[0168] EEE 21. The method according to EEE 20, wherein Nmaxis selected such that the total time duration of Nmax segments is less than or equal to 1 second, preferably less than or equal to 2 seconds, and most preferably less than or equal to 3 seconds, wherein T is the time duration of a segment.

[0169] EEE 22. The method according to any of the preceding EEEs, further comprising: obtaining an original audio signal; dividing the original audio signal into two separate signals based on frequency content, the two separate signals being the input audio signal and an auxiliary audio signal; and combining the auxiliary audio signal with the output audio signal.

[0170] EEE 23. The method according to any of the preceding EEEs, wherein the target audio source is selected from a group comprising speech, music, and the sound of a specific musical instrument.

[0171] EEE 24. The method according to any of the preceding EEEs, wherein extracting the side audio signal and difference audio signal comprises: extracting the side audio signal based on a difference between the two channels in the input audio signal; determining a panning parameter for the input audio signal; converting the side audio signal to a stereo side audio signal using the panning parameter of the input audio signal; and determining the difference audio signal as the difference between the stereo side audio signal and the two channel input audio signal.

[0172] EEE 25. An apparatus comprising: one or more processors, and memory storing instructions that when executed by the one or more processors, cause the one or more processors to perform the method of any of the preceding EEEs.

[0173] EEE 26. A non-transitory, computer readable storage medium having stored thereon instructions, that when executed by one or more processors, cause the one or more processors to perform the method of any of EEEs 1-24.

Claims

CLAIMS 1. A method for processing audio comprising: obtaining an input audio signal comprising two channels, the input audio signal comprising a plurality of consecutive segments; determining, for each segment of the input audio signal, a segment specific source activity metric indicating whether a predetermined target audio source is active in the segment; determining, for a specific segment associated with a segment specific source activity metric indicating that the target audio source is inactive, a processing application coefficient by smoothing the segment specific source activity metric across a group of segments associated with the specific segment, wherein the group of segments comprises a segment associated with a segment specific source activity metric indicating that the target audio source is inactive and a segment associated with a segment specific source activity metric indicating that the target audio source is active; for the specific segment: extracting, from the input audio signal, a side audio signal; extracting a difference audio signal based on a difference between the side audio signal and the input audio signal; extracting, from the difference audio signal, a source suppressed difference audio signal by: - using a source separator configured to determine to what extent the specific segment contains audio content associated with the target audio source or residual audio content, and - forming the source suppressed difference audio signal based on the extent to which the specific segment contains audio content associated with the target audio source; forming a modified difference audio signal as a weighted sum of the difference audio signal and the source suppressed difference audio signal based on the processing application coefficient; and combining the modified difference audio signal with the side audio signal to form an output audio signal in which the target audio source is suppressed.

2. The method according to claim 1, wherein the source separator is configured to determine to what extent each of a plurality of frequency bands of the specific segment contains audio content associated with the target audio source or residual audio content, and wherein the source suppressed difference audio signal is formed as a combination of all frequency bands associated with residual audio content based on the extent to which each respective frequency band contains audio content associated with the target audio source.

3. The method according to claim 1 or claim 2, wherein smoothing the segment specific source activity metric to determine the processing application coefficient comprises limiting the maximum difference of the processing application coefficient between two consecutive segments.

4. The method according to any of the preceding claims wherein smoothing the segment specific source activity metric to the determine the processing application coefficient comprises: smoothing the segment specific source activity metric across Nsmooth segments, wherein Nsmoothis greater than or equal to two and wherein the total time duration of Nsmoothsegments is at least 1 second, preferably at least 2 seconds, or most preferably at least 2.5 seconds.

5. The method according to any of the preceding claims, wherein the source specific activity metric for each segment assumes a value of either α1or α2, where α2indicates that the target source is active and α1 indicates that the target source is inactive, and wherein smoothing the segment specific source activity metric to the determine the processing application coefficient comprises: determining, that for N number of consecutive segments the segment specific source activity metric is α1, wherein N is between one and a predetermined maximum number Nmax; and using the source suppressed difference audio signal as the modified difference audio signal for each of the N number of consecutive segments.

6. The method according to claim 5, wherein Nmax is selected such that the total time duration of Nmaxsegments is less than or equal to 1 second, preferably less than or equal to 2seconds, and most preferably less than or equal to 3 seconds, wherein T is the time duration of a segment.

7. The method according to any of the preceding claims, wherein the source separator is configured to determine, for each segment, a gain mask for removing the target audio source from the difference audio signal.

8. The method according claim 7, further comprising smoothing the gain mask over at least two segments to form a smoothed gain mask; and applying the smoothed gain mask to the difference audio signal to form the source suppressed difference audio signal.

9. The method according to claim 7 or claim 8, wherein each gain mask comprises a value for each segment, the value being in a range of [g1, g2], wherein g1and g2are positive values greater than or equal to zero, wherein g1 indicates that a corresponding segment contains audio content associated with the target source and g2indicates that a corresponding segment contains residual audio content, the method further comprising: scaling each value of the gain mask with a positive scaling factor smaller than one and / or decreasing each value of the gain mask by subtracting a positive adjustment value greater than zero to form a modified gain mask; and applying the modified gain mask to the difference audio signal to form the source suppressed difference audio signal.

10. The method according to claim 9, wherein each gain mask comprises a value for each of a plurality of frequency bands of each segment.

11. The method according to claim 9 or claim 10, wherein g1indicates that a corresponding segment or frequency band contains audio content dominated by the target source and g2indicates that a corresponding segment or frequency band contains audio content dominated by the residual audio content.

12. The method according to any of the preceding claims, further comprising:determining an inactive energy level based on the spectral energy of the difference audio signal for at least one segment associated with a segment specific activity metric indicating that the target audio source is inactive; determining an active energy level based on the spectral energy of the source suppressed audio signal for at least one segment associated with a segment specific activity metric indicating that the target audio source is active; applying a gain to at least one of the side audio signal and the source suppressed difference audio signal, based on the difference between the inactive and active energy level for the at least one segment associated with a segment specific activity metric indicating that the target audio source is active.

13. The method according to claim 12, further comprising: determining the inactive energy level as a mean spectral energy of the difference audio signal for at least two segments associated with a segment specific activity metric indicating that the target audio source is inactive.

14. The method according to claim 12 or claim 13, further comprising: determining the active energy level as a mean spectral energy of the source suppressed audio signal for at least two segments associated with a segment specific activity metric indicating that the target audio source is active.

15. The method according to any of the preceding claims, wherein the output audio signal comprises, for each segment, a plurality of frequency bands and wherein each frequency band is associated with a panning parameter, the method further comprising: scaling the panning parameter of each frequency band and segment with a panning scaling function, the panning scaling function being based on a deviation between the panning parameter and center panning, to obtain an adjusted panning parameter with greater deviation from center panning compared to the panning parameter; and forming a modified output audio signal using the adjusted panning parameter.

16. The method according to claim 15, further comprising smoothing the adjusted panning parameter of each frequency band and segment across time.

17. The method according to any of the preceding claims, wherein the output audio signal comprises, for each segment, a plurality of frequency bands and wherein each frequency band is associated with a phase parameter, the method further comprising: determining a reference phase for each frequency band and segment; scaling the phase parameter of each frequency band and segment with a phase scaling function, the phase scaling function being based on a deviation between the phase parameter and the reference phase, to obtain an adjusted phase parameter with greater deviation from reference phase compared to the phase parameter; and forming a modified output audio signal using the adjusted phase parameter.

18. The method according to claim 17, further comprising smoothing the adjusted phase parameter of each frequency band and segment across time.

19. The method according to claim 17 or claim 18 when depending on claim 15, wherein determining the reference phase for each frequency band and segment comprises: scaling a phase parameter of a corresponding frequency band of one channel of the input audio signal with the adjusted panning parameter.

20. The method according to any of the preceding claims, further comprising: obtaining an original audio signal; dividing the original audio signal into two separate signals based on frequency content, the two separate signals being the input audio signal and an auxiliary audio signal; and combining the auxiliary audio signal with the output audio signal.

21. The method according to any of the preceding claims, wherein the target audio source is selected from a group comprising speech, music, and a sound of a specific musical instrument.

22. The method according to any of the preceding claims, wherein extracting the side audio signal and difference audio signal comprises: extracting the side audio signal based on a difference between the two channels in the input audio signal; determining a panning parameter for the input audio signal;converting the side audio signal to a stereo side audio signal using the panning parameter of the input audio signal; and determining the difference audio signal as the difference between the stereo side audio signal and the two channel input audio signal.

23. An apparatus comprising: one or more processors, and memory storing instructions that when executed by the one or more processors, cause the one or more processors to perform the method of any of the preceding claims.

24. A non-transitory, computer readable storage medium having stored thereon instructions, that when executed by one or more processors, cause the one or more processors to perform the method of any of claims 1-22.

Images