Sound field related rendering

Abstract

Apparatus and method for sound field related audio representation and rendering. [Solution] 1. An apparatus comprising: means for obtaining a defocus direction; processing a spatial audio signal representing an audio scene to generate a processed spatial audio signal representing a modified audio scene based on the defocus direction to control de-emphasis in a portion of the portion of the spatial audio signal in the defocus direction relative to at least a portion of another portion of the spatial audio signal; and outputting the processed spatial audio signal based on the defocus direction, wherein the modified audio scene based on the defocus direction at least partially enables de-emphasis of a portion of the spatial audio signal in the defocus direction relative to at least a portion of the other portions of the spatial audio signal.

Description

[Technical Field]

[0001] This application relates to an apparatus and method for audio representation and rendering related to a sound field, although not limited to audio representation for an audio decoder. [Background technology]

[0002] Spatial audio playback is known for presenting media with multiple viewing directions. Examples of this playback include (at least) the visual content of such media, including playback on a head-mounted display (or a phone in a head-mount) with head direction tracking, or on a phone screen without a head-mount where the view direction can be tracked by changing the phone's position / orientation, or by any user interface gesture, or on a surrounding screen.

[0003] Videos related to "media with multiple viewing directions" can be, for example, 360-degree video, 180-degree video, or other video with a substantially wider viewing angle than traditional video. Traditional video typically refers to video content that is viewed as a whole on a screen without the option (or any particular need) to change viewing direction.

[0004] Audio associated with videos with multiple gaze directions can be presented over headphones or using a surround loudspeaker setup, where gaze direction is tracked and affects spatial audio playback.

[0005] Spatial audio associated with video with multiple viewing directions can come from a microphone array (e.g., an array attached to a VR camera like an OZO, or a handheld mobile device) or spatial audio capture from other sources, such as a studio mix. The audio content may be a mixture of several content types, such as microphone-captured sound and an added commentary track.

[0006] Spatial audio associated with video with multiple viewing directions can take various forms, such as ambisonic signals (of any order) composed of spherical harmonic audio signal components. Spherical harmonic functions can be thought of as a set of spatially selective beam signals. Ambisonics is currently used, for example, in YouTube's 360VR video service. The advantage of ambisonics is its simple, well-defined signal representation. Surround loudspeaker signals, such as 5.1, are currently used to convey typical cinematic spatial audio. The advantage of surround loudspeaker signals is their simplicity and legacy compatibility. Some audio formats similar to surround loudspeaker signal formats include audio objects, which can be thought of as audio channels with time-varying positions. Position can signal both the direction and distance of the audio object, or orientation, i.e., parametric spatial audio, such as a two-channel audio signal and associated spatial metadata in perceptually relevant frequency bands. Several state-of-the-art audio coding and spatial audio capture methods apply such signal representations. The spatial metadata essentially determines how the audio signal should be spatially reproduced at the receiver (e.g., in which directions at different frequencies). The advantages of parametric spatial audio are its versatility, quality, and ability to use low bit rates for encoding. Summary of the Invention

[0007] According to a first aspect, there is provided an apparatus comprising means configured to obtain a defocus direction; process a spatial audio signal representing an audio scene to generate a processed spatial audio signal representing a modified audio scene based on the defocus direction to at least partially control a relative de-emphasis of a portion of the spatial audio signal in the defocus direction relative to at least some other portions of the spatial audio signal; and output the processed spatial audio signal relative to at least some other portions of the spatial audio signal, wherein the modified audio scene based on the defocus direction at least partially enables de-emphasis of a portion of the spatial audio signal in the defocus direction relative to at least some other portions of the spatial audio signal.

[0008] The means may be further configured to obtain a defocus amount, and the means configured to process the spatial audio signal may be configured to at least partially control a relative de-emphasis of a portion of the spatial audio signal in the defocus direction relative to another portion of at least a portion of the spatial audio signal according to the defocus amount.

[0009] The means configured to process the spatial audio signal may be configured to at least partially reduce emphasis in portions of the spatial audio signal in a defocus direction relative to emphasis in at least partially other portions of the spatial audio signal, and at least partially increase emphasis in other portions of the spatial audio signal relative to portions of the spatial audio signal in the defocus direction.

[0010] The means configured to process the spatial audio signal may be configured to at least one of: reduce a sound level in a portion of the spatial audio signal in accordance with an amount of defocus in at least a portion of the spatial audio signal relative to another portion of the spatial audio signal; and increase a sound level in a portion of the spatial audio signal in accordance with the amount of defocus in relation to another portion of the spatial audio signal.

[0011] The means may be further configured to obtain a defocus shape, and the means configured to process the spatial audio signal may be configured to control relative de-emphasis in at least a portion of the spatial audio signal in the defocus direction and within the defocus shape relative to at least partially other portions of the spatial audio signal.

[0012] The means configured to process the spatial audio signal may be configured to at least one of: reduce emphasis in at least a portion of the spatial audio signal in portions from within a defocus direction and defocus shape of the spatial audio signal relative to other portions in at least a portion of the spatial audio signal; and increase emphasis in at least a portion of the other portions of the spatial audio signal relative to portions from within a defocus direction and defocus shape of the spatial audio signal.

[0013] The means configured to process the spatial audio signal may be configured to at least one of: reducing a sound level in at least a portion of the spatial audio signal, in a portion of the spatial audio signal in a defocus direction and from within the defocus shape, in accordance with a defocus amount relative to other portions of the at least a portion of the spatial audio signal; and increasing a sound level in the portion of the spatial audio signal, in a portion of the spatial audio signal, in a defocus direction and from the defocus shape according to the defocus amount, in other portions of the spatial audio signal.

[0014] The means can be configured to obtain playback control information for controlling at least one aspect of outputting the processed spatial audio signal, and the means configured to output the processed spatial audio signal can be configured to perform one of: processing the processed spatial audio signal representing the audio scene modified based on the defocus direction to generate the output spatial audio signal in accordance with the playback control information; processing the spatial audio signal in accordance with the playback control information before the means configured to process the spatial audio signal representing the audio scene to generate the processed spatial audio signal representing the audio scene modified based on the defocus direction; and outputting the processed spatial audio signal as the output spatial audio signal.

[0015] The spatial audio signal and the processed spatial audio signal may comprise respective Ambisonic signals, and the means configured to process the spatial audio signal into the processed spatial audio signal may be configured to extract, for one or more frequency subbands, from the spatial audio signal a single-channel target audio signal representing sound components arriving from the focus direction to generate a focused spatial audio signal, where the focused audio signal is located at a spatial position defined by the defocus direction, and to generate the processed spatial audio signal as a linear combination of the focused spatial audio signal subtracted from the spatial audio signal, and at least one of the focused spatial audio signal and the spatial audio signal is scaled by a respective scaling factor derived based on the amount of defocus to reduce the relative level of sound in the defocus direction.

[0016] The means configured to extract the single-channel target audio signal may be configured to apply a beamformer to derive a beamformed signal from the spatial audio signal, representing sound components arriving from a defocused direction, and to apply a post-filter to derive a processed audio signal based on the beamformed signal, thereby adjusting the spectrum of the beamformed signal to approximate the spectrum of sound arriving from a defocused direction.

[0017] The spatial audio signal and the processed spatial audio signal may include respective first-order Ambisonic signals.

[0018] The spatial audio signal and the processed spatial audio signal may include respective parametric spatial audio signals, the parametric spatial audio signals may include one or more audio channels and spatial metadata, the spatial metadata may include respective directional indications and energy ratio parameters for a plurality of frequency subbands, and the means configured to process the spatial audio signal to generate the processed spatial audio signal may include: calculating, for one or more frequency subbands, respective angular differences between a defocus direction and a direction indicated for each frequency subband of the spatial audio signal; deriving respective gain values ​​for the one or more frequency subbands based on the calculated angular differences for each frequency subband by using a predefined function of the angular differences and a scaling factor derived based on the amount of defocus; and calculating, for the one or more frequency subbands of the processed spatial audio signal, respective updated ambient energy values ​​based on the energy ratio parameters for each frequency subband of the spatial audio signal and the scaling factor for each frequency subband of the spatial audio signal. a directional indication of the spatial audio signal, a directional indication of the spatial audio signal, a modified energy ratio parameter, and a spectral adjustment factor for each of the one or more frequency subbands of the processed spatial audio signal based on the updated directional energy divided by the sum of the updated direct and ambient energies; and a directional indication of the spatial audio signal, a directional indication of the spatial audio signal, a modified energy ratio parameter, and a spectral adjustment factor for each of the one or more frequency subbands of the processed spatial audio signal, and a directional indication of the spatial audio signal, the modified energy ratio parameter, and the spectral adjustment factor.Each of the plurality of frequency subbands may include a respective parametric spatial audio signal, the parametric spatial audio signal may include one or more audio channels and spatial metadata, the spatial metadata may include respective directional indications and energy ratio parameters for a plurality of frequency subbands, and the means configured to process the spatial audio signal to generate a processed spatial audio signal may include: calculating, for one or more frequency subbands, respective angular differences between a defocus direction and a direction indicated for each frequency subband of the spatial audio signal; deriving respective gain values ​​for the one or more frequency subbands based on the calculated angular differences for each frequency subband by using a predefined function of the angular differences and a scaling factor derived based on the defocus amount; calculating, for one or more frequency subbands of the processed spatial audio signal, respective updated directional energy values ​​based on the energy ratio parameters for the respective frequency subbands; calculating updated ambient energy values ​​for each of the one or more frequency subbands of the processed spatial audio signal based on an energy ratio parameter and a scaling factor for each frequency subband of the spatial audio signal; calculating modified energy ratio parameters for each of the one or more frequency subbands of the processed spatial audio signal based on the updated directional energy divided by the sum of the updated direct and ambient energies; calculating respective spectral adjustment coefficients for the one or more frequency subbands of the processed spatial audio signal based on the sum of the updated direct and ambient energies; and deriving one or more extended audio channels by multiplying, in the one or more frequency subbands, a respective one of the one or more audio channels of the spatial audio signal by the derived spectral adjustment coefficient for the respective frequency subband;and constructing a processed spatial audio signal.

[0019] The spatial audio signal and the processed spatial audio signal may include respective multi-channel loudspeaker signals according to a first predetermined loudspeaker configuration, and the means configured to process the spatial audio signal to generate the processed spatial audio signal may be configured to: calculate respective angular differences between the defocus directions and the loudspeaker directions indicated for each channel of the spatial audio signal; derive respective gain values ​​for each channel of the spatial audio signal based on the angular differences calculated for each channel by using a predetermined function of the angular difference and a scaling factor derived based on the amount of defocus; derive one or more modified audio channels by multiplying each channel of the spatial audio signal by the derived gain value for the respective channel; and provide the modified audio channels as the processed spatial audio signal. The predetermined function of the angular difference may result in gain values ​​that decrease as the angular difference value decreases and that increase as the angular difference value increases. The means configured to process the spatial audio signal may be configured to at least one of: reduce emphasis in at least a portion of the spatial audio signal in portions from within a defocus direction and defocus shape of the spatial audio signal relative to other portions in at least a portion of the spatial audio signal; and increase emphasis in at least a portion of the other portions of the spatial audio signal relative to portions from within a defocus direction and defocus shape of the spatial audio signal.

[0020] The means configured to process the spatial audio signal may be configured to at least one of: reducing a sound level in at least a portion of the spatial audio signal, in a portion of the spatial audio signal in a defocus direction and from within the defocus shape, in accordance with a defocus amount relative to other portions of the at least a portion of the spatial audio signal; and increasing a sound level in the portion of the spatial audio signal, in a portion of the spatial audio signal, in a defocus direction and from the defocus shape according to the defocus amount, in other portions of the spatial audio signal.

[0021] The means can be configured to obtain playback control information for controlling at least one aspect of outputting the processed spatial audio signal, and the means configured to output the processed spatial audio signal can be configured to perform one of: processing the processed spatial audio signal representing the audio scene modified based on the defocus direction to generate the output spatial audio signal in accordance with the playback control information; processing the spatial audio signal in accordance with the playback control information before the means configured to process the spatial audio signal representing the audio scene to generate the processed spatial audio signal representing the audio scene modified based on the defocus direction; and outputting the processed spatial audio signal as the output spatial audio signal.

[0022] The spatial audio signal and the processed spatial audio signal may comprise respective Ambisonic signals, and the means configured to process the spatial audio signal into the processed spatial audio signal may be configured to extract, for one or more frequency subbands, from the spatial audio signal a single-channel target audio signal representing sound components arriving from the focus direction to generate a focused spatial audio signal, where the focused audio signal is located at a spatial position defined by the defocus direction, and to generate the processed spatial audio signal as a linear combination of the focused spatial audio signal subtracted from the spatial audio signal, and at least one of the focused spatial audio signal and the spatial audio signal is scaled by a respective scaling factor derived based on the amount of defocus to reduce the relative level of sound in the defocus direction.

[0023] The means configured to extract the single-channel target audio signal may be configured to apply a beamformer to derive a beamformed signal from the spatial audio signal, representing sound components arriving from a defocused direction, and to apply a post-filter to derive a processed audio signal based on the beamformed signal, thereby adjusting the spectrum of the beamformed signal to approximate the spectrum of sound arriving from a defocused direction.

[0024] The spatial audio signal and the processed spatial audio signal may include respective first-order Ambisonic signals.

[0025] The spatial audio signal and the processed spatial audio signal may include respective parametric spatial audio signals, the parametric spatial audio signals may include one or more audio channels and spatial metadata, the spatial metadata may include respective directional indications and energy ratio parameters for a plurality of frequency subbands, and the means configured to process the spatial audio signal to generate the processed spatial audio signal may include: calculating, for one or more frequency subbands, respective angular differences between a defocus direction and a direction indicated for each frequency subband of the spatial audio signal; deriving respective gain values ​​for the one or more frequency subbands based on the calculated angular differences for each frequency subband by using a predefined function of the angular differences and a scaling factor derived based on the amount of defocus; and calculating, for the one or more frequency subbands of the processed spatial audio signal, respective updated ambient energy values ​​based on the energy ratio parameters for each frequency subband of the spatial audio signal and the scaling factor for each frequency subband of the spatial audio signal. a directional indication of the spatial audio signal, a directional indication of the spatial audio signal, a modified energy ratio parameter, and a spectral adjustment factor for each of the one or more frequency subbands of the processed spatial audio signal based on the updated directional energy divided by the sum of the updated direct and ambient energies; a modified energy ratio parameter for each of the one or more frequency subbands of the processed spatial audio signal based on the updated directional energy divided by the sum of the updated direct and ambient energies; and a processed spatial audio signal comprising one or more audio channels of the spatial audio signal, a directional indication of the spatial audio signal, the modified energy ratio parameter, and the spectral adjustment factor, and calculating each updated directional energy value.Each of the plurality of frequency subbands may include a respective parametric spatial audio signal, the parametric spatial audio signal may include one or more audio channels and spatial metadata, the spatial metadata may include respective directional indications and energy ratio parameters for a plurality of frequency subbands, and the means configured to process the spatial audio signal to generate a processed spatial audio signal may include: calculating, for one or more frequency subbands, respective angular differences between a defocus direction and a direction indicated for each frequency subband of the spatial audio signal; deriving respective gain values ​​for the one or more frequency subbands based on the calculated angular differences for each frequency subband by using a predefined function of the angular differences and a scaling factor derived based on the defocus amount; calculating, for one or more frequency subbands of the processed spatial audio signal, respective updated directional energy values ​​based on the energy ratio parameters for the respective frequency subbands; calculating updated ambient energy values ​​for each of the one or more frequency subbands of the processed spatial audio signal based on an energy ratio parameter and a scaling factor for each frequency subband of the spatial audio signal; calculating modified energy ratio parameters for each of the one or more frequency subbands of the processed spatial audio signal based on the updated directional energy divided by the sum of the updated direct and ambient energies; calculating respective spectral adjustment coefficients for the one or more frequency subbands of the processed spatial audio signal based on the sum of the updated direct and ambient energies; and deriving one or more extended audio channels by multiplying, in the one or more frequency subbands, a respective one of the one or more audio channels of the spatial audio signal by the derived spectral adjustment coefficient for the respective frequency subband;and constructing a processed spatial audio signal.

[0026] The spatial audio signal and the processed spatial audio signal may include respective multi-channel loudspeaker signals according to a first predetermined loudspeaker configuration, and the means configured to process the spatial audio signal to generate the processed spatial audio signal may be configured to: calculate respective angular differences between the defocus directions and the loudspeaker directions indicated for each channel of the spatial audio signal; derive respective gain values ​​for each channel of the spatial audio signal based on the angular differences calculated for each channel by using a predetermined function of the angular difference and a scaling factor derived based on the amount of defocus; derive one or more modified audio channels by multiplying each channel of the spatial audio signal by the derived gain value for the respective channel; and provide the modified audio channels as the processed spatial audio signal. The predetermined function of the angular difference may result in gain values ​​that decrease as the angular difference value decreases and that increase as the angular difference value increases.

[0027] the processed spatial audio signal may comprise an Ambisonic signal, and the output spatial audio signal may comprise a two-channel binaural signal, and the playback control information may include an indication of a playback direction defining a listening direction with respect to the audio scene, and the means configured to process the processed spatial audio signal representing the modified audio scene based on a defocus direction to generate the output spatial audio signal in accordance with the playback control information may be configured to: generate a rotation matrix according to the indicated playback direction; multiply the channels of the processed spatial audio signal by the rotation matrix to derive the rotated spatial audio signal; filter the channels of the rotated spatial audio signal using a predetermined set of filter pairs generated based on a predefined dataset of finite impulse responses (FIRs), head-related transfer functions (HRTFs), or head-related impulse responses (HRIRs); and generate the left and right channels of the binaural signal as a sum of the filtered channels of the rotated spatial audio signal derived for the left and right channels, respectively.

[0028] The output spatial audio signal may comprise a two-channel binaural audio signal, and the playback control information may comprise an indication of a playback direction defining a listening direction relative to the audio scene, and the means configured to process the processed spatial audio signal representing the audio scene modified based on the defocus direction to generate the output spatial audio signal in accordance with the playback control information may be configured to derive one or more enhanced audio channels by multiplying, in said one or more frequency sub-bands, a respective frequency band of each one of the one or more audio channels of the processed spatial audio signal by a spectral adjustment factor received for the respective frequency sub-band, and convert the one or more enhanced audio channels into a two-channel binaural audio signal in accordance with the indicated playback direction.

[0029] The output spatial audio signal may comprise a two-channel binaural audio signal, the playback control information may comprise an indication of a playback direction defining a listening direction relative to the audio scene, and the means configured to process the processed spatial audio signal representing the audio scene modified based on the defocus direction to generate the output spatial audio signal in accordance with the playback control information may be configured to convert one or more enhanced audio channels into a two-channel binaural audio signal in accordance with the indicated playback direction.

[0030] The output spatial audio signal may comprise a two-channel binaural signal, the playback control information may comprise an indication of a playback direction defining a listening direction relative to the audio scene, and the means configured to process the processed spatial audio signal representing the audio scene modified based on the defocus direction to generate the output spatial audio signal in accordance with the playback control information may be configured to select a set of head related transfer functions (HRTFs) in response to the indicated playback direction and to convert the channels of the processed spatial audio signal into a two-channel binaural signal carrying the audio scene rotated using the selected set of HRTFs.

[0031] The playback control information may include an indication of a second predetermined loudspeaker configuration, and the output spatial audio signal may include a multi-channel loudspeaker signal with the second predetermined loudspeaker configuration, and the means configured to process the processed spatial audio signal representing the audio scene modified based on the defocus direction to generate the output spatial audio signal in accordance with the playback control information may be configured to derive a transformation matrix including amplitude panning gains that provide a mapping from the first predetermined loudspeaker configuration to the second predetermined loudspeaker configuration, thereby deriving the channels of the output spatial audio signal based on the channels of the processed spatial audio signal using amplitude panning, and multiplying the channels of the output spatial audio signal by the channels of the processed spatial audio signal using the transformation matrix.

[0032] The means may be further configured to obtain a defocus input from a sensor arrangement including at least one direction sensor and at least one user input, and the defocus input may include an indication of the defocus direction based on the at least one direction sensor direction.

[0033] The defocus input may further include an indicator of the amount of defocus.

[0034] The defocus input may further include an indicator of the defocus shape.

[0035] The defocus shape can include at least one of a defocus shape width, a defocus shape height, a defocus shape radius, a defocus shape distance, a defocus shape depth, a defocus shape range, a defocus shape diameter, and a defocus shape characterizer.

[0036] The defocus direction may be an arc defined by a range of defocus directions.

[0037] According to a second aspect, there is provided a method comprising the steps of: obtaining a defocus direction; processing a spatial audio signal representing an audio scene to generate a processed spatial audio signal representing a modified audio scene based on the defocus direction so as to control, at least in part, a relative de-emphasis of the de-focus direction relative to other parts of the spatial audio signal; and outputting the processed spatial audio signal, wherein the modified audio scene based on the de-focus direction enables the de-emphasis, at least in part, of the parts of the spatial audio signal in the de-focus direction at least in part on other parts of the spatial audio signal.

[0038] The method may further include obtaining a defocus amount, and processing the spatial audio signal may include controlling de-emphasis of a portion of the spatial audio signal in the defocus direction relative to another portion of the spatial audio signal at least partially according to the defocus amount.

[0039] Processing the spatial audio signal may include at least one of decreasing emphasis of portions of the spatial audio signal, at least partially in a defocus direction relative to other portions of the spatial audio signal, and increasing emphasis of other portions of the spatial audio signal, at least partially in a defocus direction relative to portions of the spatial audio signal in the defocus direction.

[0040] Processing the spatial audio signal may include at least one of: reducing a sound level in portions of the spatial audio signal in a defocus direction, at least in part depending on the amount of defocus relative to other portions of the spatial audio signal; and increasing a sound level in other portions of the spatial audio signal, at least in part depending on the amount of defocus relative to the portions of the spatial audio signal in the defocus direction.

[0041] The method may further include obtaining a defocus shape, and processing the spatial audio signal may include, at least in part, controlling relative de-emphasis within the defocus direction and defocus shape of the spatial audio signal relative to at least a portion of other portions of the spatial audio signal.

[0042] Processing the spatial audio signal may include at least one of at least partially decreasing emphasis on portions of the spatial audio signal in the defocus direction relative to other portions of the spatial audio signal, at least partially from within the defocus shape, and at least partially increasing emphasis on portions of the spatial audio signal in the defocus direction and within the defocus shape relative to other portions of the spatial audio signal.

[0043] Processing the spatial audio signal may include at least one of decreasing a sound level within portions of the spatial audio signal in a defocus direction from within the defocus shape, at least in part, according to an amount of defocus relative to at least some other portions of the spatial audio signal, and increasing a sound level within other portions of the spatial audio signal from the defocus shape, at least in part, for portions of the spatial audio signal in the defocus direction and according to the amount of defocus.

[0044] The method includes obtaining playback control information to control at least one aspect of outputting a processed spatial audio signal, and outputting the processed spatial audio signal may include one of the following steps: processing the processed spatial audio signal representing the modified audio scene based on the defocus direction in accordance with the playback control information to generate an output spatial audio signal; processing the spatial audio signal in accordance with the playback control information before processing the spatial audio signal representing the audio scene to generate the processed spatial audio signal representing the modified audio scene based on the defocus direction; and outputting the processed spatial audio signal as an output spatial audio signal.

[0045] The spatial audio signal and the processed spatial audio signal may include respective Ambisonic signals, and processing the spatial audio signal into the processed spatial audio signal may include extracting, for one or more frequency subbands, from the spatial audio signal a single-channel target audio signal representing sound components arriving from the focus direction, generating the focused spatial audio signal where the focused audio signal is located at a spatial position defined by the defocus direction, and generating the processed spatial audio signal as a linear combination of the focused spatial audio signal subtracted from the spatial audio signal. At least one of the focused spatial audio signal and the spatial audio signal is scaled by a respective scaling factor derived based on the amount of defocus to reduce the relative level of sound in the defocus direction.

[0046] The step of extracting the single-channel target audio signal may include applying a beamformer to derive a beamformed signal from the spatial audio signal, the beamformed signal representing sound components arriving from a defocused direction, and applying a post-filter to derive a processed audio signal based on the beamformed signal, thereby adjusting the spectrum of the beamformed signal to approximate the spectrum of sound arriving from a defocused direction.

[0047] The spatial audio signal and the processed spatial audio signal may include respective first-order Ambisonic signals.

[0048] The spatial audio signal and the processed spatial audio signal can include respective parametric spatial audio signals, where the parametric spatial audio signals can include one or more audio channels and spatial metadata, where the spatial metadata can include respective directional indications and energy ratio parameters for a plurality of frequency subbands. Processing the spatial audio signal to generate the processed spatial audio signal can include: calculating, for one or more frequency subbands, respective angular differences between a defocus direction and a direction indicated for each frequency subband of the spatial audio signal; deriving respective gain values ​​for the one or more frequency subbands based on the calculated angular differences for each frequency subband by using a scaling factor derived based on a predefined function of the angular differences and the defocus amount; calculating, for one or more frequency subbands of the processed spatial audio signal, respective updated directional energy values ​​based on the energy ratio parameters and gain values ​​for each frequency subband of the spatial audio signal; and generating, for one or more frequency bands of the processed spatial audio signal, respective updated directional energy values ​​for the spatial audio signal. calculating updated ambient energy values ​​based on the respective frequency subband energy ratio parameters and scaling factors of the processed spatial audio signal; calculating respective modified energy ratio parameters for one or more frequency subbands of the processed spatial audio signal based on the updated directional energy divided by the sum of the updated directional and ambient energies; calculating respective spectral adjustment factors for the one or more frequency subbands of the processed spatial audio signal based on the updated directional energy and the sum of the ambient energies; and configuring the processed spatial audio signal comprising the one or more audio channels of the spatial audio signal, the directional indication of the spatial audio signal, the modified energy ratio parameters, and the spectral adjustment factors.The spatial audio signal and the processed spatial audio signal can include respective parametric spatial audio signals, where the parametric spatial audio signals can include one or more audio channels and spatial metadata, where the spatial metadata can include respective directional indications and energy ratio parameters for a plurality of frequency subbands.Processing the spatial audio signal to generate a processed spatial audio signal includes calculating, for one or more frequency subbands, respective angular differences between a defocus direction and a direction indicated for each frequency subband of the spatial audio signal; deriving respective gain values ​​for the one or more frequency subbands based on the angular differences calculated for each frequency subband by using a predefined function of the angular differences and a scaling factor derived based on the amount of defocus; calculating, for one or more frequency subbands of the processed spatial audio signal, respective updated directional energy values ​​based on an energy ratio parameter and a gain value for each frequency subband of the spatial audio signal; and calculating, for one or more frequency bands of the processed spatial audio signal, respective updated ambient energy values ​​based on the energy ratio parameter and the scaling factor for each frequency subband of the spatial audio signal; The method may include calculating respective modified energy ratio parameters for one or more frequency subbands of the processed spatial audio signal based on the updated direct energy divided by a sum of the updated direct and ambient energies; calculating respective spectral adjustment factors for one or more of the processed spatial audio signal based on the sum of the updated direct and ambient energies; deriving one or more enhanced audio channels by multiplying, in the one or more frequency subbands, respective frequency bands of one or more audio channels of the spatial audio signal by the derived spectral adjustment factor for the respective frequency subband; and providing the processed spatial audio signal including the one or more enhanced audio channels, a directional indication of the spatial audio signal, and the modified energy ratio parameters.

[0049] The spatial audio signal and the processed spatial audio signal may include respective multi-channel loudspeaker signals according to a first predetermined loudspeaker configuration, and processing the spatial audio signal to generate the processed spatial audio signal may include calculating respective angular differences between defocus directions and loudspeaker directions indicated for each channel of the spatial audio signal, deriving respective gain values ​​for each channel of the spatial audio signal based on the calculated angular differences for the respective channels by using a scaling factor derived based on a predetermined function of the angular differences and the amount of defocus, deriving one or more modified audio channels by multiplying each channel of the spatial audio signal by the derived gain value for the respective channel, and providing the modified audio channels as the processed spatial audio signal.

[0050] The predetermined function of the angular difference may result in a gain value that decreases as the angular difference value decreases and increases as the angular difference value increases.

[0051] The processed spatial audio signal may include an Ambisonic signal, and the output spatial audio signal may include a two-channel binaural signal. Here, the playback control information may include an indication of a playback direction defining a listening direction with respect to the audio scene. And generating the processed spatial audio signal representing the modified spatial audio signal according to the playback control information may include generating a rotation matrix according to the indicated playback direction, multiplying the channels of the processed spatial audio signal by the rotation matrix to derive the rotated spatial audio signal, filtering the channels of the rotated spatial audio signal with a filter pair generated based on a predefined set of finite impulse responses (FIRs), head-related impulse response functions (HRTFs), or head-related impulse response (HRIR) datasets, and generating left and right channels of the binaural signal as a sum of the filtered channels of the rotated spatial audio signal derived for the left and right channels, respectively.

[0052] The output spatial audio signal may comprise a two-channel binaural audio signal, and the playback control information may comprise an indication of a playback direction defining a listening direction relative to the audio scene, and processing the processed spatial audio signal representing the audio scene based on the defocus direction / modified in accordance with the playback control information to generate the output spatial audio signal may comprise deriving one or more enhanced audio channels by multiplying, in the one or more frequency sub-bands, respective frequency bands of one or more audio channels of the processed spatial audio signal by the spectral adjustment coefficients received for the respective frequency sub-bands, and converting the one or more enhanced audio channels into a two-channel binaural audio signal in accordance with the indicated playback direction.

[0053] The output spatial audio signal may include a two-channel binaural audio signal, and the playback control information defining a listening direction relative to the audio scene may include an indication of a playback direction, and processing the processed spatial audio signal representing the audio scene modified based on the defocus direction to generate the output spatial audio signal in accordance with the playback control information may include converting one or more enhanced audio channels into a two-channel binaural audio signal in accordance with the indicated playback direction.

[0054] The output spatial audio signal may comprise a two-channel binaural signal, wherein the playback control information may comprise an indication of a playback direction defining a listening direction with respect to the audio scene, and wherein processing the processed spatial audio signal representing the audio scene modified based on the defocus direction to generate the output spatial audio signal in accordance with the playback control information may comprise selecting a set of head-related transfer functions (HRTFs) depending on the indicated playback direction, and converting the channels of the processed spatial audio signal into a two-channel binaural signal conveying the rotated audio scene using the selected set of HRTFs.

[0055] The playback control information may include an indication of a second predetermined loudspeaker configuration, and the output spatial audio signals may include multi-channel loudspeaker signals according to the second predetermined loudspeaker configuration, and processing the processed spatial audio signals representing the audio scene modified based on the defocus direction to generate the processed spatial audio signals in accordance with the playback control information may include deriving channels of the output spatial audio signal based on the channels of the processed spatial audio signal using amplitude panning, configured to derive a transformation matrix including amplitude panning gains that provide a mapping from the first predetermined loudspeaker configuration to the second predetermined loudspeaker configuration, and using the transformation matrix to multiply the channels of the output spatial audio signal by the channels of the processed spatial audio signal.

[0056] The method may further include obtaining a defocus input from a sensor arrangement including at least one direction sensor and at least one user input, wherein the defocus input may include an indication of a defocus direction based on an orientation of the at least one direction sensor.

[0057] The defocus input may further include an indicator of the amount of defocus.

[0058] The defocus input may further include an indicator of the defocus shape.

[0059] The defocus shape can include at least one of a defocus shape width, a defocus shape height, a defocus shape radius, a defocus shape distance, a defocus shape depth, a defocus shape range, a defocus shape diameter, and a defocus shape characterizer.

[0060] The defocus direction may be an arc defined by a range of defocus directions.

[0061] According to a third aspect, there is provided an apparatus comprising at least one processor and at least one memory containing computer program code configured, using the at least one processor, to at least obtain a defocus direction, process the spatial audio signal representing an audio scene to generate a spatial audio signal representing a modified audio scene based on the defocus direction so as to control relative de-emphasis in at least some other portions of the spatial audio signal in the defocus direction, and output the processed spatial audio signal, wherein the modified audio scene based on the defocus direction enables de-emphasis in at least some portions of the spatial audio signal in a de-focus direction relative to at least some other portions of the spatial audio signal.

[0062] The apparatus may further be adapted to obtain a defocus amount, and the apparatus for processing the spatial audio signal may be adapted to control relative de-emphasis in parts of the spatial audio signal in a defocus direction relative to other parts of the spatial audio signal at least partially according to the defocus amount.

[0063] The device adapted to process the spatial audio signal may at least one of: at least partially decreasing emphasis of portions of the spatial audio signal in a defocus direction relative to at least partially other portions of the spatial audio signal; and at least partially increasing emphasis of portions of the spatial audio signal in a defocus direction relative to other portions.

[0064] The device for processing a spatial audio signal may at least one of: decreasing a sound level in at least a portion of the spatial audio signal in a defocus direction relative to another portion of the spatial audio signal, at least in part according to the amount of defocus; and increasing a sound level in another portion of the spatial audio signal relative to the portion of the spatial audio signal in the defocus direction, at least in part according to the amount of defocus.

[0065] The apparatus may further be adapted to obtain a defocus shape, and the apparatus for processing the spatial audio signal may be adapted to control, at least in part, a relative de-emphasis within the defocus shape for a portion of the spatial audio signal in the defocus direction and for at least a portion of even another portion of the spatial audio signal.

[0066] The device for processing the spatial audio signal may perform at least one of: decreasing emphasis at least partially in portions of the spatial audio signal in the defocus direction and at least partially from within the defocus shape relative to other portions of the spatial audio signal; and increasing emphasis at least partially in portions of the spatial audio signal in the defocus direction and at least partially in other portions of the spatial audio signal within the defocus shape.

[0067] The device for processing the spatial audio signal may at least one of: reducing sound levels from within the defocus shape in portions of the spatial audio signal in the defocus direction at least in part according to the amount of defocus relative to at least partial other portions of the spatial audio signal; and increasing sound levels from the defocus shape in other portions of the spatial audio signal at least in part according to the amount of defocus relative to the portion of the spatial audio signal in the defocus direction.

[0068] The device can obtain playback control information for controlling at least one aspect of outputting the processed spatial audio signal, and the device causing the output of the processed spatial audio signal can perform one of: processing the processed spatial audio signal representing the audio scene modified based on the defocus direction to generate the output spatial audio signal in accordance with the playback control information; processing the processed spatial audio signal representing the audio scene in accordance with the playback control information before processing the spatial audio signal representing the audio scene to generate the processed spatial audio signal representing the audio scene modified based on the defocus direction; and outputting the processed spatial audio signal as an output spatial audio signal.

[0069] The spatial audio signal and the processed spatial audio signal may include respective Ambisonic signals, and the device for processing the spatial audio signal into the processed spatial audio signal may extract, for one or more frequency subbands, from the spatial audio signal a single-channel target audio signal representing sound components arriving from a focus direction, and generate a focused spatial audio signal, where the focused audio signal is located at a spatial position defined by the defocus direction, and generate the processed spatial audio signal as a linear combination of the focused spatial audio signal subtracted from the spatial audio signal, and at least one of the focused spatial audio signal and the spatial audio signal is scaled by a respective scaling factor derived based on the amount of defocus to reduce the relative level of sound in the defocus direction.

[0070] The apparatus for extracting a single-channel target audio signal applies a beamformer to derive, from the spatial audio signal, a beamformed signal representing sound components arriving from a defocused direction, and applies a post-filter to derive a processed audio signal based on the beamformed signal, thereby adjusting the spectrum of the beamformed signal to approximate the spectrum of sound arriving from a defocused direction.

[0071] The spatial audio signal and the processed spatial audio signal may include respective first-order Ambisonic signals.

[0072] The spatial audio signal and the processed spatial audio signal may include respective parametric spatial audio signals, the parametric spatial audio signals may include one or more audio channels and spatial metadata, the spatial metadata may include respective directional indications and energy ratio parameters for a plurality of frequency subbands. An apparatus for processing a spatial audio signal to generate a processed spatial audio signal may calculate, for one or more frequency subbands, respective angular differences between a defocus direction and a direction indicated for each frequency subband of the spatial audio signal, derive respective gain values ​​for the one or more frequency subbands based on the calculated angular differences for each frequency subband by using a scaling factor derived based on a predetermined function of the angular differences and the defocus amount, calculate, for the one or more frequency subbands of the processed spatial audio signal, respective updated directional energy values ​​based on the energy ratio parameters and gain values ​​for each frequency subband of the spatial audio signal, and calculating updated ambient energy values ​​for each frequency subband of the spatial audio signal based on the energy ratio parameter and the scaling factor for that frequency subband; calculating modified energy ratio parameters for each of the one or more frequency subbands of the processed spatial audio signal based on the updated directional energy divided by the sum of the updated direct and ambient energies; and calculating spectral adjustment factors for each of the one or more frequency subbands of the processed spatial audio signal based on the updated sum of the direct and ambient energies; and configuring the processed spatial audio signal comprising one or more audio channels of the spatial audio signal, a directional indication of the spatial audio signal, the modified energy ratio parameter, and the spectral adjustment factor.The spatial audio signal and the processed spatial audio signal can include respective parametric spatial audio signals, where the parametric spatial audio signals can include one or more audio channels and spatial metadata, where the spatial metadata can include respective directional indications and energy ratio parameters for a plurality of frequency subbands. The device for processing the spatial audio signal to generate the processed spatial audio signal can calculate, for one or more frequency subbands, respective angular differences between a defocus direction and a direction indicated for each frequency subband of the spatial audio signal, derive respective gain values ​​for the one or more frequency subbands based on the calculated angular differences for each frequency subband by using a scaling factor derived based on a predetermined function of the angular differences and the amount of defocus, and calculate, for the one or more frequency subbands of the processed spatial audio signal, respective updated directional energy values ​​based on the energy ratio parameters and the gain values ​​for each frequency subband of the spatial audio signal.For one or more frequency bands of the processed spatial audio signal and gain values, calculate respective updated ambient energy values ​​based on the energy ratio parameters and scaling factors for the respective frequency subbands of the spatial audio signal; calculate respective modified energy ratio parameters for the one or more frequency subbands of the processed spatial audio signal based on the updated directional energy values ​​divided by the sum of the updated direct and ambient energies; calculate respective spectral adjustment coefficients for the one or more frequency subbands of the processed spatial audio signal based on the updated sum of the direct and ambient energies; derive one or more extended audio channels by multiplying, in the one or more frequency subbands, each frequency band of a plurality of respective one audio channels of the spatial audio signal by the spectral adjustment coefficient derived for the respective frequency subband; and configure a processed spatial audio signal comprising the one or more extended audio channels, a directional indication of the spatial audio signal, and the modified energy ratio parameters.

[0073] The spatial audio signal and the processed spatial audio signal may include respective multi-channel loudspeaker signals according to a first predetermined loudspeaker configuration, and the device for processing the processed spatial audio signal to generate the processed spatial audio signal may calculate respective angular differences between defocus directions and loudspeaker directions indicated for each channel of the spatial audio signal, derive respective gain values ​​for each channel of the spatial audio signal based on the angular differences calculated for the respective channels by using a predetermined function of the angular differences and a scaling factor derived based on the amount of defocus, derive one or more modified audio channels by multiplying each channel of the spatial audio signal by the derived gain value for the respective channel, and provide the modified audio channels as the processed spatial audio signal.

[0074] The predetermined function of the angular difference may result in a gain value that decreases as the angular difference value decreases and increases as the angular difference value increases.

[0075] The processed spatial audio signal may comprise an Ambisonic signal, and the output spatial audio signal may comprise a two-channel binaural signal. Here, the playback control information may include an indication of a playback direction defining a listening direction with respect to the audio scene. An apparatus configured to process the processed spatial audio signal representing the modified audio scene based on the defocus direction to generate the output spatial audio signal according to the playback control information may generate a rotation matrix depending on the indicated playback direction, multiply the channels of the processed spatial audio signal by the rotation matrix to derive rotated spatial audio signals, filter the channels of the rotated spatial audio signal with a filter pair generated based on a predetermined set of finite impulse responses (FIRs), head-impulse response functions (HRTFs), or head-impulse responses (HRIRs), and generate, for the left and right channels, left and right channels of the binaural signal as a sum of the filtered channels of the rotated spatial audio signal.

[0076] the output spatial audio signal may comprise a two-channel binaural audio signal, the playback control information may comprise an indication of a playback direction defining a listening direction relative to the audio scene, and the processed spatial audio signal representing the audio scene modified based on the defocus direction to generate the output spatial audio signal in accordance with the playback control information; and the processing device may derive one or more enhanced audio channels in the one or more frequency subbands by multiplying respective frequency bands of one or more audio channels of the processed spatial audio signal by the received spectral adjustment coefficients for the respective frequency subbands, and convert the one or more enhanced audio channels into a two-channel binaural audio signal in accordance with the indicated playback direction.

[0077] The output spatial audio signal may comprise a two-channel binaural audio signal, and the playback control information may comprise an indication of a playback direction defining a listening direction relative to the audio scene. The device for processing the processed spatial audio signal representing the modified audio scene based on the defocus direction to generate an output spatial audio signal in accordance with the playback control information may convert the one or more extended audio channels into the two-channel binaural audio signal in accordance with the indicated playback direction.

[0078] The output spatial audio signal may comprise a two-channel binaural signal, wherein the playback control information may comprise an indication of a playback direction defining a listening direction with respect to the audio scene, and a device adapted to process the processed spatial audio signal representing the modified audio scene based on the defocus direction may select a set of head-related transfer functions (HRTFs) dependent on the specified playback direction to generate the output spatial audio signal responsive to the playback control information, and may convert the channels of the processed spatial audio signal into a two-channel binaural signal conveying the rotated audio scene using the selected set of HRTFs.

[0079] The playback control information may include an indication of a second predetermined loudspeaker configuration, and the output spatial audio signal may include a multi-channel loudspeaker signal with the second predetermined loudspeaker configuration, and the means adapted to process the processed spatial audio signal representing the audio scene modified based on the defocus direction to generate the output spatial audio signal in accordance with the playback control information may be configured to derive a transformation matrix including amplitude panning gains that provide a mapping from the first loudspeaker configuration to the second loudspeaker configuration, thereby deriving channels of the output spatial audio signal based on the channels of the processed spatial audio signal using amplitude panning, and multiplexing channels of the processed spatial audio signal into channels of the output spatial audio signal using the transformation matrix.

[0080] The device may be adapted to obtain the defocus input from a sensor arrangement including at least one directional sensor and at least one user input, where the defocus input may include an indication of the defocus direction based on the at least one directional sensor direction.

[0081] The defocus input may further include an indicator of the amount of defocus.

[0082] The defocus input may further include an indicator of the defocus shape.

[0083] The defocus shape can include at least one of a defocus shape width, a defocus shape height, a defocus shape radius, a defocus shape distance, a defocus shape depth, a defocus shape range, a defocus shape diameter, and a defocus shape characterizer.

[0084] The defocus direction may be an arc defined by a range of defocus directions.

[0085] According to a fourth aspect, there is provided an apparatus comprising: circuitry configured to obtain a defocus direction; the spatial audio signal processing circuitry configured to process an audio scene based on the defocus direction to generate a processed spatial audio signal representing a modified audio scene based on the defocus direction to control at least in part relative de-emphasis of a portion of the spatial audio signal in the defocus direction relative to at least some other portions of the defocus direction; an output circuitry configured to control an output of the processed spatial audio signal; and an output circuitry configured to control an output of the processed spatial audio signal, wherein the modified audio scene based on the defocus direction enables de-emphasis of a portion of the spatial audio signal in the defocus direction relative to at least some other portions of the spatial audio signal.

[0086] According to a fifth aspect, there is provided a computer program comprising instructions [or a computer readable medium comprising program instructions] for causing an apparatus to at least obtain a defocus direction; process the spatial audio signal representing an audio scene to generate a processed spatial audio signal representing an audio scene that is modified based on the defocus direction to control a relative de-emphasis of the defocus direction in at least some portions of the spatial audio signal; and output the processed spatial audio signal, wherein the audio scene modified based on the defocus direction enables de-emphasis of portions of the spatial audio signal in the defocus direction relative to at least some other portions of the spatial audio signal, in at least some portions of the spatial audio signal.

[0087] According to a sixth aspect, there is provided a non-transitory computer-readable medium comprising program instructions to cause an apparatus to at least obtain a defocus direction; process the spatial audio signal representing an audio scene to generate a processed spatial audio signal representing an audio scene that is modified based on the defocus direction so as to control a relative de-emphasis of the de-focus direction in at least some portions of the spatial audio signal; and output the processed spatial audio signal, wherein the audio scene modified based on the de-focus direction enables de-emphasis of portions of the de-focus direction relative to at least some other portions of the spatial audio signal, in at least some portions of the spatial audio signal.

[0088] According to a seventh aspect, there is provided an apparatus comprising: means for obtaining a defocus direction; means for processing a spatial audio signal representing an audio scene to generate a processed spatial audio signal representing a modified audio scene based on the defocus direction so as to control de-emphasis of a part of the spatial audio signal in at least a part of the spatial audio signal relative to the defocus direction with respect to another part of the spatial audio signal; and means for outputting the processed spatial audio signal, wherein the modified audio scene based on the defocus direction enables de-emphasis of a part of the spatial audio signal in at least a part of the spatial audio signal in the defocus direction with respect to at least a part of the other part of the spatial audio signal.

[0089] According to an eighth aspect, there is provided a computer-readable medium comprising program instructions for causing an apparatus to at least obtain a defocus direction and process a spatial audio signal representing the audio scene to generate a processed spatial audio signal representing the audio scene that is modified based on the defocus direction so as to control relative de-emphasis of portions of the spatial audio signal in the defocus direction relative to at least some other portions of the spatial audio signal.

[0090] An apparatus comprising means for performing the operations set forth above.

[0091] An apparatus configured to perform the operations of the above-described method.

[0092] A computer program comprising program instructions for causing a computer to carry out the method described above.

[0093] A computer program product stored on the medium can cause an apparatus to perform the methods described herein.

[0094] The electronic device may comprise an apparatus as described herein.

[0095] The chipset may comprise an apparatus as described herein.

[0096] Embodiments of the present application aim to address problems associated with the state of the art. [Brief explanation of the drawings]

[0097] For a better understanding of the present application, reference is made, by way of example, to the accompanying drawings, in which: [Figure 1a] 1a, 1b and 1c show an exemplary sound scene showing an audio focus region or area. [Figure 1b] 1a, 1b and 1c show an exemplary sound scene showing an audio focus region or area. [Figure 1c] 1a, 1b and 1c show an exemplary sound scene showing an audio focus region or area. [Figure 2a] 2a and 2b schematically illustrate an exemplary playback device and method for operating the playback device, according to some embodiments. [Figure 2b] 2a and 2b schematically illustrate an exemplary playback device and method for operating the playback device, according to some embodiments. [Figure 3a] 3a and 3b schematically illustrate the exemplary focus processor shown in FIG. 2a with a higher-order Ambisonic audio signal input and a method of operating the exemplary focus processor according to some embodiments. [Figure 3b] 3a and 3b schematically illustrate the exemplary focus processor shown in FIG. 2a with a higher-order Ambisonic audio signal input and a method of operating the exemplary focus processor according to some embodiments. [Figure 4a] 4a and 4b illustrate schematically the exemplary focus processor shown in FIG. 2a with a parametric spatial audio signal input and a method of operating the exemplary focus processor, according to some embodiments. [Figure 4b] 4a and 4b illustrate schematically the exemplary focus processor shown in FIG. 2a with a parametric spatial audio signal input and a method of operating the exemplary focus processor, according to some embodiments. [Figure 5a] 5a and 5b schematically illustrate an exemplary focus processor shown in FIG. 2a having a multi-channel and / or audio object signal input and a method of operating the exemplary focus processor according to some embodiments. [Figure 5b]5a and 5b schematically illustrate an exemplary focus processor shown in FIG. 2a having a multi-channel and / or audio object signal input and a method of operating the exemplary focus processor according to some embodiments. [Figure 6a] 6a and 6b schematically illustrate an exemplary playback processor, such as that shown in FIG. 2a, having a higher-order Ambisonic audio signal input, and a method of operating the exemplary playback processor, according to some embodiments. [Figure 6b] 6a and 6b schematically illustrate an exemplary playback processor, such as that shown in FIG. 2a, having a higher-order Ambisonic audio signal input, and a method of operating the exemplary playback processor, according to some embodiments. [Figure 7a] 7a and 7b illustrate schematically an exemplary playback processor as shown in FIG. 2a having a parametric spatial audio signal input and a method of operating the exemplary playback processor according to some embodiments. [Figure 7b] 7a and 7b illustrate schematically an exemplary playback processor as shown in FIG. 2a having a parametric spatial audio signal input and a method of operating the exemplary playback processor according to some embodiments. [Figure 8] FIG. 8 illustrates an exemplary implementation of some embodiments. [Figure 9] FIG. 9 illustrates an exemplary controller for controlling the focus direction, focus amount, and focus width according to an embodiment. [Figure 10] FIG. 10 illustrates an example of a processing output based on processing a high-order Ambisonic audio signal according to some embodiments. [Figure 11] FIG. 11 shows an exemplary apparatus suitable for implementing the illustrated apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0098] Suitable apparatus and possible mechanisms for providing efficient rendering and playback of spatial audio signals are described in further detail below.

[0099] In the previous example of spatial audio signal playback, the user could control the focus direction and amount. However, in some situations, such focus direction / amount control may not be sufficient. The concepts described below are devices and methods featuring additional focus control that can indicate the elimination or de-emphasis of sounds in specific directions. For example, a sound field may have several different features, such as multiple dominant sound sources in specific directions, as well as ambient sounds. Some users may prefer to remove certain features of the sound field, while others may prefer to hear the complete audio scene or remove alternative features of the sound field. In particular, a user may want to remove undesired sounds so that the rest of the spatial sound scene is reproduced as originally intended.

[0100] Figures 1a to 1c, described below, show what a user will perceive when listening to a reproduced spatial audio signal.

[0101] As an example, Figure 1a shows a user 101 with a defined direction. Within the audio scene there is a source of interest 105, e.g. a speaker. Additionally, there may be other ambient audio content 107 surrounding the user.

[0102] Additionally, the user can identify interfering sound sources such as an air conditioner 103. Traditionally, the user can control playback to focus on sources of interest 105 and emphasize them over interfering sources 103. However, the concept described in the embodiment instead attempts to improve sound quality by performing "removal" (or defocusing or negative focus) of the identified sources, as shown in Figure 1a by defocusing or negative focus identified sources 103.

[0103] As another example, as shown in FIG. 1b, a user may wish to defocus or negatively focus any source within a shape or region within the sound scene. Thus, for example, FIG. 1b shows a user 101 positioned at a defined orientation within an audio or sound scene having a source of interest 105, such as a speaker, other ambient audio content 107, such as environmental audio content, and an interfering source 155 within a defined region 153. In this example, the region of defocus or negative focus is represented by a defocus arc 151 of a defined width and direction relative to the user 101. The defocus arc 151 of a defined width and direction relative to the user 101 covers the interfering source 155 within the interfering source region 153.

[0104] A further way in which regions of defocus or negative focus may be represented is shown in Figure 1c, where a defocus region or volume (for a 3D region) 161 covers an interferer 155 within an interferer region 153. In this example, the defocus region may be defined by distance as well as direction and "width."

[0105] Accordingly, the embodiments described herein attempt to provide control of defocus shape (in addition to defocus direction and amount). The concepts as described with respect to the embodiments described herein relate to spatial audio reproduction and enable audio playback with controls to reduce / remove audio elements (or regions or volumes) from selectable spatial directions (or regions or volumes) by a desired amount (e.g., 0% to 100%), so as to de-emphasize the audibility of audio elements in selected spatial directions (or regions or volumes) while maintaining the audibility of desired audio elements in the selected spatial directions (or regions or volumes), while allowing the spatial audio signal format to remain the same.

[0106] This embodiment provides at least one defocus (or negative focus) parameter corresponding to a selectable direction and amount. Furthermore, in some embodiments, this defocus (or negative focus) parameter can define a defocus (or negative focus) shape, which can be defined by any one (or a combination of two or more) of the following parameters corresponding to direction, width, height, radius, distance, and depth. This parameter set in some embodiments includes parameters that define any defocus shape.

[0107] In some embodiments, at least one defocus parameter is provided to further emphasize the audibility of selected spatial directions (or shapes, areas, or volumes).

[0108] In some embodiments, spatial audio signal processing may be performed by obtaining spatial audio signals associated with media having multiple viewing directions, obtaining focus / defocus direction and amount parameters (which may optionally include obtaining at least one focus / defocus shape information), modifying the spatial audio signals to have desired (focus) and defocus characteristics, and playing back (over headphones or loudspeakers) the modified spatial audio signals.

[0109] The resulting spatial audio signal may be in a parametric spatial audio format, such as an Ambisonic signal, a loudspeaker signal, an audio channel configuration and associated spatial metadata.

[0110] Focus / defocus information can be defined as follows: focus refers to increasing the relative prominence of audio coming from a selectable direction (or shape or region), while defocus refers to decreasing the relative prominence of audio coming from that direction (or shape or region).

[0111] The focus / defocus amount determines how much to focus or defocus, which can be, for example, 0% to 100%, where 0% means to keep the original sound scene unmodified and 100% means to maximally focus / defocus in the desired orientation or within a specified range.

[0112] The focus / defocus control in some embodiments may be a switch control to determine whether to focus or defocus, or may be controlled in other ways, for example, by extending the focus amount range from -100% to 100%, where negative values ​​indicate a defocus (or negative focus) effect and positive values ​​indicate a focus effect.

[0113] Note that different users may desire different focus / defocus characteristics: the original spatial audio signal may be modified and played back individually for each user based on the user's personal preferences.

[0114] 2a illustrates a block diagram of some components and / or entities of a spatial audio processing configuration 250 according to one example. It will be understood that the two separate steps (focus / defocus processor + playback processor) illustrated in this figure and further detailed below can be implemented as an integrated process or, in some examples, can be implemented in the reverse order (the playback processor operation then follows the focus / defocus processor operation) as described herein. The spatial audio processing configuration 250 includes an audio focus processor 201 configured to receive an input audio signal and further focus / defocus parameters 202, and to derive an audio signal having a focused / defocused sound component 204 based on the input audio signal 200 and depending on the focus / defocus parameters 202 (which may include a focus / defocus direction, a focus / defocus amount, a focus / defocus height, a focus / defocus radius, a focus / defocus distance, and a focus depth for the focus / defocus element). The spatial audio processing arrangement 250 may further comprise an audio playback processor 207 configured to receive an audio signal having focused / defocused sound components 204 and playback control information 206. The audio playback processor 207 is further configured to derive an output audio signal 208 in a predetermined audio format based on the audio signal having focused / defocused sound components 204, further relying on the playback control information 206, which functions to control at least one aspect related to the processing of the spatial audio signal having focused / defocused components in the audio playback processor 207. The playback control information 206 may include an indication of a playback direction (or playback orientation) and / or an indication of an applicable speaker configuration.In view of the above-described method for processing spatial audio signals, the audio focus processor 201 may be configured to implement an aspect of processing the spatial audio signals by modifying an audio scene to control emphasis or de-emphasis in at least a portion of the spatial audio signals in a received focus region or direction according to a received focus / defocus amount. The audio playback processor 207 may output the processed spatial audio signals as a modified audio scene based on the observed direction and / or position, the modified audio scene exhibiting emphasis according to the received focus amount for at least said portion of the spatial audio signals within the focus region.

[0115] In FIG. 2a, an input audio signal, an audio signal with focused / defocused sound components, and an output audio signal are provided as respective spatial audio signals in a predetermined spatial audio format. Accordingly, these signals may be referred to as the input spatial audio signal, the spatial audio signal with focused / defocused sound components, and the output spatial audio signal, respectively. In line with the foregoing description, a spatial audio signal typically conveys an audio scene that includes both one or more directional sound sources at each specific position in the audio scene and the ambience of the audio scene. However, in some scenarios, a spatial audio scene may include one or more directional sound sources without bidirectional sound sources, or bidirectional sound sources without any directional sound sources. In this regard, a spatial audio signal comprises information conveying one or more directional sound components representing distinct sound sources having a fixed position within the audio scene (e.g., a fixed direction of arrival and a fixed intensity relative to the listening point) and / or ambient sound components representing environmental sounds within the audio scene. It should be noted that dividing an audio scene into a directional sound component and an ambient component is typically only a representation or approximation, but a real sound scene may contain more complex features such as wide sound sources and coherent acoustic reflections. Nevertheless, even with such complex acoustic features, the conceptualization of an audio scene as a combination of a direct component and an ambient component is usually a fair representation or approximation, at least in a perceptual sense.

[0116] Typically, the input audio signal and the audio signal with focused / defocused tonal components are provided in the same predefined spatial format, while the output audio signal may be provided in the same spatial format as that applied to the input audio signal (and the audio signal with focused / defocused tonal components) or may adopt a different predefined spatial format for the output audio signal. The spatial audio format of the output audio signal is selected taking into account the characteristics of the sound reproduction hardware applied to reproduce the output audio signal. Generally, the input audio signal may be provided in a first predefined spatial audio format, and the output audio signal may be provided in a second predefined spatial audio format. Non-limiting examples of spatial audio formats suitable for use as the first and / or second spatial audio formats include Ambisonic, surround loudspeaker signals according to a predefined loudspeaker configuration, and predefined parametric spatial audio formats. More detailed, non-limiting examples of the use of these spatial audio formats as the first and / or second spatial audio formats within the framework of the spatial audio processing arrangement 250 are provided later in this disclosure.

[0117] The spatial audio processing arrangement 250 is applied to process the input spatial audio signal 200, typically as a sequence of input frames, into a respective sequence of output frames, each input (output) frame including a respective segment of a digital audio signal for each channel of the input (output) spatial audio signal, provided as a respective time series of input (output) samples at a predetermined sampling frequency. In some embodiments, the input signal to the spatial audio processing arrangement 250 may be in an encoded format, e.g., AAC, or AAC+embedded metadata. In such embodiments, the encoded audio input may initially be a decoder. Similarly, in some embodiments, the output from the spatial audio processing arrangement 250 may be encoded in any suitable manner.

[0118] In a typical example, the spatial audio processor 250 uses a fixed, predetermined frame length, with each frame containing L samples for each channel of the input spatial audio signal, and maps them to corresponding durations at a predetermined sampling frequency. As an example in this regard, the fixed frame length is 20 milliseconds (ms), resulting in frames of L = 160, L = 320, L = 640, and L = 960 samples per channel at sampling frequencies of 8, 16, 32, or 48 kHz, respectively. The frames may be non-overlapping or partially overlapping, depending on whether the processor applies filter banks and how these filter banks are configured. However, these values ​​serve as non-limiting examples, and different frame lengths and / or sampling frequencies may be used instead, depending on, for example, the desired audio bandwidth, the desired framing delay, and / or the available processing capacity.

[0119] In the spatial audio processing device 250, focus / defocus refers to a user-selectable directional / quantity parameter (or spatial region of interest). Focus / defocus may generally be a certain direction, distance, radius, or arc of the audio scene. Another example is a focus / defocus region where a (directional) sound source of interest is currently located. In the former scenario, the user-selectable focus / defocus may indicate a region that remains constant or changes infrequently because the focus is primarily in a particular direction (or spatial region). In the latter scenario, the user-selected focus / defocus may change more frequently because the focus / defocus is set on a particular sound source that may (or may not) change its position (or shape / size) within the audio scene over time. In one example, focus / defocus may be defined as, for example, an azimuth angle that defines a direction.

[0120] The functionality described above with reference to the components of the spatial audio processing device 250 may be provided, for example, according to a method 260 illustrated by the flowchart shown in FIG. 2b. Method 260 may be provided, for example, by an apparatus configured to implement the spatial audio processing system 250 described in this disclosure through several examples. Method 260 functions as a method for processing an input spatial audio signal representing an audio scene into an output spatial audio signal representing a modified audio scene. Method 260 includes receiving an indication of a focus / defocus direction and an indication of a focus / defocus strength or amount, as shown in block 261. Method 260 further includes processing the input spatial audio signal into an intermediate spatial audio signal representing the modified audio scene, in which the relative levels of sounds arriving from the focus / defocus direction are modified according to the focus / defocus strength, as shown in block 263. Method 260 further includes receiving playback control information for controlling the processing of the intermediate spatial signal into an output spatial audio signal, as shown in block 265. The playback control information may, for example, define at least one of a playback direction (e.g., listening direction or viewing direction) or a speaker configuration of the output spatial audio signals. Method 260 further includes processing the intermediate spatial audio signals into output spatial audio signals according to the playback control information, as shown in block 267.

[0121] The method 260 may be modified in a number of ways, for example, according to the examples of the functionality of each of the components of the spatial audio processing unit 250 provided above and below.

[0122] In the following examples, the defocus operation is described in more detail, but it should be understood that the same operation can be applied to further focus operations as well as further defocus operations.

[0123] In some embodiments, the input to the spatial audio processing configuration 250 is an Ambisonic signal. The device can be configured to receive (and apply the method to) Ambisonic signals of any order. The Ambisonic audio signal could be a first-order Ambisonic (FOA) signal, consisting of an omnidirectional signal and three orthogonal first-order patterns along the y, z, and x coordinate axes. The y, z, and x coordinate orders are chosen here because they are the same orders as the first-order coefficients in a typical ACN (Ambisonics Channel Numbering) channel ordering for Ambisonic signals.

[0124] It should be noted that the Ambisonics audio format can represent spatial audio signals in terms of spatial beam patterns, and it would be straightforward for one skilled in the art to design alternative sets of spatial beam patterns to represent spatial audio, as illustrated herein. Furthermore, the Ambisonics audio format is a particularly relevant audio format because it is a typical way of representing spatial audio in the context of 360 video. Typical sources of Ambisonics audio signals include microphone arrays and content from VR video streaming services (such as YouTube® 360).

[0125] 3a, focus processor 350 is shown in the context of Ambisonic input and output. The diagram assumes a first-order Ambisonic (FOA) signal (four channels), but higher-order Ambisonic (HOA) may be applied instead of FOA. In embodiments implementing an HOA input format, the number of channels instead of four could be, for example, nine channels (second-order Ambisonics) or sixteen channels (third-order Ambisonics).

[0126] An example ambisonic signal x FOA (t) 300 and (de)focus direction 304 , (de)focus amount and (de)focus control 310 are inputs to a focus processor 350 .

[0127] In some embodiments, the focus processor 350 includes a filter bank 301. The filter bank 301 is configured to convert an Ambisonic (FOA) signal 300 (corresponding to an Ambisonic or spherical harmonic pattern) to generate a time-frequency domain version of the time-domain input audio signal. In some embodiments, the filter bank 301 may be any other suitable filter bank for spatial audio processing, such as a short-time Fourier transform (STFT) or complex-modulated quadrature mirror filter (QMF) bank. The output of the filter bank 301 is a frequency-band time-frequency domain Ambisonic audio signal 302. A frequency band can be one or more frequency bins (individual frequency components) of the applied filter bank 301. The frequency bands can approximate a perceptually relevant resolution, such as Bark frequency bands, which are spectrally more selective at low frequencies than at high frequencies. Alternatively, in some implementations, the frequency bands can correspond to frequency bins.

[0128] The (unfocused) time-frequency domain Ambisonic audio signal 302 is output to a mono focuser 303 and a mixer 311 .

[0129] The focus processor 301 may further comprise a mono focuser 303 configured to receive the transformed (unfocused) time-frequency domain Ambisonic signal 302 from the filter bank 301 and further to receive a (de)focusing direction parameter 304.

[0130] The mono (de)focuser 303 can implement any known method for generating a mono-focused audio output based on the FOA input. In this example, the mono-focuser 303 implements a minimum variance distortion-free response (MVDR) mono-focused audio output. The MVDR beamforming operation attempts to obtain the target signal from the desired focus direction without distortion, while adaptively finding beamforming weights that attempt to minimize the output energy (in other words, suppress interference energy) due to this constraint.

[0131] In some embodiments, the mono focuser 303 focuses the frequency band signal (e.g., four channels in the case of FOA) into

number

[0132] A monofocuser 303 implementing an MVDR beamformer can be used for each frequency band k. An estimate of the covariance matrix of the signal x(b,n) in the bins in band k (and possibly time-averaged over some time index n). A steering vector according to the focus direction. In the example of an FOA signal, the steering vector may be generated based on a unit vector oriented in the focus direction. For example, the steering vector for FOA is

number

[0133] Based on the covariance matrix estimates and the steering vectors, the weights w(k,n) can be generated using the known MVDR formula.

[0134] Thus, the mono focuser 303 can provide a single channel focused output signal 306 that, in some embodiments, is provided to an Ambisonic panner 305 .

[0135] In some embodiments, an Ambisonics panner 305 is configured to receive the channel (de)focused output signals 306 and the (de)focus direction 304 and generate an Ambisonic signal, where the mono focused signal is positioned in the focus direction. The focused time-frequency Ambisonic signal 308 output by the Ambisonics panner 305 is

number

[0136] In some embodiments, the (de)focused time-frequency Ambisonic signal y FOA The (b,n) 308 may then be output to a mixer 311 .

[0137] In some embodiments, the output of a beamformer, such as an MVDR, can be cascaded with a post-filter. A post-filter is a process that adaptively modifies the gain or energy of the beamformer output within a frequency band. For example, MVDR is known to be effective at suppressing strong individual interfering sources, but only performs moderately well in ambient acoustic scenes, such as outdoor recordings with traffic noise. This is because MVDR effectively aims to steer the beam pattern minimum in the direction of the interferer. When the interfering sound is spatially dispersed, such as traffic noise, MVDR does not effectively suppress the interference.

[0138] Thus, in some embodiments, a post-filter can be implemented to estimate the sound energy within a frequency band in the focus direction. The beamformer output energy is then measured in the same frequency band, and a gain is applied in the frequency band to correct the sound spectrum to improve the estimated target spectrum. In such embodiments, the post-filter can further suppress interfering sounds.

[0139] An example of a post-filter is described by Delikaris Manias, Symeon, and Ville Pulkki in "Cross-Sectional Pattern Coherence Algorithm for Spatial Filtering Applications Utilizing Microphone Arrays," IEEE Transactions on Audio, Speech, and Language Processing 21, No. 11 (2013): 2356-2367, where the target energy in the viewing direction is estimated using cross-spectral energy estimates between the first and second order spherical harmonic signals. Cross-spectral estimates can also be obtained for other patterns, such as between the zeroth (omnidirectional) and first (dipole) order spherical harmonic signals. The cross-spectral estimate provides an energy estimate relative to the target direction.

[0140] If post-filtering is implemented, a gain g(k,n) can be added to the beamforming equation.

number

[0141] This gain g(k,n) can be derived using the cross-spectral energy estimation method as follows: First, we formulate the cross-correlation between the omnidirectional FOA signal component with a positive lobe toward the focus direction and the figure-eight signal,

number

number

number

number

[0142] In other words, if the energy estimate C(k,n) is less than the beamformed output energy D(k,n), then the beamformed output energy in band k is reduced by the spatial filter. Thus, the function of the spatial filter is to further adjust the spectrum of the beamformer output to more closely match the spectrum of the sound arriving from the focus direction.

[0143] In some embodiments, a (de)focusing processor can utilize this post-filtering. The beamformed output y(b,n) of the monofocuser 303 can be processed in frequency bands with a post-filter gain to produce a post-filtered beamformed output y'(b,n), where y'(b,n) is applied instead of y(b,n). It will be appreciated that there are a variety of suitable beamformers and post-filters that can be applied other than those described as examples above.

[0144] In some embodiments, the focus processor 350 comprises a mixer 311. The mixer mixes the (de)focused time-frequency Ambisonic signal y FOA (b,n) 308 and the unfocused time-frequency Ambisonic signal x(b,n) 302 (with potential delay adjustment where MVDR estimation and processing includes look-ahead processing). Additionally, the mixer 311 receives the (de)focus amount and focus / defocus control parameters 310.

[0145] In this example, the (de)focus control parameter is a binary switch of "focus" or "defocus." The (de)focus amount parameter a(n), expressed as a factor between 0..1 (where 1 is maximum focus), is utilized to describe either the amount of focus or the amount of defocus, depending on which mode is used.

[0146] In some embodiments, when the defocus parameter is in "focus" mode, the output of the mixer 311 is:

number

[0147] In some embodiments, the mixer, when the defocus parameter is in "defocus" mode:

number

[0148] In other words, when a(n) is 0, the defocusing is also zero; however, when a(n) is greater than or equal to 1, the blending procedure converts the spatial FOA signal x(b,n) into a spatialized focus signal, y(b,n). FOA (b,n) is subtracted. The subtraction reduces the amplitude of the signal components from the focus direction. In other words, the defocusing process is performed, and the resulting Ambisonic spatial audio signal has reduced amplitude for sounds from the focus direction. In some configurations, y MIX (b,n) 312 can be amplified in principle as a function of a(n) to account for the average loss of loudness due to the defocusing process.

[0149] The output of the mixer 311, the mixed time-frequency Ambisonic audio signal 312, is passed to an inverse filter bank 313.

[0150] In some embodiments, the focus processor 350 includes an inverse filter bank 313 configured to receive the mixed time-frequency Ambisonic audio signal 312 and transform the audio signal into the time domain, which generates an appropriate pulse code modulated Ambisonic audio signal with added focus / defocus.

[0151] FIG. 3b shows a flow chart of the operation 360 of the FOA focus processor shown in FIG. 3a.

[0152] The initial operation is receiving an Ambisonic (FOA) audio signal (and focus parameters such as direction, width, amount or other control information) by step 361, as shown in Figure 3b.

[0153] The next operation is to generate a converted Ambisonic audio signal in the time-frequency domain, as shown in Figure 3b, by step 363.

[0154] Having generated the time-frequency domain Ambisonic audio signal, the next operation is one of generating a mono-focused Ambisonic audio signal from the time-frequency domain Ambisonic audio signal based on a focus direction (e.g., using beamforming), as shown in FIG. 3b by step 365.

[0155] Then, via step 367, Ambisonic spanning is performed on the mono-(de)focused Ambisonic audio signal based on the focus direction as shown in Figure 3b.

[0156] The panned Ambisonic audio signal (the (de)focused time-frequency Ambisonic signal) is then mixed with the unfocused time-frequency Ambisonic signal in step 369 based on the (de)focus amount and the (de)focus control parameters, as shown in Figure 3b.

[0157] The mixed Ambisonic audio signal can then be inverse transformed as shown in Figure 3b by step 371.

[0158] Step 373 then outputs a time domain Ambisonic audio signal as shown in Figure 3b.

[0159] Referring to FIG. 4a, a focus processor configured to receive a parametric spatial audio signal as input is shown. The parametric spatial audio signal includes an audio signal and spatial metadata, such as one or more directions in a frequency band and one or more direct-to-total energy ratios. The structure and generation of parametric spatial audio signals are known, and their generation has been described from microphone arrays (e.g., mobile phones, VR cameras). Furthermore, parametric spatial audio signals can be generated from loudspeaker signals and Ambisonic signals. In some embodiments, the parametric spatial audio signal may be generated from an IVAS (Immersive Voice and Audio Services) audio stream, which may be decoded and demultiplexed into the form of spatial metadata and audio channels. While a typical number of audio channels in such a parametric spatial audio stream is two audio channels, in some embodiments the number of audio channels may be any number of audio channels.

[0160] In some examples, the parametric information includes depth / distance information that can be implemented in six degrees of freedom (6DOF) playback, where distance metadata (along with other metadata) is used to determine how sound energy and direction should change as a function of the user's movement.

[0161] In this example, each spatial metadata direction parameter is associated with both a direct-to-total energy ratio and a distance parameter. The estimation of distance parameters associated with parametric spatial audio capture has been detailed in previous applications such as GB patent documents GB1710093.4 and GB1710085.0 and will not be explored further for clarity.

[0162] A focus processor 450 configured to receive the parametric spatial audio 400 is configured to use (de)focus parameters to determine how much the direct and ambient components of the parametric spatial audio signal should be attenuated or emphasized to enable a (de)focus effect. The focus processor 450 is described in two configurations below. The first uses (defocus) parameters, i.e., direction and amount, and further includes width, which results in a focus / defocus arc. In this configuration, the 6DOF distance parameter is optional. The second uses parameters (de)focus direction and amount, as well as distance and radius, which results in a focus / defocus sphere at a certain position. In this configuration, the 6DOF distance parameter is required. The differences between these configurations are expressed only where necessary in the following description.

[0163] In the examples below, the methods (and formulas) are presented as being time-invariant, but it should be understood that all parameters may vary over time.

[0164] In some embodiments, the focus processor comprises a ratio modifier and spectral adjustment coefficient determiner 401 configured to receive focus parameters 408 and further spatial metadata consisting of direction 402 (and in some embodiments distance 422) and direct-to-total energy ratio 404 in frequency bands.

[0165] The ratio modifier and spectral adjustment coefficient determiner 401 is configured to receive a focus parameter and further receive spatial metadata consisting of a direction 402, a direct-to-total energy ratio in a frequency band 404 (and in some embodiments, a distance 422).

[0166] In the following description, unless otherwise specified, it is assumed that the focus parameters include direction, width, and amount. In some embodiments, the ratio modifier and spectral adjustment factor determiner 401 is configured to determine the angular difference between the focus direction (one for every frequency band k) and the spatial metadata direction (potentially different for different frequency bands k). In some embodiments, v m (k) is determined as a column-wise vector pointing to the direction parameter of the spatial metadata in band k and as a column-wise vector pointing to the focus direction. The angular distance β(k) is

number

[0167] The ratio modifier and spectral adjustment coefficient determiner 401 is then configured to determine the direct gain parameter f(k). The focus amount parameter a may be expressed as a normalized number between 0..1 (where 0 means zero focus / defocus and 1 is maximum focus / defocus) and a focus width β, which may be, for example, 20 degrees at some point.

[0168] When the ratio modifier and spectral adjustment factor determiner 401 is configured to perform focusing (as opposed to defocusing), an exemplary gain formula is:

number

number

[0169] The exemplary formula shows that in some embodiments, the constant c may have different values ​​for focus imperfections and defocus. Furthermore, in practice, it may be desirable to smooth the above focus gain function so that it transitions smoothly from high values ​​in focus regions to low values ​​in out-of-focus regions.

[0170] In the following description, unless otherwise specified, it is assumed that the focus parameters include direction, distance, radius, and amount. In some embodiments, the ratio modifier and spectral adjustment factor determiner 401 is formulated as follows: f and metadata position p m In some embodiments, v m (k) is determined as a column-wise vector pointing to the direction parameter of the spatial metadata in band k and as a column-wise vector pointing to the focus direction. The focus position is determined by p f =v f d f where d f is the focus distance. The spatial metadata position is formulated as being a distance parameter in the spatial metadata in band k. In some embodiments, the ratio modifier and spectral adjustment factor determiner 401 is configured to determine the difference between the focus position (one for every frequency band k) and the spatial metadata position, potentially different positions in different frequency bands k. The position difference is

number

[0171] The ratio modifier and spectral adjustment coefficient determiner 401 is then configured to determine the direct gain parameter f(k). The focus amount parameter may be expressed as a normalized value r0 between 0..1 (where 0 means zero focus / defocus and 1 means maximum focus / defocus), and the focus radius may be, for example, 1 meter at a certain time instance.

[0172] When the ratio modifier and spectral adjustment factor determiner 401 is configured to perform focusing (as opposed to defocusing), an exemplary gain formula is:

number

number

[0173] In some embodiments, the constant c may have different values ​​for the defocused and focused cases. Furthermore, in practice, it may be desirable to smooth the focus gain function so that it transitions smoothly from high values ​​in focused regions to low values ​​in unfocused regions.

[0174] The remaining description is applicable to both focus parameter configurations described above. In some embodiments, the ratio modifier and spectral adjustment factor determiner 401 is further configured to determine new direct part values ​​of the parametric spatial audio signal as follows:

number

number

[0175] In one embodiment, the ratio modifier and spectral adjustment factor determiner 401 is configured to determine the new ambient component in the defocusing process using A(k)=(1−r(k)), which means that the defocusing process does not affect the ambient energy spatially.

[0176] The ratio modifier and spectral adjustment coefficient determiner 401 is then configured to determine spectral correction coefficients that are output to the spectral adjustment processor 403, which are then formulated based on the overall modification of the acoustic energy. For example,

number

[0177] In some embodiments, the ratio modifier and spectral adjustment factor determiner 401 is

number

[0178] If it is not numerically determined, D(k)=A(k)=0, then r'(k) can also be set to zero.

[0179] The direction values ​​402 (and distance values ​​422) in the spatial metadata may be passed through and output unmodified in some embodiments.

[0180] The focus processor in some embodiments comprises a spectral adjustment processor 403. The spectral adjustment processor 403 is configured to receive an audio signal (which in some embodiments is a time-frequency representation, or alternatively, which is first transformed to the time-frequency domain) 406 and spectral adjustment coefficients 412. In some embodiments, the output audio signal 414 may also be in the time-frequency domain and may be transformed back to the time domain before being output. The domains of the input and output may depend on the implementation.

[0181] The spectral adjustment processor 403 is configured to, for each band k, multiply the frequency bins (of the time-to-frequency transform) of all channels in band k by a spectral adjustment factor s(k). In other words, the spectral adjustment processor 403 is configured to perform spectral adjustment. To avoid processing artifacts, the multiplications / spectral corrections can be smoothed over time.

[0182] In other words, the focus processor 450 is configured to modify the spectrum of the audio signal and the spatial metadata such that the procedure results in a parametric spatial audio signal modified according to the (de)focus parameters.

[0183] With reference to Figure 4b, a flow diagram 460 of the operation of a parametric spatial audio input processor such as that shown in Figure 4a is shown.

[0184] The initial operation is by step 461 receiving a parametric spatial audio signal (and focus / defocus parameters or other control information), as shown in Figure 4b.

[0185] The next action is to modify the parametric metadata and generate spectral adjustment coefficients, as shown in FIG. 4b by step 463.

[0186] The next action is to perform a spectral adjustment on the audio signal, as shown in FIG. 4b by step 465.

[0187] The spectrally adjusted audio signal and the modified (and unmodified) metadata may then be output by step 467 as shown in Figure 4b.

[0188] Referring to FIG. 5a, a focus processor 550 configured to receive a multi-channel or object audio signal as input 500 is shown. The focus processor in such an example may include a focus gain determiner 501. The focus gain determiner 501 is configured to receive focus / defocus parameters 508 and channel / object position / orientation information, which may be static or time-varying. Based on the (de)focus parameters 508 (such as (de)focus direction, (de)focus amount, (de)focus control, and optionally (de)focus distance and radius or (de)focus width) and spatial metadata information 502 from the input signal 500, the focus gain determiner 501 signals, or in some embodiments assumes, a channel signal direction in some embodiments. For example, if six channels are present, the direction may be assumed to be a 5.1 audio channel direction. In some embodiments, there may be a lookup table used to determine the channel direction as a function of the number of channels.

[0189] In some embodiments, there is no filter bank, i.e., there is only one frequency band k. The direct gain f(k) for each audio channel is output to the focus gain processor 503 as a focus gain.

[0190] In some embodiments, the focus gain processor 503 is configured to receive the audio signal and the focus gain values ​​512 and process the audio signal 506 based on the focus gain values ​​512 (per channel), potentially with some temporal smoothing. The processing based on the focus gain values ​​512 may in some embodiments be a multiplication of the focus gain values ​​with the channel / object signal.

[0191] The output of the focus gain processor 503 is the focus processed audio channel. The channel direction / position information is unchanged and is also provided as output 510.

[0192] In some embodiments, the defocus process can be configured to be more extensive than one direction. For example, a focus width may be included as an input parameter. In these embodiments, a user may generate a defocus arc. In another example, a focus distance and a focus radius may be included as input parameters. In these embodiments, a user may generate a defocus sphere at a determined location. Similar procedures may be employed for other input spatial audio signal types.

[0193] In some embodiments, audio objects (spatial metadata) may include distance parameters, which may also be taken into account. For example, focus / defocus parameters may determine the focus location (direction and distance), and a radius parameter may control the focus / defocus region around that location. In such embodiments, a user may generate a defocus pattern such as that shown in FIG. 1c and described above. Similarly, other spatially related parameters may be defined to allow a user to control different shapes of the defocus region. In some embodiments, attenuation of audio objects within the defocus region may be achieved by multiplying the attenuation by a fixed number of decibels (e.g., 10 dB) by a desired amount of defocus between 0 and 1, leaving audio objects outside the defocus direction. A direct gain f(k) parameter 512 may be generated without gain modification (or without applying any focus-related gain or attenuation to audio objects outside the defocus direction). In formulating the direct gain (output as focus gain), the focus gain determiner 501 can utilize the same formula as described in the context of the ratio modifier and spectral adjustment coefficient determiner 401 of Figure 4a to determine the direct gain f(k). The exception is in the case of audio objects / channels, where typically only one frequency band is present and the spatial metadata typically indicates only the direction / distance of the object, not the ratio. If the distance is not available, a fixed distance can be assumed, e.g., 2 meters.

[0194] FIG. 5b shows a flow chart 560 of the operation of the multi-channel / object audio input processor shown in FIG. 5a.

[0195] The initial operation is to receive a multi-channel / object audio signal and, in some embodiments, to receive channel information such as the number of channels and / or channel distribution (and focus / defocus parameters or other control information), as shown in FIG. 5b by step 561.

[0196] The next operation generates the focus gain factor as shown in FIG. 5b by step 563.

[0197] The next operation is to apply a focus gain to each channel audio signal, as shown in FIG. 5b by step 565.

[0198] Step 567 can then output the processed audio signal and the unaltered channel direction (and distance), as shown in Figure 5b.

[0199] With reference to Figure 6a, an example of a playback processor 650 based on Ambisonic audio input is shown (which may be configured to receive the output from a sample focus processor, for example as shown in Figure 3a).

[0200] In these examples, the playback processor may comprise an Ambisonic rotation matrix processor 601. The Ambisonic rotation matrix processor 601 is configured to receive the Ambisonic signal with focus / defocus processing 600 and view direction 602. The Ambisonic rotation matrix processor 601 is configured to generate a rotation matrix based on the view direction parameter 602. This may, in some embodiments, be any suitable method, such as that applied in head-tracking Ambisonic binauralization (or more generally, as such rotations of spherical harmonics are used in many fields, including outside of audio). The rotation matrix is ​​then applied to the Ambisonic audio signal. The result is a rotated Ambisonic signal plus focus / defocus 604, which is output to the Ambisonic-to-binaural filter 603.

[0201] The Ambisonic-to-Binaural Filter 603 is configured to receive the rotated Ambisonic signal with added focus / defocus 604. The Ambisonic-to-Binaural Filter 603 may include a pre-formulated 2xK matrix of finite impulse response (FIR) filters to be applied to the K Ambisonic signals to generate two binaural signals 606. In this example, where a four-channel FOA audio signal is shown, K=4. The FIR filters may be generated by a least-squares optimization method with respect to a set of Head-Responsive Impulse Responses (HRIRs). One example of such a design procedure is to transform the HRIR dataset into frequency bins (e.g., by FFT) to obtain an HRTF dataset, and for each frequency bin, determine, in a least-squares manner, a complex-valued processing matrix that approximates the HRTF dataset available at the data points of the HRTF dataset. When the complex-valued matrix for all frequency bins is determined in this manner, the result can be inverted (e.g., by reciprocal FFT) as a time-domain FIR filter. The FIR filters may also be windowed, for example, by using a Hann window.

[0202] In some embodiments, the rendering is for loudspeakers rather than headphones. There are many known methods that can be used to render Ambisonic signals to loudspeaker outputs. One example may be linear decoding of the Ambisonic signals to a target loudspeaker configuration. This can be applied with good expected spatial fidelity if the order of the Ambisonic signals is sufficiently high, e.g., at least third order, preferably fourth order. In a specific example of such linear decoding, an Ambisonic decoding matrix may be designed that, when applied to the Ambisonic signals (corresponding to Ambisonic beam patterns), generates loudspeaker signals corresponding to beam patterns that, in a least-squares sense, approximate a vector-based amplitude panning (VBAP) beam pattern appropriate for the target loudspeaker configuration. Processing the Ambisonic signals with such a designed Ambisonic decoding matrix can be configured to generate loudspeaker audio output. In such an embodiment, the playback processor is configured to receive information about the speaker configuration, and no rotation processing is required.

[0203] FIG. 6b shows a flow chart 660 of the operation of the Ambisonic input playback processor shown in FIG. 6a.

[0204] The initial operation is via step 661 receiving a focused / defocused Ambisonic audio signal (and view direction) as shown in Figure 6b.

[0205] The next operation is one of generating a rotation matrix based on the view direction as shown in FIG. 6b by step 663.

[0206] The next operation is to apply a rotation matrix to the Ambisonic audio signal to generate a rotated Ambisonic audio signal with focus / defocus processing as shown in FIG. 6b by step 665.

[0207] The next operation is then to convert the signal into a suitable audio output format, for example a binaural format (or multi-channel audio format or loudspeaker format), as shown in FIG. 6b by step 667.

[0208] Next, step 669 outputs the output audio format as shown in Figure 6b.

[0209] With reference to FIG. 7a, an example of a playback processor 750 based on parametric spatial audio input (which may be configured to receive output from the exemplary focus processor shown in FIG. 4a, for example) is shown.

[0210] In some embodiments, the playback processor includes a filter bank 701 configured to receive the audio channel 700 audio signal (unless the input is already in the appropriate time-frequency domain) and transform the audio channel into frequency bands. Examples of suitable filter banks include short-time Fourier transform (STFT) and complex quadrature mirror filter (QMF) banks. The time-frequency audio signal 702 can be output to a parametric binaural synthesizer 703.

[0211] In some embodiments, the playback processor comprises a parametric binaural synthesizer 703 configured to receive a time-frequency audio signal 702, modified (and unmodified) metadata 704, and a view direction 706 (or appropriate playback-related control or tracking information). In the context of 6DOF playback, the user position may be provided along with the view direction parameter.

[0212] The parametric binaural synthesizer 703 can be configured to implement any suitable known parametric spatial synthesis method configured to generate a binaural audio signal (in frequency bands) 708, since focus correction has already been performed on the signal and metadata before the parametric binauralization block. One known method for parametric binaural synthesis is to divide the time-frequency audio signal 702 into frequency-band direct and ambient partial signals based on frequency-band direct-to-total ratio parameters, process the frequency-band direct portion with HRTFs corresponding to frequency-band directional parameters, process the ambient portion with a decorrelator to obtain binaural diffuse sound field coherence, and combine the processed direct and ambient portions. The binaural audio signal (in frequency bands) 708 has two channels, regardless of how many channels the time-frequency audio signal 702 has. The binauralized time-frequency audio signal 708 can then be passed to an inverse filter bank 705. The embodiment may further feature a reproduction processor including an inverse filter bank 705 configured to receive the binauralized time-frequency audio signal 708 and apply the inverse to the applied forward filter bank, thus producing a time-domain binauralized audio signal 710 having focusing characteristics suitable for reproduction by headphones (not shown in Figure 7a).

[0213] In an embodiment, the binaural audio signal output is replaced with loudspeaker channel audio signals output from the parametric spatial audio signal using a suitable loudspeaker synthesis method. Any suitable approach can be used, for example, the view direction parameters are replaced with information about the loudspeaker positions, and the parametric binaural synthesizer 703 is replaced with a parametric loudspeaker synthesizer based on a suitable known method. One known method for parametric loudspeaker synthesis is to divide the time-frequency audio signal 702 into direct and ambient partial signals in frequency bands based on a direct-to-sum ratio parameter in the frequency band, process the direct partial in the frequency band with a vector-based amplitude panning (VBAP) gain corresponding to the loudspeaker configuration and the direction parameter in the frequency band, process the ambient partial with a decorrelator to obtain an incoherent loudspeaker signal, and combine the processed direct and ambient partials. The loudspeaker audio signal (in frequency bands) has a number of channels that is determined by the loudspeaker configuration, regardless of how many channels the time-frequency audio signal 702 has.

[0214] With reference to Figure 7b, a flow diagram 760 of the operation of a parametric spatial audio input reproduction processor as shown in Figure 7a is shown.

[0215] The first operation is to receive the focus / defocus processed parametric spatial audio signal (and view direction or other playback related control or tracking information), as shown in FIG. 7b by step 761.

[0216] The next operation is one of time-frequency transforming the audio signal as shown in FIG. 7b by step 763.

[0217] The next operation is to apply a parametric binaural (or loudspeaker channel format) processor based on the time-frequency transformed audio signal, metadata and gaze direction (or other information), as shown in FIG. 7b by step 765.

[0218] The next operation is then, by step 767, to inverse transform the generated binaural or loudspeaker channel audio signals as shown in Figure 7b.

[0219] Next, step 769 outputs the output audio format as shown in Figure 7b.

[0220] Considering the speaker output for a playback processor when the audio signal is in the form of multi-channel audio and the focus processor 550 of FIG. 5a is applied, in some embodiments, the playback processor can include a pass-through where the output loudspeaker configuration is the same as the format of the input signal. In some embodiments where the output loudspeaker configuration is different from the input loudspeaker configuration, the playback processor can include a vector-based amplitude panning (VBAP) processor. Each of the focus-processed audio channels can then be processed using VBAP, a known amplitude panning technique, to spatially reproduce them using a target loudspeaker configuration. In this way, the output audio signal matches the output loudspeaker setup.

[0221] In some embodiments, the conversion from the first loudspeaker configuration to the second loudspeaker configuration may be performed using any suitable amplitude panning technique. For example, the amplitude panning technique may include deriving an N×M matrix of amplitude panning gains that defines the conversion from the M channels of the first loudspeaker configuration to the N channels of the second loudspeaker configuration, and then using the matrix to multiply the channels of an intermediate spatial audio signal that is provided as a multi-channel loudspeaker signal according to the first loudspeaker configuration. The intermediate spatial audio signal can be understood to be similar to an audio signal with focused / defocused sound components 204, as shown in FIG. 2a. As a non-limiting example, the derivation of VBAP amplitude panning gains is provided in Ville, Pulkki: "Virtual Sound Source Localization Using Vector-Based Amplitude Panning," Journal of Audio Engineering 45, no. 6 (1997), pp. 456-466.

[0222] For binaural output, any suitable binauralization of multi-channel loudspeaker signal formats (and / or objects) may be implemented. For example, typical binauralization may include processing audio channels with head-related transfer functions (HRTFs) and adding synthetic room reverberation to create the auditory impression of a listening room. The distance and direction (i.e., location) information of audio object sounds can be utilized for 6DOF playback with user movement, for example, by employing the principles outlined in UK Patent Application GB1710085.0.

[0223] An exemplary apparatus suitable for implementation is shown in Figure 8 in the form of a mobile phone or mobile device 901 running suitable software 903. Video can be played, for example, by attaching the mobile phone 901 to a Daydream view type device (for clarity, video processing is not described here).

[0224] The audio bitstream obtainer 923 is configured to obtain an audio bitstream 924, for example, received / retrieved from a memory. In some embodiments, the mobile device includes a decoder 925 configured to receive and decode compressed audio. An example of a decoder is an AAC decoder in the case of AAC decoding. The resulting decoded (e.g., Ambisonic (Ambisonic) implementing the examples shown in Figures 3a and 6a) audio signal 926 can be forwarded to a focus processor 927.

[0225] The mobile phone 901 receives controller data 900 from an external controller at a controller data receiver 911 (e.g., via Bluetooth) and passes the data to a focus parameter (from controller data) determiner 921. The focus parameter (from controller data) determiner 921 determines the focus parameters based, for example, on the orientation of the controller device and / or button events. The focus parameters can include any type of combination of proposed focus parameters (e.g., focus / defocus direction, focus / defocus amount, focus / defocus height, and focus / defocus width). The focus parameters 922 are forwarded to a focus processor 927.

[0226] Based on the Ambisonic audio signal and the focus parameters, 927 is configured to generate modified Ambisonic signals 928 having desired focus characteristics. These modified Ambisonic signals 928 are forwarded to the Ambisonic processor 929. The Ambisonic binaural processor 929 is also configured to receive head orientation information 904 from the orientation tracking device 913 of the mobile phone 901. Based on the modified Ambisonic signal 928 and the head direction information 904, the Ambisonic / binaural processor 929 is configured to generate a head-tracked binaural signal 930 that can be output from the mobile phone and played back using, for example, headphones.

[0227] 9 shows an example device (or focus / defocus parameter control unit) 1050 that can be configured to control or generate appropriate focus / defocus parameters, such as focus / defocus direction, focus / defocus amount, and focus / defocus width. A user of the device can be configured to select a focus direction by pointing the controller in a desired direction 1009 and pressing a focus direction selection button 1005. The control unit has an orientation tracker 1001, and orientation information may be used to determine the focus / defocus direction (e.g., in a focus parameter determiner 921 (from control unit data) as shown in FIG. 8). The focus / defocus direction in some embodiments can be visualized in a visual display while selecting the focus / defocus direction.

[0228] In some embodiments, the focus amount can be controlled using focus amount buttons (shown as + and - in FIG. 9) 1007. Each press increases or decreases the focus amount by, for example, 10 percentage points. In some embodiments, the focus amount is set to 0% and the user presses the minus button, the focus amount is set to 10%, and the focus / defocus control is set to "defocus" mode; correspondingly, if the focus amount is set to 0% and the user presses the plus button, the focus amount is set to 10%, and the focus / defocus control is set to "focus" mode.

[0229] In some embodiments, it may be desirable to further specify the focus or defocus processing, for example, by determining the desired frequency range or spectral characteristics of the focus signal. In particular, it may be useful to emphasize or de-emphasize the audio spectrum in the audio frequency range to improve intelligibility or to block out a talker, for example, by attenuating low frequency content (e.g., below 200 Hz) and high frequency content (e.g., above 8 kHz), thus leaving a particularly useful frequency range associated with the audio.

[0230] Similarly, when a user indicates a direction to be defocused, the audio processing system can analyze the spectrum or type (e.g., speech, noise) of interference in the direction to be attenuated. Then, based on this analysis, the system could determine a frequency range or amount of defocus per frequency that best suits that interferer. For example, the interferer may be a device that generates high-frequency noise, and the high frequencies for that defocus direction would be attenuated more than, for example, mid- and low-frequencies. In another example, the defocus direction may have a speaker, and therefore the amount of defocus can be configured per frequency to primarily suppress typical audio frequency ranges.

[0231] It will be appreciated that the focus processed signal may be further processed with any known audio processing technique, such as automatic gain control or enhancement techniques (eg, bandwidth extension, noise suppression).

[0232] In some further embodiments, focus / defocus parameters (including direction, amount, and control) are generated by the content creator, and the parameters are transmitted along with the spatial audio signal. For example, in a VR video / audio documentary with an on-site commentator, a dynamic focus parameter preset can be selected instead of the user having to select the commentator's direction to be defocused. The preset can be fine-tuned by the content creator to follow the commentator's movements. For example, defocus is enabled only when the commentator is speaking. In other words, the content creator can generate several expected or estimated preference profiles for the focus / defocus parameters. This approach is beneficial because it only requires transmitting one spatial audio signal, but different preference profiles can be added. Legacy players that are not focus-enabled can be configured to simply decode Ambisonic or other signal types without applying focus / defocus processing.

[0233] An exemplary processing output is shown in FIG. 10 based on the described implementation for Ambisonic signals. In this example, three sound sources are in the audio scene: a talker in front, a talker at -90 degrees to the right, and a white noise interferer at 110 degrees to the left. FIG. 10 illustrates how, with the focus / defocus control set to "focus," focus processing is used to broadly emphasize the direction in which the noise source is located, and how, with the focus / defocus control set to "defocus," focus processing is used to broadly de-emphasize the direction in which the noise source is located while preserving the two talker signals in the spatial audio output. Thus, Ambisonic signals are shown in three columns (omni W 1101, horizontal dipole Y 1103 and X 1105) for an exemplary situation illustrated by the Ambisonic signal in row 1111 with a talker in front (particularly indicated by signal X), a talker at -90 degrees to the right (particularly indicated by signal Y), and a noise interferer at 110 degrees to the left (indicated by all signals). The next column 1113 shows the Ambisonic audio signal with full focus towards the noise source, while the bottom row 1115 shows the Ambisonic audio signal with full defocus towards the noise source (i.e., de-emphasis of the noise), leaving the majority of the speech source active.

[0234] Referring to Figure 11, an exemplary electronic device that can be used as an analysis or synthesis device is shown. The device may be any suitable electronic device or apparatus. For example, in some embodiments, device 1700 is a mobile device, user equipment, tablet computer, computer, audio playback device, etc. In some embodiments, device 1200 includes at least one processor or central processing unit 1207. Processor 1207 may be configured to execute various program code, such as the methods described herein.

[0235] In some embodiments, the apparatus 1200 comprises a memory 1211. In some embodiments, at least one processor 1207 is coupled to the memory 1211. The memory 1211 may be any suitable storage means. In certain embodiments, the memory 1211 includes program code sections for storing program code implementable on the processor 1207. Additionally, in some embodiments, the memory 1211 may further comprise a stored data section for storing data, e.g., data that has been processed or is to be processed in accordance with embodiments described herein. The implemented program code stored in the program code sections and the data stored in the stored data sections may be retrieved by the processor 1207 whenever needed via the memory-processor coupling.

[0236] In some embodiments, device 1200 comprises a user interface 1205. User interface 1205, in some embodiments, may be coupled to processor 1207. In some embodiments, processor 1207 may control the operation of user interface 1205 and receive input from user interface 1205. In some embodiments, user interface 1205 may allow a user to input commands into device 1200, for example, via a keypad. In some embodiments, user interface 1205 may allow a user to obtain information from device 1200. For example, user interface 1205 may include a display configured to display information from device 1200 to the user. User interface 1205, in some embodiments, may comprise a touchscreen or touch interface that can both allow information to be input into device 1200 and also display information to the user of device 1200.

[0237] In some embodiments, apparatus 1200 comprises an input / output port 1209. The input / output port 1209, in some embodiments, comprises a transceiver. The transceiver in such embodiments may be coupled to processor 1207 and configured to enable communication with other apparatuses or electronic devices, for example, via a wireless communication network. The transceiver or any suitable transceiver or transmitter and / or receiver means, in some embodiments, may be configured to communicate with other electronic devices or apparatuses via a wire or wired coupling.

[0238] The transceiver may communicate with the further device by any suitable known communication protocol, for example, in some embodiments the transceiver may use a suitable Universal Mobile Telecommunications System (UMTS) protocol, a Wireless Local Area Network (WLAN) protocol such as IEEE 802.X, a suitable short-range radio frequency communication protocol such as Bluetooth, or an Infrared Data Path (IRDA).

[0239] The transceiver input / output port 1209 can be configured to receive signals, and in some embodiments, obtain focus parameters, as described herein.

[0240] In some embodiments, device 1200 may be used to generate appropriate audio signals by using processor 1207 executing appropriate code. Input / output port 1209 may be coupled to any suitable audio output, for example, a multi-channel speaker system and / or headphones (which may be head-tracked or non-tracked headphones) or the like.

[0241] In general, various embodiments of the present invention may be implemented in hardware or special-purpose circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although various aspects of the present invention may be illustrated and depicted as block diagrams, flowcharts, or using some other pictorial representations, it should be appreciated that these blocks, devices, systems, techniques, or methods depicted herein may be implemented in, by way of non-limiting example, hardware, software, firmware, special-purpose circuits or logic, general-purpose hardware or controller, or other computing device, or some combination thereof.

[0242] Embodiments of the present invention can be implemented by computer software executable by a data processor of a mobile device, such as within a processor entity, or by hardware, or by a combination of software and hardware. It should be further noted in this regard that any blocks of logic flows, such as those shown, can represent program steps, or interconnected logic circuits, blocks and functions, or combinations of program steps and logic circuits, blocks and functions. This software can be stored on physical media, such as memory chips, or memory blocks implemented within a processor, magnetic media, such as a hard disk or floppy disk, and optical media, such as a DVD or its data variants.

[0243] The memory may be of any type suitable for the local technology environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed and removable memory, etc. The data processor may be of any type suitable for the local technology environment and may include, by way of non-limiting examples, one or more of a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a gate-level circuit, and a processor based on a multi-core processor architecture.

[0244] Embodiments of the present invention can be implemented in a variety of components, such as integrated circuit modules. The design of integrated circuits is a highly automated process and is large-scale. Complex and powerful software tools are available to convert logic-level designs into complete semiconductor circuit designs ready to be etched and formed on semiconductor substrates.

[0245] Programs such as those offered by Synopsys, Inc., of Mountain View, California, and Cadence Design, San Jose, California, use well-established rules of design and a library of pre-stored design modules to automatically route conductors and identify the location of components on a semiconductor chip. Once the design of a semiconductor circuit is complete, the resulting design in a standardized electronic format (e.g., Opus, GDSII, etc.) may be sent to a semiconductor manufacturing facility or "fab" for fabrication.

[0246] The foregoing description has provided a complete and informative description of exemplary embodiments of the present invention, by way of illustrative and non-limiting examples. However, various modifications and adaptations will become apparent to those skilled in the art in view of the foregoing description upon perusal of the accompanying drawings and the appended claims. However, all such similar modifications of the teachings of this invention will still fall within the scope of the present invention, as defined in the appended claims.

Claims

1. 1. An apparatus comprising at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code being configured to, using the at least one processor, cause the apparatus to perform at least: Obtaining a defocus direction; processing a spatial audio signal representing an audio scene using a defocus amount in the defocus direction to generate a processed spatial audio signal representing a modified audio scene so as to control a relative de-emphasis of at least some portions of the spatial audio signal in the defocus direction relative to at least some other portions of the spatial audio signal; outputting the processed spatial audio signal, wherein the modified audio scene based on the defocus direction enables the de-emphasis of at least some of the portions of the spatial audio signal in the defocus direction relative to at least some of other portions of the spatial audio signal; configured to cause The apparatus is further configured to perform the step of obtaining the defocus amount; processing the spatial audio signal includes causing the device to perform a step of controlling a relative de-emphasis of at least a portion of the portion of the spatial audio signal in the defocus direction relative to at least a portion of another portion of the spatial audio signal according to the amount of defocus. Device.

2. Processing the spatial audio signal may include: - reducing emphasis of at least some of the portions of the spatial audio signal in the defocus direction relative to at least some of other portions of the spatial audio signal; increasing emphasis of at least some of the portions of the spatial audio signal relative to the portions of the spatial audio signal in the defocus direction; The apparatus of claim 1 , further comprising:

3. processing the spatial audio signal includes causing the device to reduce a sound level in at least a portion of the portion of the spatial audio signal in the defocus direction relative to at least a portion of another portion of the spatial audio signal according to the defocus amount; increasing a sound level of at least some of the portions of the spatial audio signal relative to other portions of the spatial audio signal in the defocus direction according to the defocus amount; The apparatus of claim 1 , further comprising:

4. 10. The apparatus of claim 1, further comprising the step of: causing the apparatus to acquire a defocus shape, processing the spatial audio signal includes causing the device to perform a step of controlling relative de-emphasis of at least a portion of the portion of the spatial audio signal, in the defocus direction and within the defocus shape, relative to at least a portion of another portion of the spatial audio signal. Device.

5. processing the spatial audio signal causes the device to reduce emphasis on at least some of the portions of the spatial audio signal from in the defocus direction and within the defocus shape relative to at least some of other portions of the spatial audio signal; increasing emphasis of at least some of the other portions of the spatial audio signal relative to the portion of the spatial audio signal within the defocus direction and the defocus shape of the spatial audio signal; The apparatus of claim 4 , further comprising:

6. processing the spatial audio signal, - reducing the sound level in at least a portion of the portion of the spatial audio signal relative to at least a portion of another portion of the spatial audio signal in a defocus direction of the spatial audio signal and according to a defocus amount from within the obtained defocus shape; increasing a relative sound level from within the obtained defocus shape in at least a portion of another portion of the spatial audio signal according to the portion of the defocus direction of the spatial audio signal and the defocus amount; The apparatus of claim 1 , further comprising causing the apparatus to perform at least one of the following:

7. further configured to perform the step of obtaining playback control information for controlling at least one aspect of outputting the processed spatial audio signal; the device is adapted to perform the step of outputting the processed spatial audio signal; Processing the spatial audio signal further comprises causing the device to: - processing the processed spatial audio signals representing the modified audio scene based on the defocus direction to generate an output spatial audio signal in accordance with the playback control information; or processing the spatial audio signal in accordance with the playback control information before causing the device to process the spatial audio signal representing the audio scene to generate the processed spatial audio signal representing the modified audio scene based on the defocus direction and output the processed spatial audio signal as the output spatial audio signal; The apparatus of claim 1 , further comprising:

8. the spatial audio signal and the processed spatial audio signal comprise respective Ambisonic signals; Processing the spatial audio signal includes, for one or more frequency sub-bands, causing the device to: extracting from the spatial audio signal a single-channel target audio signal representing a sound component arriving from a focus direction; generating a focused spatial audio signal, the focused spatial audio signal being positioned at a spatial position defined by the defocus direction; or generating the processed spatial audio signal as a linear combination of the focused spatial audio signal subtracted from the spatial audio signal, wherein at least one of the focused spatial audio signal and the spatial audio signal is scaled by a respective scaling factor derived based on the amount of defocus so as to reduce the relative level of sound in the direction of defocus; The apparatus of claim 1 , further comprising:

9. Extracting the single-channel target audio signal comprises: applying beamforming to derive a beamformed signal representing the sound component arriving from the defocus direction from the spatial audio signal; or applying a post-filter to derive the processed spatial audio signal based on the beamformed signal, thereby adjusting the spectrum of the beamformed signal to approximate the spectrum of the sound arriving from the defocused direction; including causing 9. The apparatus of claim 8.

10. the spatial audio signal and the processed spatial audio signal comprise respective parametric spatial audio signals; the parametric spatial audio signal comprises one or more audio channels and spatial metadata; the spatial metadata includes respective directional indicators and energy ratio parameters for a plurality of frequency subbands; Processing the spatial audio signal causes the device to: - calculating, for one or more frequency sub-bands, a respective angular difference between the defocus direction and a direction indicated for each frequency sub-band of the spatial audio signal; deriving a gain value for each of the one or more frequency subbands based on the angular difference calculated for each of the frequency subbands using a predetermined function of the angular difference and a scaling factor derived based on the amount of defocus; calculating, for one or more frequency subbands of the processed spatial audio signal, respective updated directional energy values ​​based on the energy ratio parameter and the gain value for the respective frequency subband of the spatial audio signal; calculating, for the one or more frequency bands of the processed spatial audio signal, respective updated ambient energy values ​​based on the scaling factor and an energy ratio parameter of the respective frequency sub-bands of the spatial audio signal; calculating a modified energy ratio parameter for each of the one or more frequency sub-bands of the processed spatial audio signal based on the updated directional energy divided by the sum of the updated direct energy and ambient energy; calculating a spectral adjustment factor for each of the one or more frequency sub-bands of the processed spatial audio signal based on the updated sum of direct and ambient energy; or constructing a directed processed spatial audio signal comprising the one or more audio channels of the spatial audio signal, the directional indication of the spatial audio signal, a modified energy ratio parameter, and the spectral adjustment coefficients; including causing 10. The apparatus of claim 1.

11. the spatial audio signal and the processed spatial audio signal comprise respective parametric spatial audio signals, the parametric spatial audio signals comprise one or more audio channels and spatial metadata, the spatial metadata comprise respective directional indications and energy ratio parameters for a plurality of frequency subbands; where: Processing the spatial audio signal causes the device to: - calculating, for one or more frequency sub-bands, a respective angular difference between the defocus direction and a direction indicated for each frequency sub-band of the spatial audio signal; deriving a gain value for each of the one or more frequency sub-bands based on the angular difference calculated for each frequency sub-band using a predetermined function of the angular difference and a scaling factor derived based on the amount of defocus; calculating, for one or more frequency subbands of the processed spatial audio signal, respective updated directional energy values ​​based on an energy ratio parameter of each frequency subband of the spatial audio signal and the gain value; calculating, for the one or more frequency bands of the processed spatial audio signal, respective updated ambient energy values ​​based on the energy ratio parameters of the respective frequency sub-bands of the spatial audio signal and on the scaling factor; calculating a modified energy ratio parameter for each of one or more frequency sub-bands of the processed spatial audio signal based on the updated directional energy divided by the sum of the updated direct energy and ambient energy; calculating respective spectral adjustment factors for one or more frequency subbands of the processed spatial audio signal based on the updated sum of direct and ambient energies; deriving one or more extended audio channels in the one or more frequency sub-bands by multiplying a frequency band of each of one or more audio channels of the spatial audio signal by the spectral adjustment coefficient derived for the respective frequency band; or constructing the processed spatial audio signal including the one or more extended audio channels, the directional indication of the spatial audio signal, and the modified energy ratio parameter; including causing 10. The apparatus of claim 1.

12. the spatial audio signal and the processed spatial audio signal include respective multi-channel loudspeaker signals according to a first predetermined loudspeaker configuration; Processing the spatial audio signal may include: - calculating a respective angular difference between said defocus direction and a loudspeaker direction indicated for each channel of said spatial audio signal; deriving a respective gain value for each channel of the spatial audio signal based on the angular difference calculated for each channel using a predetermined function of the angular difference and a scaling factor derived based on the amount of defocus; deriving one or more modified audio channels by multiplying the respective channels of the spatial audio signal by the derived gain value for the respective channel; or providing the modified audio channels as a processed spatial audio signal; including causing 10. The apparatus of claim 1.

13. the processed spatial audio signal comprises an Ambisonic signal; the output spatial audio signal comprises a two-channel binaural signal; the playback control information includes a playback direction indication that defines a listening direction for the audio scene; processing the spatial audio signals representing the modified audio scene based on the defocus direction to generate an output spatial audio signal in accordance with the playback control information; generating a rotation matrix that depends on the indicated playback orientation; multiplying the channels of the processed spatial audio signal with the rotation matrix to derive a rotated spatial audio signal; filtering the channels of the rotated spatial audio signal using a predetermined set of Finite Impulse Response (FIR) filter pairs generated based on a dataset of Head-Related Impulse Response Functions (HRTFs) or Head-Related Impulse Responses (HRIRs); or generating the left and right channels of the two-channel binaural signal as a sum of the filtered channels of the rotated spatial audio signal derived for each one of the left and right channels; including causing 8. The apparatus of claim 7.

14. the output spatial audio signal comprises a two-channel binaural audio signal; the playback control information includes a playback direction indication defining a listening direction for the audio scene; processing the spatial audio signals representing the modified audio scene based on the defocus direction to generate the output spatial audio signals in accordance with the playback control information, deriving one or more extended audio channels in the one or more frequency subbands by multiplying a respective one frequency band of one or more audio channels of the processed spatial audio signal by the received spectral adjustment coefficient for the respective frequency subband; converting the one or more enhanced audio channels into the two-channel binaural audio signal according to the indicated playback direction; and performing at least one of:

8. The apparatus of claim 7.

15. the output spatial audio signal comprises a two-channel binaural signal; the playback control information includes a playback direction indication defining a listening direction for the audio scene; 8. The apparatus of claim 7, wherein the processed spatial audio signal representing the modified audio scene based on a defocus direction to generate the output spatial audio signal in accordance with the playback control information is configured to select a set of head-related transfer functions, HRTFs, depending on an indicated playback direction, or to convert channels of the processed spatial audio signal into a two-channel binaural signal conveying a rotated audio scene using the selected set of HRTFs.

16. the playback control information includes an indication of a second predetermined loudspeaker configuration; the output spatial audio signals include multi-channel loudspeaker signals according to the second predetermined loudspeaker configuration; deriving, according to the playback control information, a transformation matrix including amplitude panning gains that provides a mapping from a first predetermined loudspeaker configuration to a second predetermined loudspeaker configuration, wherein the processed spatial audio signals representing the modified audio scene based on the defocus direction to generate the output spatial audio signals; configured to use the transformation matrix to multiply the channels of the output spatial audio signal by the channels of the processed spatial audio signal, thereby deriving the channels of the output spatial audio signal based on the channels of the processed spatial audio signal using amplitude panning.

8. The apparatus of claim 7.

17. obtaining a defocus input from a sensor arrangement comprising at least one direction sensor and at least one user input, the defocus input comprising at least one of an indication of a defocus direction based on a direction of the at least one direction sensor, an indication of a defocus amount based on the at least one user input, or an indication of a resulting defocus shape; The apparatus of claim 1 further configured to perform:

18. 5. The apparatus of claim 4, wherein the defocus shape comprises at least one of a defocus shape width, a defocus shape height, a defocus shape radius, a defocus shape distance, a defocus shape depth, a defocus shape range, a defocus shape diameter, and a defocus shape characterizer.

19. 1. A method for an apparatus, comprising: Obtaining a defocus direction; processing a spatial audio signal representing an audio scene such that a defocus amount in the defocus direction is used to control a relative de-emphasis of at least some of the portions of the spatial audio signal in the defocus direction relative to at least some of the other portions of the spatial audio signal to generate a processed spatial audio signal representing a modified audio scene; outputting the processed spatial audio signal, wherein the modified audio scene based on the defocus direction enables the de-emphasis of at least some of the portions of the spatial audio signal in the defocus direction relative to at least some of other portions of the spatial audio signal; A method comprising: further, obtaining the defocus amount; controlling, by the processed spatial audio signal, a relative de-emphasis of at least some of the portions of the spatial audio signal in the defocus direction relative to at least some of other portions of the spatial audio signal at least in part according to the de-focus amount; A method comprising:

Images