Sound field related rendering method and device

Abstract

An apparatus configured to: determine at least one region within an audio scene represented with a spatial audio signal; obtaining at least one piece of focusing / defocusing information; based at least in part on the at least one piece of focus / defocus information, processing the spatial audio signal to generate a processed spatial audio signal representing a modified audio scene, at least in part in the modified audio scene, at least one determined region is relatively de-emphasized at least partially with respect to other portions of the spatial audio signal; and outputting the processed spatial audio signal wherein the modified audio scene comprises at least one determined region that is at least partially de-emphasized with respect to other portions of the spatial audio signal according to at least one focus / defocus information.

Description

[0001] This application is a divisional application of the Chinese invention patent application entitled “A method and device for sound field related rendering” (application number 202080042725.0, application date June 3, 2020). Technical Field

[0002] The present application relates to apparatus and methods for audio representation and rendering associated with a sound field, but not exclusively to apparatus and methods for audio representation for an audio decoder. Background Art

[0003] Spatial audio playback of media presented with multiple viewing directions is known. Examples of such playback include viewing visual content of such media, including playback in the following ways: on a head-mounted display (or head-mounted phone) with (at least) head orientation tracking; or on a non-head-mounted phone screen, where the viewing direction can be tracked by changing the position / orientation of the phone or by any user interface gesture; or on a surrounding screen.

[0004] Videos associated with "media with multiple viewing directions" may be, for example, 360-degree videos, 180-degree videos, or other videos with a much wider viewing angle than traditional videos. Traditional video refers to video content that is typically displayed in its entirety on a screen without the option (or any specific need) to change the viewing direction.

[0005] Audio associated with a video with multiple viewing directions may be presented on headphones, where the viewing directions are tracked and affect the spatial audio playback, or may be presented with a surround speaker setup.

[0006] Spatial audio associated with video with multiple viewing directions can be derived from spatial audio captured from a microphone array (e.g., an array mounted on an OZO-like VR camera or a handheld mobile device), or from other sources such as a studio mix. The audio content can also be a mix of several content types such as sound captured by a microphone and an added commentator track.

[0007] Spatial audio associated with videos with multiple viewing directions can take various forms, such as: an Ambisonic signal (arbitrary order) composed of spherical harmonic audio signal components. Spherical harmonics can be considered as a set of spatially selective beam signals. Ambisonics is currently used, for example, in the YouTube 360VR video service. The advantage of Ambisonics is that it is a simple and well-defined signal representation; surround speaker signals, such as 5.1. Currently, the spatial audio of a typical movie is transmitted in this form. The advantages of surround speaker signals are simplicity and legacy compatibility. Some audio formats similar to the surround speaker signal format include audio objects, which can be regarded as audio channels with time-varying positions. The position can inform both the direction and distance of the audio object, or the direction; parameterized spatial audio, such as two audio channel audio signals and associated spatial metadata in perceptually relevant frequency bands. Some of the most advanced audio coding methods and spatial audio capture methods apply this signal representation. The spatial metadata essentially determines how the audio signal should be reproduced spatially at the receiver end (e.g., to those directions at different frequencies). The advantages of parameterized spatial audio are its versatility, quality, and the ability to use low bitrate encoding. Summary of the invention

[0008] According to a first aspect, a device is provided, comprising components configured to: obtain a defocus direction; process a spatial audio signal representing an audio scene to generate a processed spatial audio signal representing a modified audio scene based on the defocus direction, so as to at least partially control a relative de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction; and output the processed spatial audio signal, wherein the modified audio scene based on the defocus direction at least partially enables de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction.

[0009] The component may further be configured to obtain a defocus amount, and wherein the component configured to process the spatial audio signal may be configured to at least partially control a relative de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in a defocus direction based on the defocus amount.

[0010] A component configured to process a spatial audio signal may be configured to perform at least one of: at least partially reducing an emphasis of a portion of the spatial audio signal relative to other portions of the spatial audio signal at least partially in a defocused direction; and at least partially increasing an emphasis of other portions of the spatial audio signal relative to a portion of the spatial audio signal in a defocused direction.

[0011] The component configured to process the spatial audio signal may be configured to perform at least one of: at least partially reducing a sound level of a portion of the spatial audio signal relative to other portions of the spatial audio signal at least partially in a defocus direction based on the defocus amount; and at least partially increasing a sound level of other portions of the spatial audio signal relative to a portion of the spatial audio signal in a defocus direction based on the defocus amount.

[0012] The component may further be configured to obtain a defocus shape, and wherein the component configured to process the spatial audio signal may be configured to at least partially control a relative de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in a defocus direction and within the defocus shape.

[0013] A component configured to process a spatial audio signal may be configured to perform at least one of: at least partially reducing an emphasis of a portion of the spatial audio signal relative to other portions of the spatial audio signal at least partially in a defocus direction and within a defocus shape; and at least partially increasing an emphasis of other portions of the spatial audio signal relative to a portion of the spatial audio signal in a defocus direction and within a defocus shape.

[0014] The component configured to process the spatial audio signal may be configured to perform at least one of: at least partially reducing a sound level of a portion of the spatial audio signal relative to other portions of the spatial audio signal in a defocus direction and within a defocus shape based on the defocus amount; and at least partially increasing a sound level of other portions of the spatial audio signal relative to a portion of the spatial audio signal in a defocus direction and within a defocus shape based on the defocus amount.

[0015] The component may be configured to obtain reproduction control information to control at least one aspect of outputting a processed spatial audio signal, and wherein the component configured to output the processed spatial audio signal may be configured to perform one of the following: processing the processed spatial audio signal representing a modified audio scene based on a defocus direction to generate an output spatial audio signal according to the reproduction control information; processing the spatial audio signal according to the reproduction control information before the component configured to process the spatial audio signal representing the audio scene based on the defocus direction to generate the processed spatial audio signal representing the modified audio scene and output the processed spatial audio signal as the output spatial audio signal.

[0016] The spatial audio signal and the processed spatial audio signal may include corresponding Ambisonic signals, and wherein the component configured to process the spatial audio signal into the processed spatial audio signal may be configured to, for one or more frequency subbands, perform the following operations: extract a single-channel target audio signal representing a sound component arriving from a focus direction from the spatial audio signal; generate a focused spatial audio signal, wherein the focused audio signal is arranged at a spatial position defined by the defocus direction; and create the processed spatial audio signal as a linear combination of the focused spatial audio signal subtracted from the spatial audio signal, wherein at least one of the focused spatial audio signal and the spatial audio signal is scaled by a corresponding scaling factor derived based on the defocus amount to reduce the relative level of the sound in the defocus direction.

[0017] A component configured to extract a single-channel target audio signal may be configured to: apply a beamformer to obtain a beamformed signal representing a sound component arriving from a defocused direction from a spatial audio signal; and apply a post-filter to obtain a processed audio signal based on the beamformed signal, thereby adjusting the spectrum of the beamformed signal to approximate the spectrum of the sound arriving from the defocused direction.

[0018] The spatial audio signal and the processed spatial audio signal may include corresponding first-order Ambisonic signals.

[0019] The spatial audio signal and the processed spatial audio signal may include a corresponding parameterized spatial audio signal, wherein the parameterized spatial audio signal may include one or more audio channels and spatial metadata, wherein the spatial metadata may include corresponding direction indications and energy ratio parameters for multiple frequency subbands, wherein the component configured to process the spatial audio signal to generate the processed spatial audio signal may be configured to: calculate, for one or more frequency subbands, a corresponding angular difference between a defocus direction and a direction indicated for a corresponding frequency subband of the spatial audio signal; derive, for one or more frequency subbands, a corresponding gain value based on the angular difference calculated for the corresponding frequency subband by using a predefined angular difference function and a scaling factor derived based on the defocus amount; calculate, for one or more frequency subbands of the processed spatial audio signal, a corresponding updated directional energy value based on the energy ratio parameters of the corresponding frequency subbands of the spatial audio signal and the gain value; calculate, for one or more frequency bands of the processed spatial audio signal, a corresponding updated ambient energy based on the energy ratio parameters of the corresponding frequency subbands of the spatial audio signal and the scaling factor. energy) value; calculating, for one or more frequency subbands of the processed spatial audio signal, a corresponding modified energy ratio parameter based on the updated directional energy divided by the sum of the updated directional energy and the updated ambient energy; calculating, for one or more frequency subbands of the processed spatial audio signal, a corresponding spectral adjustment factor based on the sum of the updated directional energy and the updated ambient energy; and composing a processed spatial audio signal comprising one or more audio channels of the spatial audio signal, the directional indication of the spatial audio signal, the modified energy ratio parameter, and the spectral adjustment factor.

[0020] The spatial audio signal and the processed spatial audio signal may include a corresponding parameterized spatial audio signal, wherein the parameterized spatial audio signal may include one or more audio channels and spatial metadata, wherein the spatial metadata may include corresponding direction indications and energy ratio parameters for multiple frequency subbands, wherein the component configured to process the spatial audio signal to generate the processed spatial audio signal may be configured to: calculate, for one or more frequency subbands, a corresponding angular difference between a defocus direction and a direction indicated for a corresponding frequency subband of the spatial audio signal; derive, for one or more frequency subbands, a corresponding gain value based on the angular difference calculated for the corresponding frequency subband by using a predefined angular difference function and a scaling factor derived based on the defocus amount; calculate, for one or more frequency subbands of the processed spatial audio signal, a corresponding updated directional energy value based on the energy ratio parameters of the corresponding frequency subbands of the spatial audio signal and the gain value; For one or more frequency bands of the processed spatial audio signal, calculate corresponding updated ambient energy values ​​based on energy ratio parameters of corresponding frequency subbands of the spatial audio signal and a scaling factor; for one or more frequency subbands of the processed spatial audio signal, calculate corresponding modified energy ratio parameters based on updated directional energy divided by the sum of updated directional energy and updated ambient energy; for one or more frequency subbands of the processed spatial audio signal, calculate corresponding spectrum adjustment factors based on the sum of updated directional energy and updated ambient energy; in the one or more frequency subbands, obtain one or more enhanced audio channels by multiplying corresponding frequency bands of corresponding audio channels in one or more audio channels of the spatial audio signal by the spectrum adjustment factors derived for the corresponding frequency subbands; and compose a processed spatial audio signal, the processed spatial audio signal comprising the one or more enhanced audio channels, the directional indication of the spatial audio signal, and the modified energy ratio parameters.

[0021] The spatial audio signal and the processed spatial audio signal may include corresponding multi-channel speaker signals according to a first predefined speaker configuration, and wherein the component configured to process the spatial audio signal to generate the processed spatial audio signal may be configured to: calculate a corresponding angular difference between a defocus direction and a speaker direction indicated for a corresponding channel of the spatial audio signal; derive a corresponding gain value for each channel of the spatial audio signal based on the angular difference calculated for the corresponding channel by using a predefined angular difference function and a scaling factor derived based on the defocus amount; obtain one or more modified audio channels by multiplying the corresponding channel of the spatial audio signal by the gain value derived for the corresponding channel; and provide the modified audio channels as the processed spatial audio signal.

[0022] The predefined angle difference function may generate a gain value that decreases as the value of the angle difference decreases and increases as the value of the angle difference increases.

[0023] The processed spatial audio signal may include an Ambisonic signal, and the output spatial audio signal may include a two-channel binaural signal, wherein the reproduction control information may include an indication of a reproduction orientation defining a listening direction with respect to an audio scene, and wherein a component configured to process the processed spatial audio signal representing a modified audio scene based on a defocus direction to generate an output spatial audio signal according to the reproduction control information may be configured to: generate a rotation matrix based on the indicated reproduction orientation; multiply the channels of the processed spatial audio signal by the rotation matrix to obtain a rotated spatial audio signal; filter the channels of the rotated spatial audio signal using a predefined set of finite impulse response (FIR) filter pairs, wherein the set of FIR filter pairs is generated based on a data set of head-related impulse response functions (HRTFs) or head-related impulse responses (HRIRs); and generate left and right channels of the binaural signal as the sum of the filtered channels of the rotated spatial audio signal obtained for a corresponding one of the left and right channels.

[0024] The output spatial audio signal may include a two-channel binaural audio signal, wherein the reproduction control information may include an indication of a reproduction orientation defining a listening direction with respect to an audio scene, and a component configured to process the processed spatial audio signal representing a modified audio scene based on a defocus direction according to the reproduction control information to generate the output spatial audio signal may be configured to: obtain one or more enhanced audio channels in the one or more frequency sub-bands by multiplying corresponding frequency bands of corresponding audio channels in one or more audio channels of the processed spatial audio signal by a spectral adjustment factor received for the corresponding frequency sub-band; and convert the one or more enhanced audio channels into a two-channel binaural audio signal according to the indicated reproduction direction.

[0025] The output spatial audio signal may include a two-channel binaural audio signal, wherein the reproduction control information may include an indication defining a reproduction orientation relative to a listening direction of an audio scene, and wherein a component configured to process a processed spatial audio signal representing a modified audio scene based on a defocus direction to generate an output spatial audio signal according to the reproduction control information may be configured to convert one or more enhanced audio channels into a two-channel binaural audio signal according to the indicated reproduction direction.

[0026] The output spatial audio signal may include a two-channel binaural signal, wherein the reproduction control information may include an indication of a reproduction orientation defining a listening direction with respect to an audio scene, and wherein a component configured to process a processed spatial audio signal representing a modified audio scene based on a defocus direction according to the reproduction control information to generate an output spatial audio signal may be configured to: select a set of head-related transfer functions (HRTFs) according to the indicated reproduction direction; and convert channels of the processed spatial audio signal into a two-channel binaural signal that transmits the rotated audio scene using the selected set of HRTFs.

[0027] The reproduction control information may include an indication of a second predefined speaker configuration, and the output spatial audio signal may include a multi-channel speaker signal according to the second predefined speaker configuration, and wherein the component configured to process the processed spatial audio signal representing a modified audio scene based on the defocus direction to generate the output spatial audio signal according to the reproduction control information may be configured to: obtain the channels of the output spatial audio signal based on the channels of the processed spatial audio signal and using amplitude panning by being configured to perform the following operations: deriving a transformation matrix including amplitude panning gains, and using the transformation matrix to multiply the channels of the processed spatial audio signal to obtain the channels of the output spatial audio signal, wherein the amplitude panning gains provide a mapping from the first predefined speaker configuration to the second predefined speaker configuration.

[0028] The component may be further configured to obtain a defocus input from a sensor arrangement comprising at least one direction sensor and at least one user input, wherein the defocus input comprises an indication of a defocus direction based on a direction of the at least one direction sensor.

[0029] The defocus input may also include an indicator of the amount of defocus.

[0030] The defocus input may also include an indicator of the defocus shape.

[0031] The defocus shape may include at least one of: a defocus shape width; a defocus shape height; a defocus shape radius; a defocus shape distance; a defocus shape depth; a defocus shape range; a defocus shape diameter; and a defocus shape characterizer.

[0032] The defocus direction may be an arc defined by a range of defocus directions.

[0033] According to a second aspect, a method is provided, comprising: obtaining a defocus direction; processing a spatial audio signal representing an audio scene to generate a processed spatial audio signal representing a modified audio scene based on the defocus direction, so as to at least partially control a relative de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction; and outputting the processed spatial audio signal, wherein the modified audio scene based on the defocus direction at least partially enables de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction.

[0034] The method may further include obtaining an amount of defocus, and wherein processing the spatial audio signal may include controlling at least partially a relative de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in a defocus direction based on the amount of defocus.

[0035] Processing the spatial audio signal may include at least one of: at least partially reducing the emphasis of a portion of the spatial audio signal relative to other portions of the spatial audio signal at least partially in a defocused direction; and at least partially increasing the emphasis of other portions of the spatial audio signal relative to a portion of the spatial audio signal in the defocused direction.

[0036] Processing the spatial audio signal may include at least one of: at least partially reducing a sound level of a portion of the spatial audio signal relative to other portions of the spatial audio signal at least partially in a defocus direction based on the defocus amount; and at least partially increasing a sound level of other portions of the spatial audio signal relative to a portion of the spatial audio signal in the defocus direction based on the defocus amount.

[0037] The method may further include obtaining a defocus shape, and wherein processing the spatial audio signal may include at least partially controlling a relative de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction and within the defocus shape.

[0038] Processing the spatial audio signal may include at least one of: at least partially reducing the emphasis of a portion of the spatial audio signal relative to other portions of the spatial audio signal at least partially in a defocus direction and within a defocus shape; and at least partially increasing the emphasis of other portions of the spatial audio signal relative to a portion of the spatial audio signal in the defocus direction and within the defocus shape.

[0039] Processing the spatial audio signal may include at least one of: at least partially reducing a sound level of a portion of the spatial audio signal relative to other portions of the spatial audio signal in a defocus direction and within a defocus shape based on the defocus amount; and at least partially increasing a sound level of other portions of the spatial audio signal relative to a portion of the spatial audio signal in a defocus direction and within a defocus shape based on the defocus amount.

[0040] The method may include obtaining reproduction control information to control at least one aspect of outputting a processed spatial audio signal, and wherein outputting the processed spatial audio signal may include one of: processing the processed spatial audio signal representing a modified audio scene based on a defocus direction to generate an output spatial audio signal according to the reproduction control information; processing the spatial audio signal according to the reproduction control information before a component is configured to process the spatial audio signal representing the audio scene based on the defocus direction to generate the processed spatial audio signal representing the modified audio scene and output the processed spatial audio signal as an output spatial audio signal.

[0041] The spatial audio signal and the processed spatial audio signal may include corresponding Ambisonic signals, and wherein processing the spatial audio signal into the processed spatial audio signal may include, for one or more frequency subbands: extracting from the spatial audio signal a single-channel target audio signal representing a sound component arriving from a focused direction; generating a focused spatial audio signal, wherein the focused audio signal is arranged at a spatial position defined by a defocused direction; and creating the processed spatial audio signal as a linear combination of the focused spatial audio signal subtracted from the spatial audio signal, wherein at least one of the focused spatial audio signal and the spatial audio signal is scaled by a corresponding scaling factor derived based on an amount of defocus to reduce a relative level of the sound in the defocused direction.

[0042] Extracting a single-channel target audio signal may include: applying a beamformer to obtain a beamformed signal representing a sound component arriving from a defocused direction from a spatial audio signal; and applying a post-filter to obtain a processed audio signal based on the beamformed signal, thereby adjusting a spectrum of the beamformed signal to approximate a spectrum of the sound arriving from the defocused direction.

[0043] The spatial audio signal and the processed spatial audio signal may include corresponding first-order Ambisonic signals.

[0044] The spatial audio signal and the processed spatial audio signal may include a corresponding parameterized spatial audio signal, wherein the parameterized spatial audio signal may include one or more audio channels and spatial metadata, wherein the spatial metadata may include corresponding direction indications and energy ratio parameters for multiple frequency subbands, wherein processing the spatial audio signal to generate the processed spatial audio signal may include: calculating, for one or more frequency subbands, corresponding angular differences between defocus directions and directions indicated for corresponding frequency subbands of the spatial audio signal; deriving, for one or more frequency subbands, corresponding gain values ​​based on the angular differences calculated for the corresponding frequency subbands by using a predefined angular difference function and a scaling factor derived based on the defocus amount; and deriving, for one or more frequency subbands of the processed spatial audio signal, scaling factors based on the corresponding frequency subbands of the spatial audio signal. The method comprises the steps of: calculating a corresponding updated directional energy value based on an energy ratio parameter of a corresponding frequency subband and a gain value of the processed spatial audio signal; calculating a corresponding updated ambient energy value based on the energy ratio parameter of the corresponding frequency subband of the spatial audio signal and a scaling factor for one or more frequency bands of the processed spatial audio signal; calculating a corresponding modified energy ratio parameter based on the updated directional energy divided by the sum of the updated directional energy and the updated ambient energy for one or more frequency subbands of the processed spatial audio signal; calculating a corresponding spectrum adjustment factor based on the sum of the updated directional energy and the updated ambient energy for one or more frequency subbands of the processed spatial audio signal; and composing a processed spatial audio signal, the processed spatial audio signal comprising one or more audio channels of the spatial audio signal, a directional indication of the spatial audio signal, the modified energy ratio parameter, and the spectrum adjustment factor.

[0045] The spatial audio signal and the processed spatial audio signal may include a corresponding parameterized spatial audio signal, wherein the parameterized spatial audio signal may include one or more audio channels and spatial metadata, wherein the spatial metadata may include corresponding direction indications and energy ratio parameters for multiple frequency subbands, wherein processing the spatial audio signal to generate the processed spatial audio signal may include: calculating, for one or more frequency subbands, a corresponding angular difference between a defocus direction and a direction indicated for a corresponding frequency subband of the spatial audio signal; deriving, for one or more frequency subbands, a corresponding gain value based on the angular difference calculated for the corresponding frequency subband by using a predefined angular difference function and a scaling factor derived based on the defocus amount; calculating, for one or more frequency subbands of the processed spatial audio signal, a corresponding updated directional energy value based on the energy ratio parameters of the corresponding frequency subbands of the spatial audio signal and the gain value; and calculating, for the processed spatial audio signal, a corresponding updated directional energy value based on the energy ratio parameters of the corresponding frequency subbands of the spatial audio signal and the gain value. The method comprises the steps of: calculating a corresponding updated ambient energy value based on the energy ratio parameter of the corresponding frequency subband of the spatial audio signal and the scaling factor for one or more frequency bands of the processed spatial audio signal; calculating a corresponding modified energy ratio parameter based on the updated directional energy divided by the sum of the updated directional energy and the updated ambient energy for one or more frequency subbands of the processed spatial audio signal; calculating a corresponding spectrum adjustment factor based on the sum of the updated directional energy and the updated ambient energy for one or more frequency subbands of the processed spatial audio signal; obtaining one or more enhanced audio channels in the one or more frequency subbands by multiplying the corresponding frequency bands of the corresponding audio channels in the one or more audio channels of the spatial audio signal by the spectrum adjustment factor derived for the corresponding frequency subbands; and composing a processed spatial audio signal, the processed spatial audio signal comprising the one or more enhanced audio channels, the directional indication of the spatial audio signal, and the modified energy ratio parameter.

[0046] The spatial audio signal and the processed spatial audio signal may include corresponding multi-channel speaker signals according to a first predefined speaker configuration, and wherein processing the spatial audio signal to generate the processed spatial audio signal may include: calculating corresponding angular differences between defocus directions and speaker directions indicated for corresponding channels of the spatial audio signal; deriving corresponding gain values ​​for each channel of the spatial audio signal based on the angular differences calculated for the corresponding channels by using a predefined angular difference function and a scaling factor derived based on the defocus amount; obtaining one or more modified audio channels by multiplying the corresponding channels of the spatial audio signal by the gain values ​​derived for the corresponding channels; and providing the modified audio channels as the processed spatial audio signal.

[0047] The predefined angle difference function may generate a gain value that decreases as the value of the angle difference decreases and increases as the value of the angle difference increases.

[0048] The processed spatial audio signal may include an Ambisonic signal, and the output spatial audio signal may include a two-channel binaural signal, wherein the reproduction control information may include an indication of a reproduction orientation defining a listening direction with respect to an audio scene, and wherein processing the processed spatial audio signal representing a modified audio scene based on a defocus direction according to the reproduction control information to generate the output spatial audio signal may include: generating a rotation matrix based on the indicated reproduction orientation; multiplying the channels of the processed spatial audio signal with the rotation matrix to obtain a rotated spatial audio signal; filtering the channels of the rotated spatial audio signal using a predefined set of finite impulse response (FIR) filter pairs, wherein the set of FIR filter pairs is generated based on a data set of head-related impulse response functions (HRTFs) or head-related impulse responses (HRIRs); and generating left and right channels of the binaural signal as the sum of the filtered channels of the rotated spatial audio signal obtained for a corresponding one of the left and right channels.

[0049] The output spatial audio signal may include a two-channel binaural audio signal, wherein the reproduction control information may include an indication defining a reproduction orientation with respect to a listening direction of the audio scene, and processing the processed spatial audio signal representing a modified audio scene based on a defocus direction according to the reproduction control information to generate the output spatial audio signal may include: obtaining one or more enhanced audio channels in the one or more frequency sub-bands by multiplying corresponding frequency bands of corresponding audio channels in one or more audio channels of the processed spatial audio signal by spectral adjustment factors received for the corresponding frequency sub-bands; and converting the one or more enhanced audio channels into a two-channel binaural audio signal according to the indicated reproduction direction.

[0050] The output spatial audio signal may include a two-channel binaural audio signal, wherein the reproduction control information may include an indication of a reproduction orientation defining a listening direction with respect to an audio scene, and wherein processing the processed spatial audio signal representing a modified audio scene based on a defocus direction to generate the output spatial audio signal according to the reproduction control information may include: converting one or more enhanced audio channels into a two-channel binaural audio signal according to the indicated reproduction direction.

[0051] The output spatial audio signal may include a two-channel binaural signal, wherein the reproduction control information may include an indication of a reproduction orientation defining a listening direction with respect to an audio scene, and wherein processing the processed spatial audio signal representing a modified audio scene based on a defocus direction according to the reproduction control information to generate the output spatial audio signal may include: selecting a set of head-related transfer functions (HRTFs) according to the indicated reproduction direction; and converting channels of the processed spatial audio signal into a two-channel binaural signal that uses the selected set of HRTFs to transmit the rotated audio scene.

[0052] The reproduction control information may include an indication of a second predefined speaker configuration, and the output spatial audio signal may include a multi-channel speaker signal according to the second predefined speaker configuration, and wherein processing the processed spatial audio signal representing a modified audio scene based on the defocus direction to generate the output spatial audio signal according to the reproduction control information may include: obtaining the channels of the output spatial audio signal based on the channels of the processed spatial audio signal and using amplitude shifting by being configured to perform the following operations: deriving a transformation matrix including amplitude shift gains, and using the transformation matrix to multiply the channels of the processed spatial audio signal to obtain the channels of the output spatial audio signal, wherein the amplitude shift gains provide a mapping from the first predefined speaker configuration to the second predefined speaker configuration.

[0053] The method may further comprise obtaining a defocus input from a sensor device comprising at least one direction sensor and at least one user input, wherein the defocus input comprises an indication of a defocus direction based on a direction of the at least one direction sensor.

[0054] The defocus input may also include an indicator of the amount of defocus.

[0055] The defocus input may also include an indicator of the defocus shape.

[0056] The defocus shape may include at least one of: a defocus shape width; a defocus shape height; a defocus shape radius; a defocus shape distance; a defocus shape depth; a defocus shape range; a defocus shape diameter; and a defocus shape characterizer.

[0057] The defocus direction may be an arc defined by a range of defocus directions.

[0058] According to a third aspect, a device is provided, comprising at least one processor and at least one memory comprising computer program code, the at least one memory and the computer program code being configured to, together with the at least one processor, cause the device to at least: obtain a defocus direction; process a spatial audio signal representing an audio scene to generate a processed spatial audio signal representing a modified audio scene based on the defocus direction, so as to at least partially control a relative de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction; and output the processed spatial audio signal, wherein the modified audio scene based on the defocus direction at least partially enables de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction.

[0059] The apparatus may further be caused to obtain the amount of defocus, and wherein the apparatus caused to process the spatial audio signal may be caused to at least partially control a relative de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in a defocus direction based on the amount of defocus.

[0060] The device configured to process the spatial audio signal may be configured to perform at least one of: at least partially reducing the emphasis of a portion of the spatial audio signal relative to other portions of the spatial audio signal at least partially in a defocused direction; and at least partially increasing the emphasis of other portions of the spatial audio signal relative to a portion of the spatial audio signal in the defocused direction.

[0061] The device configured to process a spatial audio signal may be configured to perform at least one of: at least partially reducing a sound level of a portion of the spatial audio signal relative to other portions of the spatial audio signal at least partially in a defocus direction based on an amount of defocus; and at least partially increasing a sound level of other portions of the spatial audio signal relative to a portion of the spatial audio signal in a defocus direction based on the amount of defocus.

[0062] The apparatus may further be caused to obtain a defocus shape, and wherein the apparatus caused to process the spatial audio signal may be caused to at least partially control a relative de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction and within the defocus shape.

[0063] The device configured to process the spatial audio signal may be configured to perform at least one of: at least partially reducing the emphasis of a portion of the spatial audio signal relative to other portions of the spatial audio signal at least partially in a defocus direction and within a defocus shape; and at least partially increasing the emphasis of other portions of the spatial audio signal relative to a portion of the spatial audio signal in the defocus direction and within the defocus shape.

[0064] The device configured to process a spatial audio signal may be configured to perform at least one of: at least partially reducing a sound level of a portion of the spatial audio signal relative to other portions of the spatial audio signal in a defocus direction and within a defocus shape based on an amount of defocus; and at least partially increasing a sound level of other portions of the spatial audio signal relative to a portion of the spatial audio signal in a defocus direction and within a defocus shape based on the amount of defocus.

[0065] The device may be caused to obtain reproduction control information to control at least one aspect of outputting a processed spatial audio signal, and wherein the device caused to process the spatial audio signal may be caused to perform one of the following: processing the processed spatial audio signal representing a modified audio scene based on a defocus direction to generate an output spatial audio signal according to the reproduction control information; processing the spatial audio signal according to the reproduction control information before a component is configured to process the spatial audio signal representing an audio scene based on the defocus direction to generate a processed spatial audio signal representing the modified audio scene and output the processed spatial audio signal as an output spatial audio signal.

[0066] The spatial audio signal and the processed spatial audio signal may include corresponding Ambisonic signals, and wherein the device configured to process the spatial audio signal into the processed spatial audio signal may be configured to perform the following operations for one or more frequency subbands: extracting a single-channel target audio signal representing a sound component arriving from a focused direction from the spatial audio signal; generating a focused spatial audio signal, wherein the focused audio signal is arranged at a spatial position defined by a defocused direction; and creating the processed spatial audio signal as a linear combination of the focused spatial audio signal subtracted from the spatial audio signal, wherein at least one of the focused spatial audio signal and the spatial audio signal is scaled by a corresponding scaling factor derived based on an amount of defocus to reduce a relative level of sound in the defocused direction.

[0067] The device configured to extract a single-channel target audio signal may be configured to: apply a beamformer to obtain a beamformed signal representing a sound component arriving from a defocused direction from a spatial audio signal; and apply a post-filter to obtain a processed audio signal based on the beamformed signal, thereby adjusting the spectrum of the beamformed signal to approximate the spectrum of the sound arriving from the defocused direction.

[0068] The spatial audio signal and the processed spatial audio signal may include corresponding first-order Ambisonic signals.

[0069] The spatial audio signal and the processed spatial audio signal may include a corresponding parameterized spatial audio signal, wherein the parameterized spatial audio signal may include one or more audio channels and spatial metadata, wherein the spatial metadata may include corresponding direction indications and energy ratio parameters for multiple frequency subbands, wherein the device that is caused to process the spatial audio signal to generate the processed spatial audio signal may be caused to: calculate, for one or more frequency subbands, a corresponding angular difference between a defocus direction and a direction indicated for a corresponding frequency subband of the spatial audio signal; derive, for one or more frequency subbands, a corresponding gain value based on the angular difference calculated for the corresponding frequency subband by using a predefined angular difference function and a scaling factor derived based on the defocus amount; and, for one or more frequency subbands of the processed spatial audio signal, derive a corresponding gain value based on the angular difference calculated for the corresponding frequency subband by using a predefined angular difference function and a scaling factor derived based on the defocus amount. The method comprises the steps of: calculating a corresponding updated directional energy value based on the energy ratio parameter and the gain value of the corresponding frequency subband; calculating a corresponding updated ambient energy value for one or more frequency bands of the processed spatial audio signal based on the energy ratio parameter and the scaling factor of the corresponding frequency subband of the spatial audio signal; calculating a corresponding modified energy ratio parameter for one or more frequency subbands of the processed spatial audio signal based on the updated directional energy divided by the sum of the updated directional energy and the updated ambient energy; calculating a corresponding spectrum adjustment factor for one or more frequency subbands of the processed spatial audio signal based on the sum of the updated directional energy and the updated ambient energy; and composing a processed spatial audio signal, the processed spatial audio signal comprising one or more audio channels of the spatial audio signal, a directional indication of the spatial audio signal, the modified energy ratio parameter, and the spectrum adjustment factor.

[0070] The spatial audio signal and the processed spatial audio signal may include a corresponding parameterized spatial audio signal, wherein the parameterized spatial audio signal may include one or more audio channels and spatial metadata, wherein the spatial metadata may include corresponding direction indications and energy ratio parameters for multiple frequency subbands, wherein the device that is caused to process the spatial audio signal to generate the processed spatial audio signal may be caused to: calculate, for one or more frequency subbands, a corresponding angular difference between a defocus direction and a direction indicated for a corresponding frequency subband of the spatial audio signal; derive, for one or more frequency subbands, a corresponding gain value based on the angular difference calculated for the corresponding frequency subband by using a predefined angular difference function and a scaling factor derived based on the defocus amount; calculate, for one or more frequency subbands of the processed spatial audio signal, a corresponding updated directional energy value based on the energy ratio parameters of the corresponding frequency subbands of the spatial audio signal and the gain value; For one or more frequency bands of the processed spatial audio signal, calculate corresponding updated ambient energy values ​​based on energy ratio parameters of corresponding frequency subbands of the spatial audio signal and a scaling factor; for one or more frequency subbands of the processed spatial audio signal, calculate corresponding modified energy ratio parameters based on updated directional energy divided by the sum of updated directional energy and updated ambient energy; for one or more frequency subbands of the processed spatial audio signal, calculate corresponding spectrum adjustment factors based on the sum of updated directional energy and updated ambient energy; in the one or more frequency subbands, obtain one or more enhanced audio channels by multiplying corresponding frequency bands of corresponding audio channels in one or more audio channels of the spatial audio signal by the spectrum adjustment factors derived for the corresponding frequency subbands; and compose a processed spatial audio signal, the processed spatial audio signal comprising the one or more enhanced audio channels, the directional indication of the spatial audio signal, and the modified energy ratio parameters.

[0071] The spatial audio signal and the processed spatial audio signal may include corresponding multi-channel speaker signals according to a first predefined speaker configuration, and wherein the device, which is caused to process the spatial audio signal to generate the processed spatial audio signal, may be caused to: calculate corresponding angular differences between the defocus direction and the speaker direction indicated for the corresponding channel of the spatial audio signal; derive, for each channel of the spatial audio signal, a corresponding gain value based on the angular difference calculated for the corresponding channel by using a predefined angular difference function and a scaling factor derived based on the defocus amount; obtain one or more modified audio channels by multiplying the corresponding channel of the spatial audio signal by the gain value derived for the corresponding channel; and provide the modified audio channels as the processed spatial audio signal.

[0072] The predefined angle difference function may generate a gain value that decreases as the value of the angle difference decreases and increases as the value of the angle difference increases.

[0073] The processed spatial audio signal may include an Ambisonic signal, and the output spatial audio signal may include a two-channel binaural signal, wherein the reproduction control information may include an indication defining a reproduction orientation with respect to a listening direction of an audio scene, and wherein the apparatus, which is caused to process the processed spatial audio signal representing a modified audio scene based on a defocus direction according to the reproduction control information to generate the output spatial audio signal, may be caused to: generate a rotation matrix based on the indicated reproduction orientation; multiply the channels of the processed spatial audio signal by the rotation matrix to obtain a rotated spatial audio signal; filter the channels of the rotated spatial audio signal using a predefined set of finite impulse response (FIR) filter pairs, wherein the set of FIR filter pairs is generated based on a data set of head-related impulse response functions (HRTFs) or head-related impulse responses (HRIRs); and generate a left channel and a right channel of the binaural signal as the sum of the filtered channels of the rotated spatial audio signal obtained for a corresponding one of the left channel and the right channel.

[0074] The output spatial audio signal may include a two-channel binaural audio signal, wherein the reproduction control information may include an indication defining a reproduction orientation with respect to a listening direction of an audio scene, and the device, which is caused to process the processed spatial audio signal representing a modified audio scene based on a defocus direction according to the reproduction control information to generate the output spatial audio signal, may be caused to: obtain one or more enhanced audio channels in the one or more frequency sub-bands by multiplying corresponding frequency bands of corresponding audio channels in one or more audio channels of the processed spatial audio signal by spectral adjustment factors received for the corresponding frequency sub-bands; and convert the one or more enhanced audio channels into a two-channel binaural audio signal according to the indicated reproduction direction.

[0075] The output spatial audio signal may include a two-channel binaural audio signal, wherein the reproduction control information may include an indication defining a reproduction orientation relative to a listening direction of an audio scene, and wherein the apparatus, which is caused to process the processed spatial audio signal representing a modified audio scene based on a defocus direction in accordance with the reproduction control information to generate the output spatial audio signal, may be caused to: convert one or more enhanced audio channels into a two-channel binaural audio signal in accordance with the indicated reproduction direction.

[0076] The output spatial audio signal may include a two-channel binaural signal, wherein the reproduction control information may include an indication defining a reproduction orientation with respect to a listening direction of an audio scene, and wherein the device, which is caused to process the processed spatial audio signal representing a modified audio scene based on a defocus direction according to the reproduction control information to generate the output spatial audio signal, may be caused to: select a set of head-related transfer functions (HRTFs) according to the indicated reproduction direction; and convert channels of the processed spatial audio signal into a two-channel binaural signal, which uses the selected set of HRTFs to transmit the rotated audio scene.

[0077] The reproduction control information may include an indication of a second predefined speaker configuration, and the output spatial audio signal may include a multi-channel speaker signal according to the second predefined speaker configuration, and wherein the device, which is caused to process the processed spatial audio signal representing a modified audio scene based on the defocus direction to generate the output spatial audio signal according to the reproduction control information, may be caused to: obtain the channels of the output spatial audio signal based on the channels of the processed spatial audio signal and using amplitude panning by being configured to perform the following operations: deriving a transformation matrix including amplitude panning gains, and using the transformation matrix to multiply the channels of the processed spatial audio signal to obtain the channels of the output spatial audio signal, wherein the amplitude panning gains provide a mapping from the first predefined speaker configuration to the second predefined speaker configuration.

[0078] The apparatus may be further caused to: obtain a defocus input from a sensor device comprising at least one direction sensor and at least one user input, wherein the defocus input comprises an indication of a defocus direction based on a direction of the at least one direction sensor.

[0079] The defocus input may also include an indicator of the amount of defocus.

[0080] The defocus input may also include an indicator of the defocus shape.

[0081] The defocus shape may include at least one of: a defocus shape width; a defocus shape height; a defocus shape radius; a defocus shape distance; a defocus shape depth; a defocus shape range; a defocus shape diameter; and a defocus shape characterizer.

[0082] The defocus direction may be an arc defined by a range of defocus directions.

[0083] According to a fourth aspect, a device is provided, comprising: an obtaining circuit configured to obtain a defocus direction; a spatial audio signal processing circuit configured to process a spatial audio signal representing an audio scene to generate a processed spatial audio signal representing a modified audio scene based on the defocus direction, so as to at least partially control a relative de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction; and an output circuit configured to output the processed spatial audio signal, wherein the modified audio scene based on the defocus direction at least partially enables de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction.

[0084] According to a fifth aspect, there is provided a computer program [or a computer-readable medium comprising program instructions], the instructions being used to cause an apparatus to perform at least the following operations: obtain a defocus direction; process a spatial audio signal representing an audio scene to generate a processed spatial audio signal representing a modified audio scene based on the defocus direction, so as to at least partially control a relative de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction; and output the processed spatial audio signal, wherein the modified audio scene based on the defocus direction at least partially enables de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction.

[0085] According to a sixth aspect, a non-transitory computer-readable medium is provided, comprising program instructions for causing an apparatus to perform at least the following operations: obtaining a defocus direction; processing a spatial audio signal representing an audio scene to generate a processed spatial audio signal representing a modified audio scene based on the defocus direction, so as to at least partially control a relative de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction; and outputting the processed spatial audio signal, wherein the modified audio scene based on the defocus direction at least partially enables de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction.

[0086] According to a seventh aspect, a device is provided, comprising: a component for obtaining a defocus direction; a component for processing a spatial audio signal representing an audio scene to generate a processed spatial audio signal representing a modified audio scene based on the defocus direction, so as to at least partially control the relative de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction; and a component for outputting the processed spatial audio signal, wherein the modified audio scene based on the defocus direction at least partially enables de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction.

[0087] According to an eighth aspect, a computer-readable medium is provided, comprising program instructions for causing an apparatus to perform at least the following operations: obtaining a defocus direction; processing a spatial audio signal representing an audio scene to generate a processed spatial audio signal representing a modified audio scene based on the defocus direction, so as to at least partially control a relative de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction; and outputting the processed spatial audio signal, wherein the modified audio scene based on the defocus direction at least partially enables de-emphasis of a portion of the spatial audio signal at least partially relative to other portions of the spatial audio signal in the defocus direction.

[0088] An apparatus comprises means for performing the actions of the method as described above.

[0089] A device is configured to perform the actions of the method described above.

[0090] A computer program includes program instructions for causing a computer to execute the method as described above.

[0091] A computer program product stored on a medium may cause an apparatus to perform the method described herein.

[0092] An electronic device may include an apparatus as described herein.

[0093] A chipset may include the apparatus as described herein.

[0094] The embodiments of the present application are intended to solve the problems associated with the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] For a better understanding of the present application, reference will now be made by way of example to the accompanying drawings, in which:

[0096] Figure 1a , Figure 1b and Figure 1c showing an example sound scene showing an audio focus area or zone;

[0097] Figure 2a and Figure 2b schematically illustrates an example playback device and method for operating a playback device according to some embodiments;

[0098] Figure 3a and Figure 3b Schematically illustrating a high-order Ambisonic audio signal input according to some embodiments Figure 2a The example focus processor and method of operating the example focus processor shown in;

[0099] Figure 4a and Figure 4b Schematically illustrating a method for transmitting a parameterized spatial audio signal according to some embodiments Figure 2a The example focus processor and method of operating the example focus processor shown in;

[0100] Figure 5a and Figure 5b Schematically illustrating an audio signal input with multiple channels and / or audio objects according to some embodiments Figure 2a The example focus processor and method of operating the example focus processor shown in;

[0101] Figure 6a and Figure 6b Schematically illustrating a high-order Ambisonic audio signal input according to some embodiments Figure 2a The example reproduction processor and method of operating the example reproduction processor shown in;

[0102] Figure 7a and Figure 7b Schematically illustrating a method for transmitting a parameterized spatial audio signal according to some embodiments Figure 2a The example reproduction processor and method of operating the example reproduction processor shown in;

[0103] Figure 8 illustrates example implementations of some embodiments;

[0104] Fig. 9 illustrates an example controller for controlling focus direction, focus amount, and focus width according to some embodiments;

[0105] Fig.10 shows example processing outputs based on processing a high-order Ambisonic audio signal according to some embodiments;

[0106] Fig.11 An example device suitable for implementing the illustrated means is shown. DETAILED DESCRIPTION

[0107] Suitable apparatus and possible mechanisms for providing efficient rendering and playback of spatial audio signals are described in further detail below.

[0108] Previous spatial audio signal playback examples allow the user to control the focus direction and focus amount. However, in some cases, this focus direction / amount control may not be sufficient. The concept discussed below is characterized by an apparatus and method that can indicate further focus control to eliminate or de-emphasize sounds in certain directions. For example, there may be many different features in a sound field, such as multiple dominant sound sources in certain directions as well as ambient sounds. Some users may prefer to remove certain features of the sound field, while other users may prefer to hear the complete audio scene, or remove alternative features of the sound field. In particular, a user may wish to remove undesirable sounds in such a way that the remainder of the spatial sound scene is reproduced as originally intended.

[0109] Described below Figure 1a to Figure 1c It shows what a user is expected to perceive when listening to a reproduced spatial audio signal.

[0110] As an example, Figure 1a A user 101 is shown positioned in a defined orientation. Within the audio scene there are sources of interest 105, such as a speaker. Furthermore, there may be other ambient audio content 107 around the user.

[0111] In addition, the user can identify interfering audio sources such as air conditioning 103. Conventionally, the user can control the playback to focus on the source of interest 105 to emphasize the source of interest relative to the interfering source 103. However, the concepts discussed in the embodiments attempt to instead perform a control operation on the identified one or more sources (such as in Figure 1a The sound quality can be improved by “removing” (or defocusing or negative-focusing) the image signal as indicated by the defocusing or negative-focusing identification source 103.

[0112] As another example, Figure 1b As shown in , the user may wish to defocus or negatively focus any source within a shape or region within the sound scene. Thus, for example, Figure 1b A user 101 is shown positioned with a defined orientation within an audio or sound scene having a source of interest 105 (e.g., a speaker), other ambient audio content 107 such as ambient audio content, and an interference source 155 within a defined region 153. In this example, the defocused or negative focus region is represented by a defocus arc 151 having a defined width and direction relative to the user 101. The defocus arc 151 having a defined width and direction relative to the user 101 covers the interference source 155 within the interference source region 153.

[0113] exist Figure 1c Another way in which a defocused or negative focus area may be represented is shown in , where a defocused area or volume (for a 3D area) 161 covers an interferer 155 within the interferer area 153. In this example, the defocused area may be defined by distance as well as direction and "width".

[0114] Thus, embodiments as discussed herein attempt to provide control of defocus shape (in addition to defocus direction and amount). Concepts as discussed with respect to embodiments described herein relate to spatial audio reproduction and enable audio playback utilizing control components for reducing / eliminating / removing audio elements originating from selectable spatial directions (or regions or volumes) by a desired amount (e.g., 0%-100%) relative to elements outside of these determined defocus shapes in order to de-emphasize the audibility of audio elements in selected spatial directions (or regions or volumes) while maintaining the audibility of desired audio elements in unselected spatial directions (or regions or volumes), while also keeping the spatial audio signal format the same.

[0115] Embodiments provide at least one defocus (or defocus) parameter corresponding to a selectable direction and amount. In addition, in some embodiments, the defocus (or defocus) parameter may define a defocus (or defocus) shape and may be defined by any one (or a combination of two or more) of the following parameters corresponding to direction, width, height, radius, distance, and depth. In some embodiments, the parameter set includes parameters defining any arbitrary defocus shape.

[0116] In some embodiments, at least one defocus parameter is provided together with at least one focus parameter in order to emphasize the audibility of a further selected spatial direction (or shape, area or volume).

[0117] In some embodiments, spatial audio signal processing may be performed by: obtaining a spatial audio signal associated with media having multiple viewing directions; obtaining focus / defocus direction and amount parameters (which may optionally include obtaining at least one focus / defocus shape information); modifying the spatial audio signal to have desired (focus and) defocus characteristics; and reproducing the modified spatial audio signal (with headphones or speakers).

[0118] The obtained spatial audio signal may be, for example: an Ambisonic signal; a loudspeaker signal; a parametric spatial audio format, such as a set of audio channels and associated spatial metadata.

[0119] Focus / de-focus information may be defined as follows: focus refers to increasing the relative prominence of audio originating from a selectable direction (or shape or area), while de-focus refers to reducing the relative prominence of audio originating from that direction (or shape or area).

[0120] The focus / defocus amount determines the degree of focusing or defocusing. For example, it can be from 0% to 100%, where 0% means keeping the original sound scene unchanged and 100% means maximally focusing / defocusing in the desired direction or within a defined area.

[0121] In some embodiments, the focus / defocus control can be a toggle control that determines focus or defocus, or it can be controlled in other ways, for example, by extending the focus amount over a range from -100% to 100%, where negative values ​​indicate a defocus (or negative focus) effect and positive values ​​indicate a focusing effect.

[0122] It should be noted that different users may wish to have different focus / defocus characteristics. Based on their personal preferences, the original spatial audio signal can be modified and reproduced individually for each user.

[0123] Figure 2aA block diagram of some components and / or entities of a spatial audio processing device 250 according to an example is shown. It will be understood that the two separate steps (focus / defocus processor+reproduction processor) shown in this figure and further detailed later can be implemented as an integrated process, or in some examples in the reverse order as described herein (wherein the reproduction processor operation is followed by the focus processor operation). The spatial audio processing device 250 includes an audio focus processor 201, which is configured to receive an input audio signal and, in addition, receive focus / defocus parameters 202; and based on the input audio signal 200 and according to the focus / defocus parameters 202 (which may include a focus / defocus direction about a focus / defocus element; a focus / defocus amount; a focus / defocus height; a focus / defocus radius; a focus / defocus distance; and a focus / defocus depth), obtain an audio signal 204 with a focused / defocused sound component. In addition, the spatial audio processing device 250 may further include an audio reproduction processor 207, which is configured to receive the audio signal 204 with a focused / defocused sound component and the reproduction control information 206, and is configured to obtain an output audio signal 208 in a predefined audio format based on the audio signal 204 with a focused / defocused sound component and further according to the reproduction control information 206, wherein the reproduction control information 206 is used to control at least one aspect related to processing the spatial audio signal with a focused / defocused component in the audio reproduction processor 207. The reproduction control information 206 may include an indication of a reproduction orientation (or reproduction direction) and / or an indication of an applicable speaker configuration. In view of the method for processing the above-mentioned spatial audio signal, the audio focus processor 201 may be configured to implement aspects of processing the spatial audio signal by modifying the audio scene so as to control the emphasis or de-emphasis of at least a portion of the spatial audio signal in a received focus area or direction according to the received focus / defocus amount. The audio reproduction processor 207 may output the processed spatial audio signal as a modified audio scene based on the observed direction and / or position, wherein the modified audio scene exhibits an emphasis for at least the portion of the spatial audio signal in the focus area and according to the received focus amount.

[0124] exist Figure 2aIn the diagram of , each of the input audio signal, the audio signal with focused / defocused sound components, and the output audio signal is provided as a corresponding spatial audio signal in a predefined spatial audio format. Therefore, these signals can be referred to as input spatial audio signals, spatial audio signals with focused / defocused sound components, and output spatial audio signals, respectively. Along the ideas described in the foregoing, generally, the spatial audio signal transmits an audio scene involving both one or more directional sound sources at specific locations of the audio scene and the environment of the audio scene. However, in some cases, the spatial audio scene may involve one or more directional sound sources without an environment or an environment without any directional sound sources. In this regard, the spatial audio signal includes information for transmitting one or more directional sound components and / or ambient sound components, wherein the one or more directional sound components represent different sound sources having a certain position within the audio scene (e.g., a certain arrival direction and a certain relative intensity relative to the listening point), and the ambient sound component represents the ambient sound within the audio scene. It should be noted that the division of the audio scene into directional sound components and ambient components is generally only a representation or approximation, and the actual sound scene may involve more complex features such as wide sources and coherent sound reflections. Nevertheless, even with such complex acoustic features, conceptualizing an audio scene as a combination of direct and ambient components is often a reasonable representation or approximation, at least in a perceptual sense.

[0125] Typically, the input audio signal and the audio signal with focused / defocused sound components are provided in the same predefined spatial format, while the output audio signal may be provided in the same spatial format as applied to the input audio signal (and the audio signal with focused / defocused sound components), or a different predefined spatial format may be used for the output audio signal. The spatial audio format of the output audio signal is selected in view of the characteristics of the sound reproduction hardware applied to play back the output audio signal. Typically, the input audio signal may be provided in a first predetermined spatial audio format, and the output audio signal may be provided in a second predetermined spatial audio format. Non-limiting examples of spatial audio formats suitable for use as the first and / or second spatial audio formats include Ambisonics, surround speaker signals according to a predefined speaker configuration, and predefined parametric spatial audio formats. More detailed non-limiting examples of using these spatial audio formats as the first and / or second spatial audio formats in the framework of the spatial audio processing device 250 are provided later in the present disclosure.

[0126] The spatial audio processing device 250 is typically applied to process the input spatial audio signal 200 as a sequence of input frames into a corresponding sequence of output frames, each input (output) frame comprising a corresponding digital audio signal segment for each channel of the input (output) spatial audio signal, provided as a corresponding series of input (output) samples in time at a predefined sampling frequency. In some embodiments, the input signal of the spatial audio processing device 250 may be in an encoded form, for example, AAC or AAC+embedded metadata. In such embodiments, the encoded audio input may initially have a decoder. Similarly, in some embodiments, the output from the spatial audio processing device 250 may be encoded in any suitable manner.

[0127] In a typical example, the spatial audio processing device 250 uses a fixed predefined frame length so that each frame includes corresponding L samples for each channel of the input spatial audio signal, and the fixed predefined frame length is mapped to a corresponding duration at a predefined sampling frequency. As an example in this regard, the fixed frame length can be 20 milliseconds (ms), which results in frames of L=160, L=320, L=640 and L=960 samples per channel at sampling frequencies of 8, 16, 32 or 48 kHz, respectively. These frames can be non-overlapping, or they can partially overlap, depending on whether the processor applies filter banks and how these filter banks are configured. However, these values ​​are used as non-limiting examples and frame lengths and / or sampling frequencies different from these examples can be used instead, depending on, for example, the desired audio bandwidth, the desired framing delay and / or the available processing power.

[0128] In the spatial audio processing device 250, focus / defocus refers to a user-selectable direction / quantity parameter (or spatial region of interest). Focus / defocus may typically be, for example, a certain direction, distance, radius, arc of the audio scene. In another example, the focus / defocus region is a region in which the (directional) sound source of interest is currently located. In the former case, the user-selectable focus / defocus may indicate a region that remains unchanged or changes infrequently, because the focus is mainly in a specific direction (or in a spatial region), while in the latter case, the user-selected focus / defocus may change more frequently, because the focus / defocus is set to a certain sound source that may (or may not) change its position (or shape / size) in the audio scene over time. In the example, focus / defocus may, for example, be defined as an azimuth angle defining a direction.

[0129] For example, according to Figure 2bThe method 260 shown in the flowchart depicted in FIG. 26A provides the functionality described above with reference to the components of the spatial audio processing device 250. The method 260 may be provided, for example, by a device configured to implement the spatial audio processing system 250 described in the present disclosure via a number of examples. The method 260 is used as a method for processing an input spatial audio signal representing an audio scene into an output spatial audio signal representing a modified audio scene. The method 260 includes receiving an indication of a focus / defocus direction and an indication of a focus / defocus strength or amount, as shown in block 261. The method 260 also includes processing the input spatial audio signal into an intermediate spatial audio signal representing the modified audio scene, wherein the relative level of sounds arriving from the focus / defocus direction is modified according to the focus / defocus strength, as shown in block 263. The method 260 also includes receiving reproduction control information that controls the processing of the intermediate spatial signal into the output spatial audio signal, as shown in block 265. The reproduction control information may, for example, define at least one of a reproduction orientation (e.g., a listening direction or a viewing direction) or a speaker configuration for the output spatial audio signal. The method 260 further comprises processing the intermediate spatial audio signal into an output spatial audio signal according to the reproduction control information, as shown in block 267 .

[0130] The method 260 may be varied in various ways, for example according to the examples relating to the respective functions of the components of the spatial audio processing device 250 provided above and below.

[0131] A defocusing operation is described in more detail in the following example, however, it should be understood that the same operation can be applied to other focusing operations as well as other defocusing operations.

[0132] In some embodiments, the input to the spatial audio processing device 250 is an Ambisonic signal. The device can be configured to receive (and the method can be applied to) an Ambisonic signal of any order. The Ambisonic audio signal can be a first-order Ambisonic (FOA) signal consisting of an omnidirectional signal and three orthogonal first-order modes along the y, z, x coordinate axes. The y, z, x coordinate order is chosen herein because it is the same order as the first-order coefficients of the typical ACN (Ambisonics Channel Number) channel ordering of Ambisonic signals.

[0133] Note that the Ambisonics audio format can express spatial audio signals in terms of spatial beam patterns, and it will be straightforward for a person skilled in the art to obtain the examples below and design alternative sets of spatial beam patterns to express spatial audio. Furthermore, the Ambisonics audio format is a particularly relevant audio format because it is the typical way to express spatial audio in the context of 360-degree video. Typical sources of Ambisonic audio signals include microphone arrays and content from VR video streaming services such as YouTube 360.

[0134] about Figure 3a , shows the focus processor 350 in the context of Ambisonic input and output. The figure assumes a first order Ambisonics (FOA) signal (4 channels), however, a higher order Ambisonics (HOA) may be applied instead of FOA. In embodiments implementing an HOA input format, the number of channels instead of 4 channels may be, for example, 9 channels (second order Ambisonics) or 16 channels (third order Ambisonics).

[0135] Example Ambisonic signal x FOA (t) 300 as well as (de)focus direction 304 , (de)focus amount and (de)focus control 310 are inputs to the focus processor 350 .

[0136] In some embodiments, the focusing processor 350 includes a filter bank 301. In some embodiments, the filter bank 301 is configured to convert an Ambisonic (FOA) signal 300 (corresponding to an Ambisonic or spherical harmonic mode) to generate a time-frequency domain version of a time-domain input audio signal. In some embodiments, the filter bank 301 can be a short-time Fourier transform (STFT) or any other suitable filter bank for spatial sound processing, such as a complex modulated quadrature mirror filter (QMF) bank. The output of the filter bank 301 is a time-frequency domain Ambisonic audio signal 302 in a frequency band. The frequency band can be one or more frequency bins (separate frequency components) of the filter bank 301 applied. The frequency band can approximate the resolution of perception-related calculations, such as Bark bands, which are more selective in the spectrum at low frequencies than at high frequencies. Alternatively, in some implementations, the frequency band can correspond to the frequency bin.

[0137] The (non-focused) time-frequency domain Ambisonic audio signal 302 is output to the mono focuser 303 and the mixer 311 .

[0138] The focusing processor 301 may further include a mono focuser 303. The mono focuser 303 is configured to receive the transformed (unfocused) time-frequency domain Ambisonic signal 302 from the filter bank 301 and further receive a (defocused) focusing direction parameter 304.

[0139] The mono (defocused) focuser 303 may implement any known method to generate a mono focused audio output based on the FOA input. In this example, the mono focuser 303 implements a minimum variance distortionless response (MVDR) mono focused audio output. The MVDR beamforming operation attempts to obtain a target signal from a desired focus direction without distortion, while finding adaptive beamforming weights that attempt to minimize output energy (in other words, suppress interference energy) under this constraint.

[0140] In some embodiments, the mono focuser 303 is configured to combine the frequency band signals (eg, four channels in the case of FOA) into one beamformed signal by:

[0141] y(b,n)=w H (k,n)x(b,n)

[0142] Where k is the frequency band index, b is the frequency bin index (where b is included in frequency band k), n is the time index, y(b,n) is a one-channel beamformed signal of bin b, w(k,n) is a 4x1 beamforming weight vector, and x(b,n) is a 4x1 FOA signal vector with four channels of frequency bin b signals. In this expression, the same beamforming weight w(k,n) is applied to the signal x(b,n) where bin b is included in frequency band k.

[0143] The monophonic focuser 303 implementing the MVDR beamformer may use, for each frequency band k,:

[0144] - an estimate of the covariance matrix of the signal x(b,n) in the interval x(b,n) in frequency band k (and possibly with a time average over several time indices n) and

[0145] - A steering vector according to the focus direction. In the example of a FOA signal, a steering vector may be generated based on a unit vector pointing in the focus direction. For example, a steering vector for FOA may be

[0146] where v(n) is a unit vector pointing in the focus direction (in the coordinate order y, z, x).

[0147] Based on the covariance matrix estimate and the steering vector, the weights w(k,n) can be generated using the known MVDR formula.

[0148] Thus, in some embodiments, the mono focuser 303 may provide a single channel focused output signal 306 , which is provided to the Ambisonics panner 305 .

[0149] In some embodiments, the Ambisonics panner 305 is configured to receive the channel (defocused) focus output signal 306 and the (defocused) focus direction 304 and generate an Ambisonic signal, wherein the monophonic focus signal is positioned in the focus direction. The focused time-frequency Ambisonic signal 308 output generated by the Ambisonics panner 305 may be generated based on the following formula:

[0150]

[0151] In some embodiments, the (de)focused time-frequency Ambisonic signal y FOA (b,n) 308 may then be output to mixer 311 .

[0152] In some embodiments, the output of a beamformer such as an MVDR may be cascaded with a post filter. Post filtering is generally the process of adaptively modifying the gain or energy of the beamformer output in a frequency band. For example, it is known that while MVDR is effective at suppressing isolated strong interfering sound sources, it performs poorly in ambient acoustic scenarios such as outdoor recordings with traffic noise. This is because MVDR effectively aims to steer the beam pattern minima in those directions where the interference is located. When interfering sounds are spatially diffuse, such as traffic noise, MVDR is not as effective in suppressing these interferences.

[0153] Therefore, in some embodiments, a post filter may be implemented to estimate the acoustic energy in a frequency band in the focus direction. In turn, the beamformer output energy is measured at the same frequency band, and a gain is applied in the frequency band to correct the acoustic spectrum to improve the estimated target spectrum. In such embodiments, the post filter may further suppress interfering sounds.

[0154] An example of a post filter is described in Delikaris-Manias, Symeon and Ville, Pulkki, "Cross pattern coherence algorithm for spatial filtering applications utilizing microphone arrays," IEEE Journal on Audio, Speech and Language Processing, Vol. 21, No. 11 (2013): pp. 2356-2367, where a cross-spectral energy estimate between a first-order spherical harmonic signal and a second-order spherical harmonic signal is used to estimate the target energy in the viewing direction. Cross-spectral estimates can also be obtained for other modes, such as between a zero (omnidirectional) order spherical harmonic signal and a first (dipole) order spherical harmonic signal. The cross-spectral estimate provides an energy estimate for the target direction.

[0155] When post-filtering is implemented, the beamforming equation can be appended with a gain g(k,n):

[0156] y′(b,n)=g(k,n)w H (k,n)x(b,n)

[0157] The gain g(k,n) can be derived using the cross-spectral energy estimation method as follows. First, the cross-correlation between the omnidirectional FOA signal component and the figure-of-eight signal with a positive lobe towards the focus direction is formulated:

[0158]

[0159] Wherein, the signal x with sub-indexes (W, Y, Z, X) denotes the signal components of the four FOA signals x(b,n), the asterisk (*) denotes the complex conjugate, and E denotes the desired operator which can be implemented as an average operator over the desired time region. The real-valued, non-negative cross-correlation measure for frequency band k is then formulated as follows:

[0160]

[0161] In fact, the value C(k,n) is an estimate of the energy of the sound arriving from the focus direction in frequency band k. Furthermore, the estimated beamforming output y(b,n) = w H The energy D(k,n) of the interval within the frequency band k of (k,n)x(b,n) is:

[0162]

[0163] Then, the spatial filter gain can be obtained as:

[0164]

[0165] In other words, when the energy estimate C(k,n) is less than the beam output energy D(k,n), the spatial filter will further reduce the beam output energy in frequency band k. Therefore, the function of the spatial filter is to further adjust the spectrum of the beamformer output to make it closer to the spectrum of the sound arriving from the focused direction.

[0166] In some embodiments, the (de)focusing processor may use such post-filtering. The beamformed output y(b,n) of the mono focuser 303 may be processed in a frequency band with a post-filter gain to generate a post-filtered beamformed output y′(b,n), where y′(b,n) is applied instead of y(b,n). It will be appreciated that there are various suitable beamformers and post-filters that may be applied in addition to those described as examples above.

[0167] In some embodiments, the focusing processor 350 comprises a mixer 311. The mixer is configured to receive the (de-)focused time-frequency Ambisonics signal y FOA (b,n) 308 and the unfocused time-frequency Ambisonics signal x(b,n) 302 (with potential delay adjustment, where MVDR estimation and processing involves look-ahead processing). In addition, the mixer 311 also receives the (de)focus amount and focus / defocus control parameters 310.

[0168] In this example, the (de)focus control parameter is a binary switch of "focus" or "defocus". The (de)focus amount parameter a(n) is expressed as a factor between 0..1, where 1 is maximum focus, and is used to describe the amount of focus or defocus, depending on the mode used.

[0169] In some embodiments, when the defocus parameter is in the "focus" mode, the output of the mixer 311 is:

[0170] y MIX (b,n)=a(n)y FOA (b,n)+(1-a(n))x(b,n)

[0171] In some embodiments, the value y in the above formula FOA (k,n) are modified by a factor (eg, a constant 4) before mixing to further accentuate the (de)focusing effect.

[0172] In some embodiments, when the defocus parameter is in the "defocus" mode, the mixer may be configured to perform:

[0173] y MIX(b,n)=x(b,n)-a(n)y FOA (b,n)

[0174] In other words, when a(n) is 0, the defocusing effect is also at zero, however, when a(n) is larger or reaches 1, the blending process subtracts the signal y from the spatial FOA signal x(b,n) FOA (b,n),y FOA (b,n) is the spatialized focus signal. Due to the subtraction, the amplitude of the signal component from the focus direction is reduced. In other words, a defocusing process occurs and the resulting Ambisonics spatial audio signal has a reduced amplitude for sounds from the focus direction. In some configurations, y MIX (b,n) 312 may be amplified based on a rule as a function of a(n) to account for the average loss of loudness due to the defocusing process.

[0175] The output of the mixer 311 , namely the mixed time-frequency Ambisonics audio signal 312 , is passed to an inverse filter bank 313 .

[0176] In some embodiments, the focus processor 350 includes an inverse filter bank 313 configured to receive the mixed time-frequency Ambisonics audio signal 312 and transform the audio signal to the time domain. The inverse filter bank 313 generates a suitable pulse code modulated (PCM) Ambisonics audio signal with the added focus / defocus.

[0177] about Figure 3b , showing that Figure 3a A flow chart of the operation of the FOA focusing processor 360 is shown in FIG.

[0178] like Figure 3b As shown in step 361, the initial operation is to receive an Ambisonics (FOA) audio signal (as well as focus / defocus parameters, such as direction, width, amount or other control information).

[0179] like Figure 3b As shown in step 363, the next operation is to generate the transformed Ambisonics audio signal into the time-frequency domain.

[0180] like Figure 3b As shown in step 365 , after the time-frequency domain Ambisonics audio signal has been generated, the next operation is to generate a mono focused Ambisonics audio signal from the time-frequency domain Ambisonics audio signal based on the focus direction (eg, using beamforming).

[0181] Furthermore, if Figure 3bAs shown in step 367, Ambisonics panning is performed on the mono (defocused) Ambisonics audio signal based on the focus direction.

[0182] Furthermore, if Figure 3b As shown in step 369, based on the (de-)focus amount and the (de-)focus control parameter, the panned Ambisonic audio signal (the (de-)focused time-frequency Ambisonic signal) is mixed with the unfocused time-frequency Ambisonic signal.

[0183] like Figure 3b As shown in step 371, the mixed Ambisonic audio signal may be inversely transformed.

[0184] Furthermore, if Figure 3b As shown in step 373, the time domain Ambisonic audio signal is output.

[0185] about Figure 4a , a focus processor is shown, which is configured to receive a parameterized spatial audio signal as input. The parameterized spatial audio signal includes an audio signal and spatial metadata, such as the direction in the frequency band and the direct-to-total energy ratio. The structure and generation of parameterized spatial audio signals are known, and their generation has been described from microphone arrays (e.g., mobile phones, VR cameras). In addition, parameterized spatial audio signals can also be generated from speaker signals and Ambisonic signals. In some embodiments, the parameterized spatial audio signal can be generated from an IVAS (Immersive Voice and Audio Service) audio stream, which can be decoded and demultiplexed into the form of spatial metadata and audio channels. The typical number of audio channels in such a parameterized spatial audio stream is a two audio channel audio signal, however, in some embodiments, the number of audio channels can be any number of audio channels.

[0186] In some examples, the parameterized information includes depth / distance information, which can be implemented in 6-degrees-of-freedom (6DOF) reproduction. In 6DOF, distance metadata (along with other metadata) is used to determine how the energy and direction of the sound should change based on user movement.

[0187] In this example, each spatial metadata direction parameter is associated with both a directional to total energy ratio and a distance parameter. The estimation of distance parameters in the context of parametric spatial audio capture has been detailed in earlier applications such as GB patent applications GB1710093.4 and GB1710085.0, but is not explored further for clarity reasons.

[0188] A focus processor 450 configured to receive parameterized spatial audio 400 is configured to use (de)focus parameters to determine how much the directional and ambient components of the parameterized spatial audio signal should be attenuated or emphasized to enable a (de)focus effect. The focus processor 450 is described below in two configurations. The first uses (de)focus parameters: direction and amount, further including the width of the resulting focus / defocus arc. In this configuration, the 6DOF distance parameter is optional. The second uses parameters (de)focus direction and amount and distance and radius, which results in a focus / defocus sphere at a certain position. In this configuration, the 6DOF distance parameter is required. The differences in these configurations are only stated in the following description where necessary.

[0189] In the following examples, methods (and formulas) are described without changing over time, but it is understood that all parameters may change over time.

[0190] In some embodiments, the focusing processor includes a ratio modifier and spectral adjustment factor determiner 401, which is configured to receive focusing parameters 408 and also receive spatial metadata consisting of direction 402 (and in some embodiments, distance 422) in a frequency band and a directional to total energy ratio 404.

[0191] The ratio modifier and spectral adjustment factor determiner 401 is configured to receive the focusing parameter and additionally spatial metadata consisting of the direction in the frequency band 402, the directional to total energy ratio 404 (and in some embodiments, the distance 422).

[0192] Unless otherwise stated, the following description considers the case where the focus parameters include direction, width and amount. In some embodiments, the ratio modifier and spectrum adjustment factor determiner 401 is configured to determine the angular difference β(k) between the focus direction (one for all frequency bands k) and the spatial metadata direction (which may be different for different frequency bands k). In some embodiments, v m (k) is determined as the column unit vector pointing to the direction parameter of the spatial metadata in frequency band k, and v f is determined as the column unit vector pointing in the focus direction. The angular distance β(k) can be determined as:

[0193]

[0194] in, Yes m The transpose of (k).

[0195] In turn, the ratio modifier and spectrum adjustment factor determiner 401 is configured to determine a direct-gain parameter f(k). The focus amount parameter a may be expressed as a normalized value between 0..1 (where 0 means zero focus / defocus and 1 means maximum focus / defocus), and a focus width β0, which may be, for example, 20 degrees at a certain time instance.

[0196] When the ratio modifier and spectrum adjustment factor determiner 401 is configured to perform focusing (as opposed to defocusing), an example gain formula is:

[0197]

[0198] Where c is a gain constant for focusing, for example, 4. When the ratio modifier and spectrum adjustment factor determiner 401 is configured to perform defocusing, an example formula is:

[0199]

[0200] In some embodiments, the constant c may have a different value in the defocused case than in the focused case. Furthermore, in practice, it may be desirable to smooth the above function so that the focus gain function smoothly transitions from a high value in the focused region to a low value in the unfocused region.

[0201] Unless otherwise stated, the following description considers the case where the focus parameters include direction, distance, radius and amount. In some embodiments, the ratio modifier and spectrum adjustment factor determiner 401 is configured to determine the focus position p f and metadata location p m (k), is formulated as follows. In some embodiments, v m (k) is determined as the column unit vector of the direction parameter pointing to the spatial metadata in frequency band k, v f is determined as a column unit vector pointing in the focus direction. The focus position is formulated as p f =v f d f , where d f is the focal distance. The spatial metadata position is formulated as p m (k) = v m (k)d m (k), where d m (k) is the distance parameter of the spatial metadata in frequency band k. In some embodiments, the ratio modifier and spectrum adjustment factor determiner 401 is configured to determine the focus position p f (one for all bands k) with spatial metadata position p mThe position difference γ(k) between (k) (k may be different in different frequency bands). The position difference γ(k) can be determined as:

[0202] γ(k)=|p f -p m (k),|

[0203] Among them, the |.| operator is used to determine the distance between vectors.

[0204] In turn, the ratio modifier and spectrum adjustment factor determiner 401 is configured to determine the directional gain parameter f(k). The focusing amount parameter a may be expressed as a normalized value between 0..1 (where 0 means zero focus / defocus and 1 means maximum focus / defocus), and the focusing radius is denoted γ0, which may be, for example, 1 meter at a certain time instance.

[0205] When the ratio modifier and spectrum adjustment factor determiner 401 is configured to perform focusing (as opposed to defocusing), an example gain formula is:

[0206]

[0207] Where c is a gain constant for focusing, for example, 4. When the ratio modifier and spectrum adjustment factor determiner 401 is configured to perform defocusing, an example formula is:

[0208]

[0209] In some embodiments, the constant c may have a different value in the defocused case than in the focused case. Furthermore, in practice, it may be desirable to smooth the above function so that the focus gain function smoothly transitions from a high value in the focused region to a low value in the unfocused region.

[0210] The remaining description applies to both focus parameter configurations described above. In some embodiments, the ratio modifier and spectral adjustment factor determiner 401 is further configured to determine a new direct portion value D(k) of the parameterized spatial audio signal as:

[0211] D(k)=r(k)*f(k)

[0212] where r(k) is the ratio of directional to total energy in frequency band k.

[0213] In some embodiments, the ratio modifier and spectrum adjustment factor determiner 401 is configured to determine a new ambient portion value A(k) (in the focusing process) as:

[0214] A(k)=(1-r(k))*(1-a)

[0215] In some embodiments, the ratio modifier and spectrum adjustment factor determiner 401 is configured to use A(k)=(1-r(k)) in the defocusing process to determine the new ambient component, which means that the defocusing process does not affect the spatial surrounding environment energy.

[0216] In turn, the ratio modifier and spectral adjustment factor determiner 401 is configured to determine a spectral correction factor s(k) which is output to the spectral adjustment processor 403, the spectral correction factor s(k) being in turn formulated based on the overall modification of the acoustic energy. For example:

[0217]

[0218] In some embodiments, the ratio modifier and spectral adjustment factor determiner 401 is configured to determine a new modified directional to total energy ratio parameter r'(k) to replace r(k) based on the following equation:

[0219]

[0220] In the case of numerical uncertainty, D(k)=A(k)=0, and r′(k) can also be set to zero.

[0221] In some embodiments, the direction value 402 (and distance value 422) in the spatial metadata may be passed through and outputted unmodified.

[0222] In some embodiments, the focus processor includes a spectral adjustment processor 403. The spectral adjustment processor 403 is configured to receive audio signals (which in some embodiments are in time-frequency representation, or alternatively they are first transformed to the time-frequency domain) 406 and spectral adjustment factors 412. In some embodiments, the output audio signal 414 may also be in the time-frequency domain, or inversely transformed to the time domain before being output. The domains of the input and output may depend on the implementation.

[0223] The spectrum adjustment processor 403 is configured to multiply the (time-frequency transformed) frequency bins of all channels within the frequency band k by the spectrum adjustment factor s(k) for each frequency band k. In other words, the spectrum adjustment processor 403 is configured to perform spectrum adjustment. The multiplication / spectral correction may be smoothed over time to avoid processing artifacts.

[0224] In other words, the focus processor 450 is configured to modify the frequency spectrum of the audio signal as well as the spatial metadata such that the process results in a parameterized spatial audio signal that has been modified according to the (de)focusing parameters.

[0225] about Figure 4b , showing that Figure 4aFlowchart 460 of the operation of the parameterized spatial audio input processor is shown in FIG.

[0226] like Figure 4b As shown in step 461, the initial operation is to receive a parameterized spatial audio signal (as well as focus / defocus parameters or other control information).

[0227] like Figure 4b As shown in step 463, the next operation is to modify the parameterized metadata and generate a spectrum adjustment factor.

[0228] like Figure 4b As shown in step 465, the next operation is to perform spectrum adjustment on the audio signal.

[0229] Furthermore, if Figure 4b As shown in step 467, the spectrally adjusted audio signal and the modified (and unmodified) metadata may be output.

[0230] about Figure 5a , a focus processor 550 configured to receive a multi-channel or object audio signal as input 500 is shown. In such an example, the focus processor may include a focus gain determiner 501. The focus gain determiner 501 is configured to receive focus / defocus parameters 508 and channel / object position / orientation information, which may be static or time-varying. The focus gain determiner 501 is configured to generate a directional gain f(k) parameter 512 based on the (de)focus parameters 508 (such as (de)focus direction, (de)focus amount, (de)focus control, and optional (de)focus distance and radius or (de)focus width) and spatial metadata information 502 from the input signal 500. In some embodiments, the channel signal directions are signaled, and in some embodiments, they are assumed. For example, when there are 6 channels, the directions may be assumed to be 5.1 audio channel directions. In some embodiments, there may be a lookup table that is used to determine the channel directions based on the number of channels.

[0231] In some embodiments there is no filter bank, in other words there is only one frequency band k. The directional gain f(k) for each audio channel is output to the focus gain processor 503 as focus gain.

[0232] In some embodiments, the focus gain processor 503 is configured to receive the audio signal and the focus gain value 512, and process the audio signal 506 based on the focus gain value 512 (per channel), potentially with some temporal smoothing. In some embodiments, the processing based on the focus gain value 512 can be multiplying the channel / object signal with the focus gain value.

[0233] The output of the focus gain processor 503 is the focus processed audio channel. The channel orientation / position information is not changed and is also provided as output 510.

[0234] In some embodiments, the defocusing process can be configured to be wider than for one direction. For example, the focus width β0 can be included as an input parameter. In these embodiments, the user can also generate a defocus arc. In another example, the focus distance d f and the focus radius γ0 as input parameters. In these embodiments, the user can generate a defocused sphere at the determined position. A similar process can also be used for other input spatial audio signal types.

[0235] In some embodiments, the audio object (spatial metadata) may include distance parameters that may also be taken into account. For example, the focus / defocus parameters may determine the focus position (direction and distance), and the radius parameters that control the focus / defocus area around that position. In such embodiments, the user may generate a parameter such as Figure 1c , and in the defocus pattern previously described. Similarly, other spatially related parameters can be defined to allow the user to control different shapes of the defocus area. In some embodiments, the attenuation of audio objects within the defocus area can be a fixed decibel number (e.g., 10 dB) multiplied by the attenuation of the desired defocus amount between 0 and 1, and audio objects outside the defocus direction are left without gain modification (or no gain or attenuation associated with the focusing operation is applied to audio objects outside the defocus direction). In the formulation of the directional gain f(k) (which will be output as the focus gain), the focus gain determiner 501 can use the same attenuation as in Figure 4a The directional gain f(k) is determined using the same formula described in the context of the ratio modifier and spectral adjustment factor determiner 401 in . The exception is that in the case of audio objects / channels there is usually only one frequency band and the spatial metadata usually only indicates the object direction / distance, not the ratio. When the distance is not available, a fixed distance can be assumed, for example, 2 meters.

[0236] about Figure 5b , showing that Figure 5a Flowchart 560 of the operation of the multi-channel / object audio input processor is shown in FIG.

[0237] like Figure 5b As shown in step 561, the initial operation is to receive a multi-channel / object audio signal, and in some embodiments also receive channel information, such as the number of channels and / or the distribution of channels (as well as focus / defocus parameters or other control information).

[0238] like Figure 5b As shown in step 563, the next operation is to generate a focus gain factor.

[0239] like Figure 5b As shown in step 565, the next operation is to apply a focus gain to each channel audio signal.

[0240] Furthermore, if Figure 5b As shown in step 567, the processed audio signal and the unmodified channel directions (and distances) may be output.

[0241] about Figure 6a , shows an example of an Ambisonic audio input based reproduction processor 650 (eg, which may be configured to receive audio from a processor such as Figure 3a The example shown in Figure 1 focuses the output of the processor).

[0242] In these examples, the reproduction processor may include an Ambisonic rotation matrix processor 601. The Ambisonic rotation matrix processor 601 is configured to receive the focus / defocus processed Ambisonic signal 600 and the viewing direction 602. The Ambisonic rotation matrix processor 601 is configured to generate a rotation matrix based on the viewing direction parameter 602. In some embodiments, this may use any suitable method, such as those methods applied to Ambisonic binauralization for head tracking (or more generally, such rotations of spherical harmonic functions are used in many fields including other fields besides audio). In turn, the rotation matrix is ​​applied to the Ambisonic audio signal. The result is a rotated Ambisonic signal with added focus / defocus 604, which is output to the Ambisonic to binaural filter 603.

[0243] The Ambisonic to binaural filter 603 is configured to receive the rotated Ambisonic signal with added focus / defocus 604. The Ambisonic to binaural filter 603 may include a pre-formulated 2xK finite impulse response (FIR) filter matrix that is applied to the K Ambisonic signals to generate 2 binaural signals 606. In this example, where a 4-channel FOA audio signal is shown, K=4. The FIR filters may be generated by a least squares optimization method with respect to a set of head-related impulse responses (HRIRs). An example of such a design process is to transform the HRIR dataset into frequency bins (e.g., by FFT) to obtain an HRTF dataset, and to determine a complex-valued processing matrix for each frequency bin that approximates the available HRTF dataset at the data points of the HRTF dataset in a least squares sense. When the complex-valued matrix is ​​determined for all frequency bins in this manner, the result may be inversely transformed (e.g., by inverse FFT) into a time-domain FIR filter. The FIR filter may also be windowed, for example, by using a Hann window.

[0244] In some embodiments, the rendering is not for headphones but for speakers. There are many known methods that can be used to render the Ambisonic signal as a speaker output. One example can be to linearly decode the Ambisonic signal to a target speaker configuration. When the order of the Ambisonic signal is high enough (e.g., at least 3rd order, but preferably 4th order), this can be applied with good expected spatial fidelity. In a specific example of such linear decoding, the Ambisonic decoding matrix can be designed to generate a speaker signal corresponding to a beam pattern that is approximately suitable for a vector basis amplitude translation (VBAP) beam pattern for the target speaker configuration in a least squares sense when applied to the Ambisonic signal (corresponding to the Ambisonic beam pattern). Processing the Ambisonic signal with such a designed Ambisonic decoding matrix can be configured to generate a speaker sound output. In such an embodiment, the reproduction processor is configured to receive information about the speaker configuration, and no rotation processing is required.

[0245] about Figure 6b , showing that Figure 6a A flow chart 660 of the operation of the Ambisonic input rendering processor is shown in FIG.

[0246] like Figure 6b As shown in step 661, the initial operation is to receive the focused / defocused Ambisonic audio signal (and the viewing direction).

[0247] like Figure 6bAs shown in step 663, the next operation is to generate a rotation matrix based on the viewing direction.

[0248] like Figure 6b As shown in step 665, the next operation is to apply the rotation matrix to the Ambisonic audio signal to generate a rotated focused / defocused Ambisonic audio signal.

[0249] Furthermore, if Figure 6b As shown in step 667, the next operation is to convert the Ambisonic audio signal into a suitable audio output format, such as a binaural format (or a multi-channel audio format or a speaker format).

[0250] Furthermore, if Figure 6b As shown in step 669, the output audio format is output.

[0251] about Figure 7a , shows an example of a reproduction processor 750 based on parameterized spatial audio input (e.g., which may be configured to receive a signal from a processor such as Figure 4a The example focus processor output shown in ).

[0252] In some embodiments, the reproduction processor includes a filter bank 701, which is configured to receive audio channels 700 audio signals and transform these audio channels into frequency bands (unless the input is already in a suitable time-frequency domain). Examples of suitable filter banks include short-time Fourier transform (STFT) and complex quadrature mirror filter (QMF) banks. The time-frequency audio signal 702 can be output to a parameterized binaural synthesizer 703.

[0253] In some embodiments, the reproduction processor includes a parameterized binaural synthesizer 703 configured to receive the time-frequency audio signal 702 and the modified (and unmodified) metadata 704, and further receive a viewing direction 706 (or suitable reproduction-related control or tracking information). In the context of 6DOF reproduction, the user position may be provided together with the viewing direction parameter.

[0254] The parametric binaural synthesizer 703 can be configured to implement any suitable known parametric spatial synthesis method configured to generate a binaural audio signal (in frequency band) 708, since the signal and metadata have been subjected to focus modification before the parametric binauralization block. A known method for parametric binaural synthesis is based on a directional to total energy ratio parameter in the frequency band, dividing the time-frequency audio signal 702 into directional and ambient part signals in the frequency band, processing the directional part in the frequency band with an HRTF corresponding to the directional parameter in the frequency band, processing the ambient part with a decorrelator to obtain binaural diffuse field coherence, and combining the processed directional and ambient parts. The binaural audio signal (in frequency band) 708 then has two channels, regardless of how many channels the time-frequency audio signal 702 has. Then, the binauralized time-frequency audio signal 708 can be passed to an inverse filter bank 705. The embodiment is further characterized in that the reproduction processor including the inverse filter bank 705 is configured to receive the binauralized time-frequency audio signal 708 and apply inverse filtering to the applied forward filter bank, thereby generating a binaural audio signal having a frequency band suitable for being listened to by headphones (in the case of the headphone). Figure 7a 7. The time-domain binauralized audio signal 710 of the focusing characteristics reproduced by the audio signal 710 (not shown).

[0255] In some embodiments, a suitable speaker synthesis method is used to replace the binaural audio signal output with the speaker channel audio signal output format from the parameterized spatial audio signal. Any suitable method can be used, for example, based on a suitable known method, the viewing direction parameter is replaced with information about the position of the speaker, and the parameterized binaural synthesizer 703 is replaced with a parameterized speaker synthesizer. A known method for parameterized speaker synthesis is based on a directional to total energy ratio parameter in the frequency band, dividing the time-frequency audio signal 702 into directional and ambient part signals in the frequency band, processing the direct part in the frequency band with a vector basis amplitude translation (VBAP) gain corresponding to the speaker configuration and directional parameters in the frequency band, processing the ambient part with a decorrelator to obtain an incoherent speaker signal, and combining the processed directional and ambient parts. The speaker audio signal (in the frequency band) then has the number of channels determined by the speaker configuration, regardless of how many channels the time-frequency audio signal 702 has.

[0256] about Figure 7b , showing that Figure 7a Flowchart 760 of the operation of the parameterized spatial audio input reproduction processor is shown in FIG.

[0257] like Figure 7b As shown in step 761, the initial operation is to receive a focused / defocused parametric spatial audio signal (as well as a viewing direction or other reproduction-related control or tracking information).

[0258] like Figure 7bAs shown in step 763, the next operation is to perform time-frequency conversion on the audio signal.

[0259] like Figure 7b As shown in step 765, the next operation is to apply a parameterized binaural (or speaker channel format) processor based on the time-frequency converted audio signal, metadata, and viewing direction (or other information).

[0260] Furthermore, if Figure 7b As shown in step 767, the next operation is to inverse transform the generated binaural or speaker channel audio signals.

[0261] Furthermore, if Figure 7b As shown in step 769, the output audio format is output.

[0262] Considering that when the audio signal is in the form of multi-channel audio and applied Figure 5a If the focus processor 550 in the reproduction processor is used for the speaker output of the reproduction processor, then in some embodiments the reproduction processor may include a pass-through, wherein the output speaker configuration is the same format as the input signal. In some embodiments where the output speaker configuration is different from the input speaker configuration, the reproduction processor may include a vector basis amplitude translation (VBAP) processor. In turn, each of the focused audio channels may be processed using VBAP (a known amplitude translation technique) to spatially reproduce them using the target speaker configuration. As a result, the output audio signal matches the output speaker setting.

[0263] In some embodiments, any suitable amplitude panning technique may be used to achieve the conversion from the first speaker configuration to the second speaker configuration. For example, the amplitude panning technique may include deriving an N×M matrix of amplitude panning gains that defines the conversion from the M channels of the first speaker configuration to the N channels of the second speaker configuration, and then using the matrix to multiply it with the channels of the intermediate spatial audio signal provided as a multi-channel speaker signal according to the first speaker configuration. The intermediate spatial audio signal may be understood as being a signal having a plurality of channels such as Figure 2a . As a non-limiting example, a derivation of the VBAP amplitude panning gain is provided in Pulkki, Ville, "Virtual sound source positioning using vector base amplitude panning," Journal of the Audio Engineering Society, Vol. 45, No. 6 (1997): pp. 456-466.

[0264] For binaural output, any suitable binauralization of the multi-channel loudspeaker signal format (and / or objects) may be implemented. For example, a typical binauralization may include processing the audio channels with a head-related transfer function (HRTF) and adding synthetic room reverberation to generate an auditory impression of the listening room. By employing the principles outlined, for example, in GB patent application GB1710085.0, the distance+direction (i.e., position) information of the audio object sounds may be used for 6DOF reproduction that moves with the user.

[0265] exist Figure 8 An example apparatus suitable for implementation in the form of a mobile phone or mobile device 901 running suitable software 903 is shown in FIG. The video may be reproduced, for example, by attaching the mobile phone 901 to a Daydream viewing type device (however, for the sake of clarity, video processing is not discussed here).

[0266] The audio bitstream obtainer 923 is configured to obtain an audio bitstream 924, for example, received / retrieved from a storage device. In some embodiments, the mobile device includes a decoder 925, which is configured to receive compressed audio and decode it. In the case of AAC decoding, an example of a decoder is an AAC decoder. The resulting decoded (e.g., Ambisonic, where an example implementation is as follows Figure 3a and 6a Example shown in ) The audio signal 926 can be forwarded to the focus processor 927.

[0267] The mobile phone 901 receives controller data 900 from an external controller at a controller data receiver 911 (e.g., via Bluetooth) and passes the data to a focus parameter (from controller data) determiner 921. The focus parameter (from controller data) determiner 921 determines the focus parameters, for example, based on the orientation of the controller device and / or button events. The focus parameters may include any kind of combination of the proposed focus parameters (e.g., focus / defocus direction, focus / defocus amount, focus / defocus height, and focus / defocus width). The focus parameters 922 are forwarded to a focus processor 927.

[0268] Based on the Ambisonic audio signals and the focus parameters, the focus processor 927 is configured to create modified Ambisonic signals 928 having desired focus characteristics. These modified Ambisonic signals 928 are forwarded to the Ambisonic to binaural processor 929. The Ambisonic to binaural processor 929 is also configured to receive head orientation information 904 from the orientation tracker 913 of the mobile phone 901. Based on the modified Ambisonic signals 928 and the head orientation information 904, the Ambisonic to binaural processor 929 is configured to create head tracked binaural signals 930, which can be output from the mobile phone and played back using, for example, headphones.

[0269] Fig. 9 An example device (or focus / defocus parameter controller) 1050 is shown, which can be configured to control or generate suitable focus / defocus parameters, such as focus / defocus direction, focus / defocus amount, and focus / defocus width. A user of the device can be configured to select a focus direction by pointing the controller in a desired direction 1009 and pressing a select focus direction button 1005. The controller has an orientation tracker 1001, and the orientation information can be used to determine the focus / defocus direction (e.g., in a Figure 8 The focus parameters (from controller data) shown in determiner 921). In some embodiments, the focus / defocus direction can be visualized in a visual display when the focus / defocus direction is selected.

[0270] In some embodiments, you can use the focus volume button (in Fig. 9 The focus amount is controlled by pressing the + and - buttons 1007. Each press increases / decreases the focus amount by a certain amount, for example, 10 percentage points. In some embodiments, when the focus amount is set to 0% and the user presses the minus button, the focus amount is set to 10% and the focus / defocus control is set to "defocus" mode. Correspondingly, if the focus amount is set to 0% and the user presses the plus button, the focus amount is set to 10% and the focus / defocus control is set to "focus" mode.

[0271] In some embodiments, it may be desirable to further specify the focusing or defocusing process, for example by determining a desired frequency range or spectral characteristics of the focused signal. In particular, it may be useful to emphasize or de-emphasize the audio spectrum in the speech frequency range to improve clarity or block out talkers, for example, by attenuating low frequency content (e.g., below 200 Hz) and high frequency content (e.g., above 8 kHz) to focus, thereby leaving a particularly useful frequency range associated with speech.

[0272] Similarly, when the user indicates a direction in which to defocus, the audio processing system can analyze the spectrum or type (e.g., speech, noise) of the interference source in the direction in which to attenuate. Then, based on the analysis, the system can determine a frequency range or amount of defocus per frequency that is well suited for the interference source. For example, the interference source may be a device that generates high-frequency noise, and for this defocus direction, the high frequencies will be attenuated more than, for example, mid- and low-frequency frequencies. In another example, there is a speaker in the defocus direction, and therefore, the amount of defocus can be configured by frequency to primarily suppress a typical speech frequency range.

[0273] It should be understood that the focused processed signal may be further processed using any known audio processing technique, such as automatic gain control or enhancement techniques (eg, bandwidth extension, noise suppression).

[0274] In some further embodiments, focus / defocus parameters (including direction, amount, and control) are generated by the content creator and these parameters are sent along with the spatial audio signal. For example, in a VR audio-visual nature documentary with a live narrator, instead of the user needing to select the direction of the narrator to be defocused, a dynamic focus parameter preset can be selected. The preset may have been fine-tuned by the content creator to follow the movements of the narrator. For example, defocusing is only enabled when the narrator is speaking. In other words, the content creator can generate some expected or estimated preference profiles as focus / defocus parameters. This approach is beneficial because only one spatial audio signal needs to be transmitted, but different preference profiles can be added. Furthermore, legacy players that do not support focusing can be configured to simply decode Ambisonic or other signal types without applying focus / defocus processing.

[0275] exist Fig.10 An example processing output based on the implementation described for an Ambisonic signal is shown in . In this example, there are three sound sources within the audio scene: a talker in front, a talker at -90 degrees to the right, and a white noise interference source at 110 degrees to the left. Fig.10It is shown how the focus process is used to broadly emphasize the direction where the noise source is located when the focus / defocus control is set to "focus", and how the focus process is used to broadly de-emphasize the direction where the noise source is located when the focus / defocus control is set to "defocus", while retaining both talker signals in the spatial audio output. Thus, in row 1111, in an example case shown by an Ambisonic signal with a talker in front (particularly shown as signal X), a talker at -90 degrees to the right (particularly shown as signal Y), and a noise interferer at 110 degrees to the left (shown as all signals), the Ambisonic signal is shown in 3 columns (omnidirectional W 1101, horizontal dipoles Y 1103 and X 1105). The next row 1113 shows the Ambisonic audio signal with full focus processing on the noise source. The bottom row 1115 shows the Ambisonic audio signal with full defocus processing on the noise source (i.e., de-emphasis of the noise), so that the speech source is generally active.

[0276] about Fig.11 , shows an example electronic device that can be used as an analysis or synthesis device. The device can be any suitable electronic device or apparatus. For example, in some embodiments, the device 1700 is a mobile device, a user device, a tablet computer, a computer, an audio playback device, etc.

[0277] In some embodiments, device 1200 includes at least one processor or central processing unit 1207. Processor 1207 may be configured to execute various program codes, such as the methods described herein.

[0278] In some embodiments, the device 1200 includes a memory 1211. In some embodiments, at least one processor 1207 is coupled to the memory 1211. The memory 1211 can be any suitable storage component. In some embodiments, the memory 1211 includes a program code portion for storing program codes that can be implemented on the processor 1207. In addition, in some embodiments, the memory 1211 can also include a storage data portion for storing data (e.g., data that has been processed or will be processed according to the embodiments described herein). When necessary, the implemented program code stored in the program code portion and the data stored in the storage data portion can be obtained by the processor 1207 via the memory-processor coupling.

[0279] In some embodiments, the interface device 1200 includes a user interface 1205. In some embodiments, the user interface 1205 may be coupled to a processor 1207. In some embodiments, the processor 1207 may control the operation of the user interface 1205 and receive input from the user interface 1205. In some embodiments, the user interface 1205 may enable a user to enter commands to the device 1200, for example, via a keypad. In some embodiments, the user interface 1205 may enable a user to obtain information from the device 1200. For example, the user interface 1205 may include a display configured to display information from the device 1200 to the user. In some embodiments, the user interface 1205 may include a touch screen or touch interface that enables information to be input into the device 1200 and to display information to a user of the device 1200.

[0280] In some embodiments, the device 1200 includes an input / output port 1209. In some embodiments, the input / output port 1209 includes a transceiver. In such embodiments, the transceiver can be coupled to the processor 1207 and configured to communicate with other devices or electronic devices, for example, via a wireless communication network. In some embodiments, the transceiver or any suitable transceiver or transmitter and / or receiver components can be configured to communicate with other electronic devices or devices via wired or wired coupling.

[0281] The transceiver can communicate with other devices via any suitable known communication protocol. For example, in some embodiments, the transceiver can use a suitable Universal Mobile Telecommunications System (UMTS) protocol, a wireless local area network (WLAN) protocol such as IEEE 802.X, a suitable short-range radio frequency communication protocol such as Bluetooth, or an infrared data communication path (IRDA).

[0282] The transceiver input / output port 1209 may be configured to receive signals and, in some embodiments, obtain focusing parameters as described herein.

[0283] In some embodiments, device 1200 may be used to generate suitable audio signals by executing suitable code using processor 1207. Input / output port 1209 may be coupled to any suitable audio output, such as to a multi-channel speaker system and / or headphones (which may be head-mounted tracking or non-tracking headphones), etc.

[0284] In general, various embodiments of the present invention may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although various aspects of the present invention may be illustrated and described as block diagrams, flow charts, or using some other graphical representation, it is well known that the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0285] Embodiments of the present invention may be implemented by computer software executable by a data processor of a mobile device (such as in a processor entity), or by hardware, or by a combination of software and hardware. In addition, in this regard, it should be noted that any block of the logic flow as in the accompanying drawings may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on a physical medium such as a memory chip or a memory block implemented within a processor, on a magnetic medium such as a hard disk or a floppy disk, and on an optical medium such as a DVD and its data variant CD.

[0286] The memory may be of any type suitable for the local technical environment and may be implemented using any appropriate data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor may be of any type suitable for the local technical environment and may include, as non-limiting examples, one or more of a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a gate level circuit based on a multi-core processor architecture, and a processor.

[0287] Embodiments of the present invention may be practiced in various components such as integrated circuit modules. The design of integrated circuits is generally a highly automated process. Complex and powerful software tools are available to convert a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0288] Programs, such as those offered by Synopsys, Inc. of Mountain View, Calif., and Cadence Design, of San Jose, Calif., can automatically route conductors and position components on a semiconductor chip using well-established design rules and a library of pre-stored design modules. Once the design of a semiconductor circuit is complete, the resulting design in a standardized electronic format (e.g., Opus, GDSII, etc.) can be transmitted to a semiconductor manufacturing facility or "fab" for fabrication.

[0289] The foregoing description has provided a complete and useful description of exemplary embodiments of the present invention by way of exemplary and non-limiting examples. However, various modifications and adaptations will become apparent to those skilled in the relevant art in view of the above description when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of the present invention will still fall within the scope of the present invention as defined by the appended claims.

Claims

1. A device comprising: at least one processor; as well as at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least: determining at least one region within an audio scene represented by a spatial audio signal; obtaining at least one focus / defocus information; processing the spatial audio signal based at least in part on the at least one focus / defocus information to generate a processed spatial audio signal representing a modified audio scene, at least in part in which at least one determined region is at least in part relatively de-emphasized relative to other parts of the spatial audio signal; as well as The processed spatial audio signal is outputted, wherein the modified audio scene comprises at least at least one determined region that is de-emphasized at least in part relative to the other parts of the spatial audio signal according to the at least one focus / defocus information.

2. The device according to claim 1, wherein: The at least one focus / defocus information comprises at least one of the following: Defocus amount; Focus amount; Defocus shape; Focus on shape; Defocus direction; or Focus direction.

3. The device according to claim 1, wherein: Processing the spatial audio signal includes the instructions, when executed by the at least one processor, causing the apparatus to perform at least one of: reducing at least partially an emphasis of at least one determined region at least partially relative to said other parts of said spatial audio signal; or An emphasis of the other parts of the spatial audio signal is at least partially increased with respect to the at least one determined area.

4. The device according to claim 1, wherein: Processing the spatial audio signal includes the instructions, when executed by the at least one processor, causing the apparatus to perform at least one of: at least partially reducing a sound level of at least one determined area relative to said other parts of said spatial audio signal at least partially based on said at least one focus / defocus information; or The sound level of the other part of the spatial audio signal is at least partially increased with respect to the at least one determined area depending on the at least one focus / defocus information.

5. The device according to claim 1, wherein: The at least one focus / defocus information comprises at least a focus / defocus shape, wherein processing the spatial audio signal comprises the instructions, when executed by the at least one processor, causing the apparatus to perform at least one of: reducing at least partially an emphasis of at least one determined area relative to said other parts of the spatial audio signal at least partially within said focus / defocus shape; or An emphasis of the other parts of the spatial audio signal is at least partially increased within the focus / defocus shape with respect to at least one determined area.

6. The device according to claim 1, wherein: The at least one focus / defocus information comprises at least a focus / defocus shape, wherein processing the spatial audio signal comprises the instructions, when executed by the at least one processor, causing the apparatus to perform at least one of: reducing, at least partially, a sound level of at least one determined area relative to said other parts of the spatial audio signal within said focus / defocus shape, at least partially, based on said at least one focus / defocus information; or The sound level of the other part of the spatial audio signal is at least partially increased within the focus / defocus shape according to the at least one focus / defocus information with respect to the at least one determined area.

7. The device according to claim 1, wherein: The at least one focus / defocus information comprises at least a focus / defocus shape, wherein the focus / defocus shape comprises at least one of the following: Focus / defocus shape width; Focus / defocus shape height; Focus / defocus shape radius; Focus / defocus shape distance; Focus / defocus shape depth; Focus / defocus shape range; Focus / defocus shape diameter; or Focus / defocus shape characterizer.

8. The device according to claim 1, wherein: The spatial audio signal and the processed spatial audio signal include a corresponding panoramic surround sound signal, and wherein processing the spatial audio signal includes the instructions, when executed by the at least one processor, causing the apparatus to perform, for one or more frequency subbands, at least one of: extracting from the spatial audio signal a single-channel target audio signal representing a sound component arriving from at least one determined area; or generating a focused spatial audio signal, wherein the focused spatial audio signal is arranged at a spatial position defined at least in part based on the at least one focus / defocus information; and The processed spatial audio signal is created as a linear combination of the focused spatial audio signal subtracted from the spatial audio signal, wherein the focused spatial audio signal or at least one of the spatial audio signals is scaled by a corresponding scaling factor derived at least in part based on the at least one focus / defocus information to reduce a relative sound level in at least one determined zone.

9. The device according to claim 1, wherein: The spatial audio signal and the processed spatial audio signal comprise respective parameterized spatial audio signals, wherein the parameterized spatial audio signals respectively comprise one or more audio channels and spatial metadata, wherein the spatial metadata comprises respective direction indications and energy ratio parameters for a plurality of frequency subbands, wherein processing the spatial audio signals comprises the instructions, when executed by the at least one processor, causing the apparatus to: calculating, for one or more frequency subbands, a respective angular difference between a direction of at least one determined region and a direction indicated for a respective frequency subband of the spatial audio signal; deriving, for the one or more frequency subbands, respective gain values ​​based at least in part on the angle differences calculated for the respective frequency subbands, using a predefined angle difference function and a scaling factor derived based at least in part on the at least one focus / defocus information; calculating, for one or more frequency subbands of the processed spatial audio signal, corresponding updated directional energy values ​​based at least in part on energy ratio parameters of corresponding frequency subbands of the spatial audio signal and the gain values; calculating, for the one or more frequency bands of the processed spatial audio signal, respective updated ambient energy values ​​based at least in part on energy ratio parameters of respective frequency subbands of the spatial audio signal and the scaling factors; calculating, for the one or more frequency subbands of the processed spatial audio signal, corresponding modified energy ratio parameters based at least in part on the updated directional energy value divided by a sum of the updated directional energy value and the updated ambient energy value; calculating, for the one or more frequency subbands of the processed spatial audio signal, corresponding spectral adjustment factors based at least in part on a sum of the updated directional energy value and the updated ambient energy value; and The processed spatial audio signal is composed, the processed spatial audio signal comprising the one or more audio channels of the spatial audio signal, the direction indication of the spatial audio signal, the modified energy ratio parameter, and the spectral adjustment factor.

10. The device according to claim 1, wherein: The spatial audio signal and the processed spatial audio signal comprise respective multi-channel speaker signals according to a first predefined speaker configuration, and wherein processing the spatial audio signal comprises the instructions, when executed by the at least one processor, causing the apparatus to: calculating a respective angular difference between a direction of at least one determined zone and a loudspeaker direction indicated for a respective channel of the spatial audio signal; deriving, for respective channels of the spatial audio signal, respective gain values ​​based at least in part on the angle differences calculated for the respective channels, using a predefined angle difference function and a scaling factor derived based at least in part on the at least one focus / defocus information; deriving one or more modified audio channels by multiplying the corresponding channels of the spatial audio signal by gain values ​​derived for the corresponding channels; and The one or more modified audio channels are provided as the processed spatial audio signal.

11. The device according to claim 1, wherein: Obtaining the at least one focus / defocus information comprises the at least one memory storing instructions which, when executed with the at least one processor, cause the apparatus to: The at least one focus / defocus information is obtained based at least in part on information from a sensor device, wherein the information from the sensor device comprises at least one of: at least one sensor direction; or At least one user input.

12. A method comprising: determining at least one region within an audio scene represented by a spatial audio signal; obtaining at least one focus / defocus information; processing the spatial audio signal based at least in part on the at least one focus / defocus information to generate a processed spatial audio signal representing a modified audio scene, at least in part in which at least one determined region is at least in part relatively de-emphasized relative to other parts of the spatial audio signal; as well as The processed spatial audio signal is outputted, wherein the modified audio scene comprises at least at least one determined region that is de-emphasized at least in part relative to the other parts of the spatial audio signal according to the at least one focus / defocus information.

13. The method according to claim 12, wherein: The at least one focus / defocus information comprises at least one of the following: Defocus amount; Focus amount; Defocus shape; Focus on shape; Defocus direction; or Focus direction.

14. The method of claim 12, further comprising at least one of the following: reducing at least partially an emphasis of at least one determined region at least partially relative to said other parts of said spatial audio signal; or An emphasis of the other parts of the spatial audio signal is at least partially increased with respect to the at least one determined area.

15. The method of claim 12, further comprising at least one of the following: at least partially reducing the sound level of at least one determined area relative to said other parts of the spatial audio signal at least partially based on the at least one focus / defocus information; or The sound level of the other part of the spatial audio signal is at least partially increased with respect to the at least one determined area depending on the at least one focus / defocus information.

16. The method according to claim 12, wherein: The at least one focus / defocus information comprises at least a focus / defocus shape, wherein the focus / defocus shape comprises at least one of the following: Focus / defocus shape width; Focus / defocus shape height; Focus / defocus shape radius; Focus / defocus shape distance; Focus / defocus shape depth; Focus / defocus shape range; Focus / defocus shape diameter; or Focus / defocus shape characterizer.

17. The method according to claim 12, wherein: Obtaining the at least one focus / defocus information comprises: The at least one focus / defocus information is obtained based at least in part on information from a sensor device, wherein the information from the sensor device comprises at least one of: at least one sensor direction; or At least one user input.

18. A non-transitory computer readable medium comprising program instructions stored thereon, the program instructions being configured to at least: determining at least one region within an audio scene represented by a spatial audio signal; obtaining at least one focus / defocus information; processing the spatial audio signal based at least in part on the at least one focus / defocus information to generate a processed spatial audio signal representing a modified audio scene, at least in part in which at least one determined region is at least in part relatively de-emphasized relative to other parts of the spatial audio signal; as well as The processed spatial audio signal is outputted, wherein the modified audio scene comprises at least at least one determined region that is de-emphasized at least in part relative to the other parts of the spatial audio signal according to the at least one focus / defocus information.

19. The non-transitory computer readable medium of claim 18, wherein: The at least one focus / defocus information comprises at least one of the following: Defocus amount; Focus amount; Defocus shape; Focus on shape; Defocus direction; or Focus direction.

20. The non-transitory computer readable medium of claim 18, further comprising program instructions stored thereon, the program instructions for performing at least one of: reducing at least partially an emphasis of at least one determined region at least partially relative to said other parts of said spatial audio signal; or An emphasis of the other parts of the spatial audio signal is at least partially increased with respect to the at least one determined area.

Images