Audio device and method of audio processing

Abstract

An audio apparatus and method employ a first renderer circuit, wherein the first renderer circuit is arranged to render an audio element by generating a first set of audio signals for a set of loudspeakers; and a second renderer circuit, wherein the second renderer circuit is arranged to render the audio element by generating a second set of audio signals for headphones. The audio apparatus and method select between the first renderer circuit and the second renderer circuit such that rendering of at least a first portion of a first audio element is responsive to a first audio rendering property indicator received in the audio signals. The first audio rendering property indicator indicates an audio format of the first audio element.

Description

[0001] This application is a divisional application of patent application 201980055860.6, filed on August 20, 2019, entitled "Audio Device and Method for Audio Processing". Technical Field

[0002] The present invention relates to audio devices and methods of audio processing, and particularly, but not exclusively, to the use of these devices to support augmented / virtual reality applications. Background Technology

[0003] In recent years, with the continuous development and launch of new services and methods for utilizing and consuming audiovisual content, the types and scope of experiences based on audiovisual content have increased significantly. Specifically, many spatial and interactive services, applications, and experiences are being developed to provide users with more immersive and engaging experiences.

[0004] Examples of such applications are the rapidly becoming mainstream Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) applications, many of which target the consumer market. Numerous standards are also under development by various standardization bodies. These standardization activities are proactively developing standards for various aspects of VR / AR / MR systems, including, for example, streaming, distribution, and rendering.

[0005] VR applications tend to provide a user experience corresponding to different worlds / environments / scenes, while AR (including Mixed Reality) applications tend to provide a user experience corresponding to the user's current environment but with additional or added virtual objects or information. Therefore, VR applications tend to provide fully immersive, synthetically generated worlds / scenes, while AR applications tend to provide partially synthetic worlds / scenes superimposed on the real-world scene where the user physically exists. However, the terms are often used interchangeably and have a high degree of overlap. In the following text, the term Virtual Reality / VR will be used to refer to both Virtual Reality and Augmented Reality.

[0006] As an example, an increasingly popular service delivers images and audio in a way that allows users to actively and dynamically interact with the system to change rendering parameters to suit the user's position and orientation as they move and change. A particularly attractive feature in many applications is the ability to alter the viewer's effective viewing position and orientation, for example, allowing the viewer to move and "look around" within the scene being presented.

[0007] This feature allows for the provision of virtual reality experiences to users. It allows users to move (relatively) freely within a virtual environment and dynamically change their position and the place they are viewing. Typically, such virtual reality applications are based on a 3D model of the scene, which is dynamically evaluated to provide a specific requested view. For computers and consoles, this approach is well-known in applications such as gaming (e.g., in the first-person shooter genre).

[0008] Especially for virtual reality applications, it is desirable to present three-dimensional images. In fact, to optimize the viewer's immersion, it is generally preferred to present the user experience in a three-dimensional scene. Ideally, the virtual reality experience should allow the user to choose their own position, camera viewpoint, and time relative to the virtual world.

[0009] Typically, virtual reality applications are inherently limited to models based on predetermined scenes and often on artificially created virtual world models. In some applications, the virtual reality experience can be provided based on real-world captures. In many cases, this approach tends to be based on a virtual model of the real world constructed from real-world captures. The virtual reality experience is then generated by evaluating this model.

[0010] Many current methods tend to be suboptimal, and often tend to have high computational or communication resource requirements and / or provide a suboptimal user experience with, for example, reduced quality or limited degrees of freedom.

[0011] As an example of application, virtual reality glasses that allow viewers to experience captured 360° (panoramic) or 180° video are already entering the market. These 360° videos are typically pre-captured using camera equipment, where individual images are stitched together to form a single spherical map. Common stereoscopic formats for 180° or 360° video are top / bottom and left / right. Similar to non-panoramic stereoscopic video, the left-eye and right-eye images are, for example, compressed into portions of a single H.264 video stream.

[0012] In addition to virtual rendering, most VR / AR applications also provide a corresponding audio experience. In many applications, the audio preferably provides a spatial audio experience, where the audio source is perceived as arriving from a location corresponding to the location of a corresponding object in the virtual scene. Therefore, the audio and video scenes are preferably perceived as consistent, and where both provide a fully spatial experience.

[0013] For audio, the focus until now has been primarily on headphone reproduction using binaural audio rendering technology. In many cases, headphone reproduction provides users with a highly immersive and personalized experience. Using head tracking, the rendering can respond to the user's head movements, which greatly enhances the sense of immersion.

[0014] Recently, both in the market and in standards discussions, use cases involving the "social" or "sharing" aspects of VR (and AR) have begun to emerge, namely the possibility of sharing experiences with others. These can be people in different locations, or people in the same location (or a combination of both). For example, several people in the same room can share the same VR experience using projections (audio and video) of each participant present in VR content / scenes. Similarly, in a multi-player game, each player may have a different location in the game scene and therefore different projections of the audio and video scenes.

[0015] As a specific example, MPEG attempts to standardize bitstreams and decoders for realistic, immersive AR / VR experiences with six degrees of freedom. Social VR is an important feature, allowing users to interact in shared environments (gaming, teleconferencing, online shopping, etc.). The concept of social VR also helps make VR experiences feel more like social activities for users who are physically in the same location but experience perceptual isolation from their physical environment, such as with head-mounted displays or other VR headsets.

[0016] The drawback of using headphones in such “social” or “shared” AR (or VR) use cases is that, with each user wearing an individual headphone, users in the same location (e.g., a room) are at least partially acoustically isolated from each other, which reduces the “social” aspect of the experience (e.g., it becomes difficult or inconvenient for people standing close to each other to have a natural conversation).

[0017] This can be addressed by using speakers instead of headphones for audio reproduction. However, this has the drawback that audio reproduction cannot be freely adjusted and customized for individual users. For example, it is difficult to make the audio reproduction dynamically adapt to head movements, and especially to changes in each individual user's head orientation. This effect is crucial for immersive experiences, and therefore speakers tend to be suboptimal for generating an optimized user experience.

[0018] Therefore, methods for improving audio processing, specifically for virtual / augmented / mixed reality experiences / applications, will be advantageous. Specifically, methods that allow for improved operation, increased flexibility, reduced complexity, easier implementation, improved audio experience, more consistent perception of audiovisual scenes, improved customization, improved personalization; improved virtual reality experiences and / or improved performance and / or operation will be advantageous. Summary of the Invention

[0019] Therefore, the present invention attempts to mitigate, alleviate or eliminate one or more of the disadvantages mentioned above, preferably alone or in any combination.

[0020] According to one aspect of the present invention, an audio apparatus is provided, comprising: a receiver for receiving data describing an audio scene, the data including audio data for a set of audio elements corresponding to an audio source in the scene and metadata including at least a first audio rendering property indicator for a first audio element of the set of audio elements; a first renderer for rendering the audio elements by generating a first set of audio signals for a set of speakers; a second renderer for rendering the audio elements by generating a second set of audio signals for headphones; and a selector arranged to select between the first renderer and the second renderer in response to the first audio rendering property indicator for rendering at least a first portion of the first audio element; wherein the audio rendering property indicator indicates whether the first portion of the first audio element is associated with a listener posture-related location or with a listener posture-independent location.

[0021] In many embodiments, this method can provide an improved user experience, and specifically for many virtual reality (including augmented and mixed reality) applications, including social or shared experiences. The method can use hybrid rendering to provide improved performance. For example, in many embodiments, it can allow users in the same room to talk directly more easily while still providing targeted and personalized rendering of the audio scene.

[0022] Audio rendering property indicators can indicate whether the first audio element represents an audio source with spatial properties that are fixed to head orientation or not (corresponding to listener posture-dependent and listener posture-independent positions, respectively). This method can reduce complexity and resource requirements.

[0023] In some embodiments, the apparatus may include a first driver for driving the set of speakers according to a first set of audio signals and a second driver for driving the headphones according to a second set of audio signals. The first set of audio signals may specifically be a set of surround signals, and the second set of audio signals may specifically be binaural stereo signals.

[0024] The first audio rendering property indicator can indicate the rendering property to be applied to the first audio element or the property of the first audio element.

[0025] According to an optional feature of the invention, the apparatus further includes a listener posture receiver for receiving a listener posture indicating the listener's posture, and the first renderer is arranged to generate the first set of audio signals independently of the listener posture, and the second renderer is arranged to generate the second set of audio signals in response to the listener posture.

[0026] This audio device can provide a highly advantageous and flexible user experience, allowing for close consistency between, for example, listener movement and perceived audio scene. Posture can refer to position and / or orientation data, and may also be referred to as orientation. Listener posture can be an indication of the listener's position, an indication of the listener's orientation, or a combination of both. Posture / or orientation can be represented by one or more values ​​that provide an indication of position and / or direction.

[0027] According to an optional feature of the invention, the apparatus is arranged to generate audio signals for a plurality of listeners, wherein the first renderer is arranged to generate a first set of audio signals as a common set of audio signals for the plurality of listeners; and the second renderer is arranged to generate a second set of audio signals for headphones for a first listener among the plurality of listeners, and to generate a third set of audio signals for headphones for a second listener among the plurality of listeners.

[0028] This audio device can provide favorable support for multiple users. In many applications, improved support can be implemented with low complexity and resource usage, but provides an attractive user experience that is generally consistent and natural in its audio quality.

[0029] The second set of audio signals can be generated in response to the first listener's first listener posture, and the third set of audio signals can be generated in response to the second listener's second listener posture. The first set of audio signals can be generated independently of the listener's posture.

[0030] According to an optional feature of the invention, the first portion is the frequency sub-range of the first audio element.

[0031] In many embodiments, this can provide improved performance.

[0032] According to an optional feature of the invention, the selector is arranged to select different renderers, the first renderer and the second renderer, for the first portion of the first audio element and for the second portion of the first audio element.

[0033] In many embodiments, this can provide an improved user experience. The selector can be specifically arranged to select different renderers for different frequency ranges of the first audio element.

[0034] In many applications, this can provide an efficient approach. Audio rendering property indicators can indicate whether the first audio element is part of the story.

[0035] According to an optional feature of the invention, the audio rendering property indicator indicates the audio format of the first audio element.

[0036] In many embodiments, this can provide an improved user experience. The audio rendering property indicator can indicate an audio format from a set of audio formats, including at least one audio format from the group consisting of: audio object formats; high-order stereo reverb audio formats; and audio channel signal audio formats.

[0037] According to an optional feature of the invention, the audio rendering property indicator indicates the audio source type of the first audio element.

[0038] In many embodiments, this can provide an improved user experience. The audio rendering property indicator can indicate the audio source type from a set of audio source types, including at least one audio source type from the following group: speech audio; music audio; foreground audio; background audio; narration audio; and narrator audio.

[0039] According to an optional feature of the invention, the audio rendering property indicator indicates the guided rendering property for rendering the first audio element.

[0040] In many embodiments, this can provide an improved user experience and / or performance.

[0041] According to an optional feature of the invention, the audio rendering property indicator indicates whether the first portion of the first audio item is intended for rendering on a speaker or for rendering on headphones.

[0042] In many embodiments, this can provide an improved user experience and / or performance.

[0043] According to an optional feature of the invention, the receiver is further arranged to receive visual data indicating a virtual scene corresponding to the audio scene, and the audio rendering property indicator indicates whether the first audio element represents an audio source corresponding to a virtual scene object.

[0044] In many embodiments, this can provide an improved user experience and / or performance.

[0045] In some embodiments, an audio rendering property indicator may indicate whether a first audio element represents an audio source corresponding to a scene object within a viewport determined for the current listener's posture.

[0046] According to an optional feature of the invention, the apparatus further includes a user input terminal for receiving user input, and wherein the selector is arranged to select between the first renderer and the second renderer in response to the user input for rendering at least the first portion of the first audio element.

[0047] In many embodiments, this can provide an improved user experience.

[0048] According to an optional feature of the invention, the selector is arranged to determine the audio properties of the first audio element and, in response to the audio properties, to select between the first renderer and the second renderer for rendering at least the first portion of the first audio element.

[0049] In many embodiments, this can provide an improved user experience and / or performance.

[0050] According to one aspect of the present invention, an audio processing method is provided, comprising: receiving data describing an audio scene, the data including audio data for a set of audio elements corresponding to an audio source in the scene and metadata including at least a first audio rendering property indicator for a first audio element in the set of audio elements; rendering the audio elements by generating a first set of audio signals for a set of speakers; rendering the audio elements by generating a second set of audio signals for headphones; and selecting, in response to the first audio rendering property indicator, at least a first portion of the first audio element rendered for the set of speakers and at least a first portion of the first audio element rendered for the headphones; wherein the audio rendering property indicator indicates whether the first portion of the first audio element is associated with a listener posture-related location or with a listener posture-independent location.

[0051] These and other aspects, features and advantages of the invention will become apparent with reference to one or more embodiments described below, and will be illustrated with reference to one or more embodiments described below. Attached Figure Description

[0052] Embodiments of the invention will be described by way of example only with reference to the accompanying drawings, in which...

[0053] Figure 1 The illustration shows an example of a client-server based virtual reality system; and

[0054] Figure 2 Examples of elements of an audio device according to some embodiments of the present invention are illustrated. Detailed Implementation

[0055] Virtual reality (including augmented and mixed reality) experiences that allow users to move around in virtual or augmented worlds are becoming increasingly popular, and services that meet this demand are being developed. In many of these approaches, virtual and audio data can be dynamically generated to reflect the user's (or viewer's) current posture.

[0056] In this field, the terms orientation and pose are used as common terms for position and / or direction / orientation. For example, the combination of position and direction / orientation of an object, camera, head, or view can be referred to as orientation or pose. Therefore, an orientation or pose indication can include up to six values / components / degrees of freedom, where each value / component typically describes an individual attribute of the corresponding object's position / location or orientation / orientation. Of course, in many cases, such as if one or more components are considered fixed or unrelated, orientation or pose can be represented by fewer components (e.g., if all objects are considered to be at the same height and have a horizontal orientation, four components can provide a complete representation of the object's pose). In the following text, the term pose is used to refer to position and / or orientation that can be represented by one to six values ​​(corresponding to the maximum possible degrees of freedom).

[0057] Many VR applications are based on pose, which has the maximum degree of freedom, i.e., three degrees of freedom each in position and orientation, resulting in a total of six degrees of freedom. Therefore, pose can be represented by a set or vector of six values ​​representing these six degrees of freedom, and thus, a pose vector can provide a three-dimensional position and / or three-dimensional orientation indication. However, it should be understood that in other embodiments, pose can also be represented by fewer values.

[0058] Systems or entities that offer the maximum degrees of freedom to the viewer are typically referred to as having 6 degrees of freedom (6DoF). Many systems and entities offer only orientation or position, and these are typically referred to as having 3 degrees of freedom (3DoF).

[0059] Typically, virtual reality applications generate 3D outputs in the form of separate view images for the left and right eyes. These can then be presented to the user using suitable devices, such as individual left and right eye displays typically found in VR headsets. In other embodiments, one or more view images may be presented, for example, on an autostereoscopic display, or in some embodiments, only a single 2D image may be generated (e.g., using a conventional 2D display).

[0060] Similarly, for a given viewer / user / listener posture, an audio representation of the scene can be provided. The audio scene is typically rendered to provide a spatial experience where the audio source is perceived as originating from a desired location. Since the audio source can be static within the scene, a change in the user's posture will cause a change in the relative position of the audio source with respect to the user's posture. Therefore, the spatial perception of the audio source must change to reflect its new position relative to the user. Audio rendering can be adapted accordingly based on the user's posture.

[0061] Viewer or user gesture input can be determined in different ways across different applications. In many embodiments, the user's physical movement can be directly tracked. For example, a camera monitoring the user's area can detect and track the user's head (or even eyes (eye tracking)). In many embodiments, the user can wear a VR headset that can be tracked by external and / or internal devices. For example, the headset may include accelerometers and gyroscopes that provide information about the headset and therefore the head's movement and rotation. In some examples, the VR headset may emit signals or include (e.g., visual) identifiers that enable external sensors to determine the VR headset's position.

[0062] In some systems, viewer posture can be provided through manual means, such as by the user manually controlling a joystick or similar manual input. For example, a user can manually move the virtual viewer around in the virtual scene by controlling a first analog joystick with one hand, and manually control the direction the virtual viewer is looking by manually moving a second analog joystick with the other hand.

[0063] In some applications, a combination of manual and automated methods can be used to generate input viewer poses. For example, a helmet can track head orientation, and the viewer's movement / position within the scene can be controlled by the user using a joystick.

[0064] In some systems, VR applications can be provided locally to viewers via, for example, a standalone device that does not use any remote VR data or processing, or even has no access to any remote VR data or processing. For example, a device such as a game console may include storage for storing scene data, inputs for receiving / generating viewer poses, and a processor for generating corresponding images based on the scene data.

[0065] In other systems, VR applications can be implemented and executed remotely to the viewer. For example, a local device for the user can detect / receive motion / pose data transmitted to a remote device, which processes the data to generate the viewer's pose. The remote device can then generate a view image appropriate for the viewer's pose based on scene data describing the scene. The view images are then transmitted to the local device in which they are presented. For example, the remote device can directly generate a video stream (typically a stereoscopic / 3D video stream) that is directly presented through the local device.

[0066] Similarly, remote devices can generate audio scenes that reflect a virtual audio environment. In many embodiments, this can be accomplished by generating audio elements that correspond to the relative positions of different audio sources in the virtual audio environment, wherein these audio elements are rendered as if they are perceived at the corresponding positions.

[0067] For example, a remote device can generate audio data representing an audio scene and can transmit audio components / objects / signals or other audio elements corresponding to different audio sources in the audio scene, along with location information indicating the location of these (which, for moving objects, may change dynamically, for example). Audio elements may include elements associated with a specific location, but may also include elements for more distributed or diffuse audio sources. For example, audio elements representing general (non-localized) background sound, ambient sound, diffuse reverberation, etc., can be provided.

[0068] The local VR device can then render audio elements appropriately, for example, through appropriate binaural processing that reflects the relative position of the audio source for the audio components.

[0069] For the audio side of VR services, in some embodiments, a central server can generate audio data representing an audio scene, and the audio scene can be specifically represented by multiple audio elements that can be rendered by a local client / device.

[0070] Figure 1 The illustration shows an example of a VR system in which a central server 101 communicates with multiple remote clients 103, for example, via a network 105 (e.g., the Internet). The central server 101 can be configured to simultaneously support a potentially large number of remote clients 103.

[0071] In many cases, this approach can provide an improved trade-off between complexity and the resource requirements (communication requirements, etc.) of different devices. For example, viewer poses and corresponding scene data can be transmitted at greater intervals using a local device that locally processes the viewer poses and received scene data to provide a real-time, low-latency experience. This can, for example, substantially reduce the required communication bandwidth while providing a low-latency experience, and simultaneously allow scene data to be centrally stored, generated, and maintained. It can be suitable, for example, for applications where VR experiences are provided to multiple remote devices.

[0072] Figure 2 The illustration shows elements of an audio device that can provide improved audio rendering in many applications and scenarios. Specifically, the audio device can provide improved rendering for many VR applications, and the audio device can be specifically arranged for... Figure 1 The VR client 103 performs audio processing and rendering.

[0073] Figure 2The audio device is arranged to render an audio scene by generating a mixed set of output signals using a first (sub)set and a second (sub)set of output signals, wherein the first (sub)set of output signals is generated for rendering through a set of speakers, and the second (sub)set of output signals is generated for rendering through headphones. Specifically, the first set of audio signals may be a set of surround sound signals for rendering on a surround sound speaker suite. The second set of audio signals may specifically be a binaural stereo signal for rendering on a pair of headphones.

[0074] Figure 2 The audio device can be part of a hybrid audio reproduction system for VR / AR that uses a combination of headphones and speakers to reproduce audio scenes.

[0075] In many embodiments, this approach can provide advantageous operation. For example, in many cases, using a combination of speakers and headphones, rather than either one alone, can provide an AR (or VR / MR) experience that is highly immersive for each individual user without hindering the "social" or "sharing" aspects of the experience. For example, it can allow the rendered audio to be customized to the individual user and to the user's current context. For example, it can allow the location of the audio source to be accurately adjusted to match the user's head movement / rotation. At the same time, it can reduce the complexity required for, for example, binaural processing, because the essential parts of the audio scene can be rendered through less complex audio channel / surround sound processing. It can also, for example, be based on the use of headphones with low attenuation of external sounds, thereby facilitating interaction between users in the same environment / room.

[0076] The following description will focus on an example of a system that uses a combination of a common surround speaker setup (e.g., a 5.1 or 7.1 system) for all local users and individual (open or semi-open) headphones for individual users (where “individual headphones” means: headphones for which the signal has been generated or tuned for the user wearing those headphones) to render an audio scene.

[0077] The device will be described specifically with reference to use cases of the "social" or "sharing" aspects of VR / AR / MR applications that involve multi-user shared experiences. These can be in different locations, but for the sake of example, it is more interesting to consider the same location (e.g., the same room). A specific use case example is several people in the same room sharing the same AR experience that is "projected" within their shared real-world environment. For example, a couple sitting together on a sofa watching an immersive movie virtually projected onto the wall of their living room. They can wear see-through glasses that allow them to see each other and their environment, as well as open headphones that allow for specialized personalized rendering and ambient audio (including audio generated through surround sound settings) to be heard.

[0078] Figure 2 The apparatus specifically includes a receiver 201 arranged to receive data describing a virtual scene. The data may include data providing a visual description of the scene and may include data providing an audio description of the scene. Therefore, both the audio scene description and the virtual scene description can be provided by the received data.

[0079] Receiver 201 is coupled to visual renderer 203, which renders an image corresponding to the viewer's current viewing posture. For example, the data may include spatial 3D image data (e.g., images and depth or model descriptions of a scene), and based on this, visual renderer 203 may generate stereoscopic images (images for the user's left and right eyes), as will be known to those skilled in the art. The images may be presented to the user, for example, via individual left and right eye displays of a VR headset.

[0080] The received data includes audio data describing the scene. Specifically, the audio data includes audio data for a set of audio elements corresponding to audio sources in the scene. Some audio elements can represent localized audio sources in the scene associated with a specific location (which, of course, can change dynamically for moving objects). Typically, audio elements can represent audio generated by specific scene objects in the virtual scene, and therefore can represent audio sources at locations corresponding to the positions of scene objects (e.g., a speaker).

[0081] Other elements can represent more distributed or diffuse audio sources, such as diffused ambient or background noise. As another example, some audio elements can fully or partially represent non-spatial localized components of audio from localized audio sources, such as diffuse reverberation from spatially defined audio sources.

[0082] Audio elements can be encoded audio data, such as encoded audio signals. Audio elements can be of different types, including different types of signals and components, and in fact, in many embodiments, the first receiver 201 can receive audio data that defines different types / formats of audio. For example, audio data can include audio represented by audio channel signals, individual audio objects, higher-order stereo reverberation (HOA), etc.

[0083] For a given audio component to be rendered, the audio can be represented, for example, as encoded audio. The audio data may also include location data indicating the location of the source of the audio component. The location data may, for example, include absolute location data defining the location of the audio source within the scene.

[0084] The device also includes two renderers, 205 and 207.

[0085] The first renderer 205 is configured to render audio elements on a set of speakers. Specifically, the first renderer 205 may generate a first set of audio signals for a set of speakers, wherein the first set of audio signals is, for example, a set of surround sound signals set for surround sound speakers.

[0086] The first renderer 205 can therefore generate a set of audio signals intended for rendering through a specific spatial speaker configuration. The first renderer 205 can generate signals for each speaker in the surround sound configuration, and thus for rendering from a specific location corresponding to the speaker position in the configuration.

[0087] The first renderer 205 can be configured to generate audio signals such that given audio elements are rendered, resulting in a combined effect that gives the impression that the audio elements are rendered from a desired location. Typically, for at least some audio elements, the received data may include specific location indications, and the first renderer 205 can render the audio elements such that they are perceived as originating from the indicated location. Other audio elements may, for example, be distributed and diffuse, and can be rendered in this manner.

[0088] It should be appreciated that many algorithms and methods for rendering spatial audio using loudspeakers, and specifically in surround sound systems, will be known to those skilled in the art, and any suitable method may be used without departing from the present invention.

[0089] For example, the first renderer 205 can generate audio signals for five speakers in a surround sound configuration having a center speaker, a left front speaker, a right front speaker, a left surround speaker, and a right surround speaker. The first renderer 205 can generate a set of audio signals, including audio signals for each speaker. The signals can then be amplified to generate drive signals for individual speakers.

[0090] In some embodiments, audio elements being rendered using speakers can be received, for example, in a stereo downmixed manner, and the first renderer 205 can perform upmixing to generate surround signals that, in some cases, can be rendered directly. This approach can be useful, for example, for audio elements representing diffuse sound that is not directly related to the user's posture. For instance, audio elements representing general diffuse ambient audio can be provided in a stereo downmixed manner, which can be directly upmixed to provide appropriate surround sound audio channels. Each of the resulting upmixed signals can be combined with signals generated from other audio elements for the corresponding speakers to generate a set of output signals.

[0091] Some audio elements rendered via speaker settings can be provided, for example, in the form of audio objects. Such audio objects can be represented by audio data describing a specific audio signal and associated location data describing the location of the audio source. Based on the location data and the speaker locations (whether actual or nominal locations of the surround sound speaker settings), the first renderer 205 can determine coefficients for mapping the audio signal to different surround sound channels using matrices or vectors.

[0092] In some embodiments, the first renderer 205 may also be configured to adapt the generated audio signal based on acoustic environment data. For example, if data indicating that the current environment is a highly reflective environment (e.g., a bathroom with high reflectivity or a similar acoustic environment) is provided, then the first renderer 205 may generate and apply a filter having an impulse response corresponding to the room transfer function for the environment (first reflection, etc.). In some embodiments, the filter may be applied to each of the generated audio signals for individual surround channels, or in some embodiments, it may be applied to audio elements before upmixing to different audio channels.

[0093] In some embodiments, the first renderer 205 may alternatively or additionally be arranged to add reverb, specifically based on ambient data received using the audio elements. For example, the first renderer 205 may apply a synthetic reverb, such as a Jot reverb, wherein parameters are set according to acoustic ambient data (e.g., wherein the reverb persists as indicated by the data). The reverb may typically be applied to the audio elements prior to any upmixing or mapping to the surround channel. The second renderer 207 is arranged to generate a second set of audio signals for headphones. The second set of audio signals may specifically be a binaural stereo signal.

[0094] In many embodiments, audio rendering via the second renderer 207 is a binaural rendering process that uses a suitable binaural transfer function to provide the desired spatial effect for a user wearing headphones. For example, the second renderer 207 may be arranged to use binaural processing to generate audio components perceived as arriving from a specific location.

[0095] Binaural processing is known to provide a spatial experience by virtually locating a sound source using individual signals tailored to the listener's ears. With appropriate binaural rendering, it is possible to calculate the signal required at the eardrum for the listener to perceive sound as coming from any desired direction, and to render the signal to provide the desired effect. These signals are then regenerated at the eardrum using either headphones or crosstalk cancellation methods (suitable for rendering over closely spaced speakers). Binaural rendering can be considered a method for generating signals tailored to the listener's ears to trick the human auditory system into believing that sound is coming from a desired location.

[0096] Binaural rendering is based on the binaural transfer function, which varies from person to person due to the acoustic properties of the head, ears, and reflective surfaces such as the shoulders. For example, binaural filters can be used to generate binaural recordings simulating multiple sources at various locations. This can be achieved by convolving each sound source with, for example, a pair of head-related impulse responses (HRIRs) corresponding to the location of the sound source.

[0097] The well-known method for determining the binaural transfer function is binaural recording. It is a method of recording sound using a specialized microphone setup designed for playback with headphones. Recording is performed by placing microphones in the subject's ear canals or using a mannequin head with embedded microphones, including a half-body image of the ears (outer ears). The use of such a mannequin head, including the ears, provides a spatial impression very similar to that of the person listening to the recording being present during the recording process.

[0098] By measuring the response of a sound source, for example, from a specific location in 2D or 3D space to a microphone placed in or near the listener's ear, an appropriate binaural filter can be determined. Based on such measurements, a binaural filter reflecting the acoustic transfer function to the user's ear can be generated. Binaural filters can be used to generate binaural recordings simulating multiple sources at various locations. This can be achieved, for example, by convolving each sound source with a measured impulse response pair for the desired location of the sound source. To create the illusion that the sound sources are moving around the listener, a large number of binaural filters with sufficient spatial resolution, such as 10 degrees, are typically required.

[0099] Head-related binaural transfer functions can be expressed, for example, as head-related impulse response (HRIR), or equivalently, head-related transfer function (HRTF), or binaural interaural impulse response (BRIR), or binaural interaural transfer function (BRTF). The transfer function (e.g., estimated or assumed) from a given location to the listener's ear (or eardrum) can be given, for example, in the frequency domain, in which case it is typically referred to as HRTF or BRTF; or given in the time domain, in which case it is typically referred to as HRIR or BRIR. In some scenarios, the head-related binaural transfer function is determined to include aspects or properties of the acoustic environment and, specifically, the room in which measurements are taken, while in other paradigms, only user characteristics are considered. Examples of the first type of function are BRIR and BRTF.

[0100] The second renderer 207 may accordingly include a storage device having binaural transfer functions for a typically large number of different locations, wherein each binaural transfer function provides information on how an audio signal should be processed / filtered to be perceived as originating from that location. Applying binaural processing individually to multiple audio signals / sources and combining the results can be used to generate an audio scene using multiple audio sources located at appropriate positions in the sound field.

[0101] The second renderer 207 can select and retrieve a stored binaural transfer function (or, in some cases, generate by interpolation between multiple neighboring binaural transfer functions) for a given audio element to be perceived as originating from a given position relative to the user's head. It can then apply the selected binaural transfer function to the audio signal of the audio element, thereby for the left ear and for the right ear.

[0102] The generated stereo output signals, in the form of left and right ear signals, are suitable for headphone rendering and can be amplified to generate drive signals fed to the user's helmet. The user will then perceive audio elements originating from the desired location.

[0103] It should be appreciated that in some embodiments, audio elements may also be processed to, for example, add acoustic environmental effects. For example, as described with respect to the first renderer 205, audio elements may be processed to add reverberation or, for example, decorrelation / diffusion. In many embodiments, such processing may be performed on the generated binaural signal rather than directly on the audio element signal.

[0104] Therefore, the second renderer 207 can be arranged to generate audio signals such that a given audio element is rendered so that a user wearing headphones perceives the audio element as being received from a desired location. Typically, the second renderer 207 can render audio elements such that they are perceived as originating from a location indicated in location data included along with the audio data. Other audio elements may, for example, be distributed and diffuse, and can be rendered in this manner.

[0105] The device may be part of client 103, receiving data from central server 101 including audio data describing an audio scene. In many applications, central server 101 may provide multiple audio elements in the form of audio objects, audio channels, audio components, HOAs, audio signals, etc. In many cases, some audio elements may correspond to a single audio source with a specific location. Other audio elements may correspond to more diffuse, less defined, and more distributed audio sources.

[0106] It should be appreciated that many algorithms and methods for rendering spatial audio using headphones, and specifically for binaural rendering, will be known to those skilled in the art, and any suitable method may be used without departing from the present invention.

[0107] Figure 2 The device can then be used in client 103 to process the received audio data to render the desired audio scene. Specifically, it can process each audio element based on the desired location data (where appropriate) and then combine the results.

[0108] Figure 2 The device accordingly uses two different rendering techniques to generate the audio representing the scene. Different rendering techniques can have different properties, and Figure 2 The apparatus includes a selector 209, which is configured to select which audio elements are rendered by a first renderer 205 and which audio elements are rendered by a second renderer 207. Specifically, for a given first audio element, the selector 209 can select which renderers 205 and 207 should be used for rendering. The selector 209 can accordingly receive the first audio element and feed it to the first renderer 205 or the second renderer 207 based on the selection.

[0109] In the system, in addition to audio data (and possibly visual data), receiver 201 is configured to receive metadata including audio rendering property indicators for at least one of the audio elements, and often for most or virtually all of the audio elements. Specifically, for the first audio element, at least a first audio rendering property indicator is included.

[0110] Selector 209 is configured to select which renderer to use based on the received metadata and audio rendering property indicators. Specifically, selector 209 is configured to consider the first audio rendering property indicator and determine whether the first audio element should be rendered by the first renderer 205 or the second renderer 207, i.e., whether it is rendered using speakers or headphones.

[0111] As a low-complexity example, for each audio element, the data may include encoded audio data and metadata including a location indicator (typically corresponding to the location of the audio source of the audio element) and an audio rendering property indicator for the audio element. In a particular example, the audio rendering property indicator may simply be a binary indicator of whether the audio element should be rendered by the first renderer 205 or the second renderer 207. The selector 209 can then evaluate the binary indicator and select the renderer 205, 207 indicated. The renderers 205, 207 can then generate appropriate output signals for the speaker and headphones, respectively, such that (one or more) audio elements are perceived as arriving from a location indicated by the location indicator. Contributions from each audio element indicated as being rendered using the first renderer 205 can then be combined to generate a first set of audio signals for the speaker, and contributions from each audio element indicated as being rendered using the second renderer 207 can then be combined to generate a second set of audio signals for the headphones.

[0112] In this way, Figure 2 The audio devices can render audio scenes on a hybrid audio rendering system that includes speakers and headphones. Furthermore, the distribution of audio elements on the headphones and speakers can be remotely controlled / guided. For example, the provider of the VR experience can also control and decide how audio elements should be rendered. Since the provider can often have additional information about the specific properties of the audio source for each audio element, this allows for selection of how each controlled audio element should be rendered based on additional information and knowledge that may not be available at the client's location. In many cases, this method can provide improved rendering, and in many cases, an improved user experience. This method can, for example, provide accurate and natural rendering of audio scenes, while allowing people in the same room to converse more naturally with each other.

[0113] Therefore, in many embodiments, an audio rendering property indicator can provide guidance to the client and audio device on how received audio data should be rendered. The audio rendering property indicator can indicate the guided rendering property for rendering a first audio element. In many embodiments, the guided rendering property can be a preferred, recommended, or nominal rendering property recommended for use by the local renderer. Therefore, the guided rendering property can be control data that can be used by the client to set rendering parameters.

[0114] In some embodiments, the guided rendering property may be intended as a mandatory rendering property that must be used when rendering audio elements; however, in other embodiments, the guided rendering property may be a suggested property that can be used by the client or not. Therefore, in many embodiments, the audio device may choose whether to tune its rendering to match the guided rendering property, or may choose to use a different value. However, this method provides a way to allow the audio device to tune its operation under the guidance of a remote server / provider. In many embodiments, this can achieve improved performance because the remote server / provider may have additional information. It may also, for example, allow centralized manual optimization or analysis to potentially improve rendering while still allowing the client freedom and flexibility during rendering.

[0115] In the specific example mentioned above, the audio rendering property indicator indicates whether the first audio item is intended for rendering on a speaker or on headphones. For the first audio element, selector 209 can be arranged to select first renderer 205 for rendering if the first rendering indicator for the first audio element indicates that the first audio element is intended for rendering on a speaker, and to select second renderer 207 for rendering the first audio element if the first rendering indicator indicates that the first audio element is intended for rendering on headphones. Selector 209 can then provide it to the selected renderers 205, 207 for rendering.

[0116] Therefore, in many embodiments, the audio rendering property indicator indicates the property to be applied to the rendering of the first audio element, and specifically the rendering indicator for the audio element can indicate whether the audio element is intended to be rendered by a speaker or by headphones.

[0117] In some embodiments, when using a hybrid reproduction system, the metadata in the content stream can explicitly signal whether an audio element should be rendered on speakers or headphones. This can be an explicit artistic choice made by the content producer and thus can provide improved control / guidance for rendering.

[0118] exist Figure 2 In the apparatus, audio rendering (and visual rendering) can be based on the viewer's posture. Specifically, the apparatus includes a listener posture receiver 211, which is arranged to receive a listener posture indicating the listener's posture. The listener posture can be specifically represented, for example, by tracking the posture of a VR headset worn by the user / listener. It should be appreciated that any suitable method for generating, estimating, receiving, and providing the listener posture can be used without departing from the invention.

[0119] The listener pose receiver 211 is connected to the visual renderer 203 and is used to generate visual output corresponding to a specific pose. Additionally, the listener pose receiver 211 is coupled to a second renderer 207 and used in the rendering of audio elements for the headphones. Therefore, the second renderer 207 is configured to generate a second set of audio signals in response to the listener's pose.

[0120] The second renderer 207 can specifically perform binaural rendering, such that audio elements are rendered to be perceived as originating from an appropriate position relative to the listener's current orientation and position. For example, for a first audio element, the second renderer 207 can first determine its position in scene space by a position indication received for the first audio element in the data stream. The relative position of the first audio element relative to the user can then be determined by analyzing the current listener's pose and the corresponding pose in scene space. The second renderer 207 can then retrieve the HRTF corresponding to that relative position and filter the first audio signal using the retrieved HRTF to generate a binaural stereo signal component for the first audio element. This component can then be added to corresponding components generated from other audio elements to generate an output binaural stereo signal.

[0121] It should be recognized that many different methods are known for generating headphone signals (and specifically binaural signals) corresponding to audio sources at spatial locations, and any such suitable method or algorithm can be used by the second renderer 207.

[0122] Compared to the second renderer 207, rendering via the first renderer 205 (i.e., rendering for the speaker) is independent of the listener's pose, and therefore... Figure 2 In the example, the first renderer 205 is configured to generate the first set of audio signals independently of the listener's posture.

[0123] The first renderer 205 can specifically consider the positional indication of the audio element to be rendered by the first renderer 205 and map this to the position in the speaker's rendering space. The first renderer 205 can then generate a signal for the speaker to provide spatial awareness of the audio element corresponding to the determined position.

[0124] It should be recognized that many different methods are known for generating speaker signals (and specifically surround sound signals) corresponding to audio sources at spatial locations, and any such suitable method or algorithm can be used by the first renderer 205.

[0125] Therefore, in this example, the headphone signal is continuously generated to reflect the listener's head movement and rotation, thereby providing a continuous and consistent user experience. Simultaneously, the rendering of the speaker remains invariant relative to the listener's head movement and rotation, further providing a consistent approach. This method can provide a way for different rendering methods to deliver a consistent representation of the audio scene relative to a non-stationary listener.

[0126] Previous examples have focused on the case where the device generates a representation of an audio scene for a single user. However, in many embodiments, the device can generate a representation of an audio scene for multiple users, such as two or more users located in the same room.

[0127] In this configuration, the first renderer 205 can be configured to generate a common set of audio signals for multiple users, while the second renderer 207 is configured to generate an individual headphone signal for each user.

[0128] Therefore, for the audio elements selected for rendering by the first renderer 205, a single set of output signals is generated only for all users, such as a single speaker signal for each speaker in the configuration, and these typically do not depend on any user-specific properties. Specifically, the first set of audio signals generated for speaker rendering is generated without considering any listener pose. The same rendering of the audio scene is generated for all users.

[0129] However, for the audio elements rendered by the second renderer 207, different sets of audio signals can be generated for each user. Specifically, binaural stereo signals can be generated for each user. These individual signals can be generated to reflect the properties or specific characteristics of the individual listener, and can be specifically generated to reflect the listener's posture. Therefore, binaural signals reflecting the user's current position and orientation can be generated.

[0130] The device can therefore provide highly efficient support for multi-user scenarios. It can substantially reduce the audio processing required to support multiple listeners. For example, binaural processing is typically relatively complex and resource-intensive, and in many embodiments, the amount of audio signals that need to be generated using binaural processing can be substantially reduced, thereby substantially reducing complexity and computational burden.

[0131] Therefore, in an example where the device supports two users in the same room, the first renderer 205 can be arranged to generate a first set of common audio signals for rendering using speakers, and the second renderer 207 can be arranged to generate a second set of audio signals for headphones for the first listener and a third set of audio signals for headphones for the second listener. The first set of audio signals can be generated independently of the listener postures of the first and second listeners, and the second set of audio signals can be generated in response to the listener posture of the first listener, and the third set of audio signals can be generated in response to the listener posture of the second listener.

[0132] In different embodiments, the audio rendering property indicator provided in the received data stream can represent different data.

[0133] The audio rendering property indicator indicates whether the first part of the first audio element is associated with a position related to the listener's posture or a position unrelated to the listener's posture. The audio rendering property indicator can specifically indicate whether the first audio element is part of the storyline.

[0134] As a particular example, in some embodiments, selector 209 may be arranged to distribute audio elements on first renderer 205 and second renderer 207 based on whether it is “fixed to head orientation” or “non-fixed to head orientation” according to MPEG terminology for audio element audio rendering properties indicators.

[0135] Audio elements indicated by the "fixed to head" audio rendering property indicator are audio elements designed to have a fixed position relative to the user's head. Such audio elements can be rendered using the second renderer 207 and can be rendered independently of the listener's posture. Therefore, the rendering of such audio elements does not take into account changes in the user's head orientation; in other words, such audio elements are audio elements whose relative position does not change when the user turns their head (e.g., non-spatial audio such as ambient noise or music designed to follow the user without changing its relative position).

[0136] An audio element indicated by an audio rendering property indicator of "non-fixed-to-head" is an audio element designed to have a fixed position in a (virtual or real) environment, and therefore its rendering dynamically adapts to changes in the user's head orientation. In many embodiments, this can be more realistic when such an audio element is rendered as a binaural headphone signal adapted based on the current listener posture. For example, the perception of the position of an audio source rendered by surround sound speaker settings can depend on the user's position and orientation, and therefore an audio element rendered by such speaker settings as "non-fixed-to-head" can result in the perception of a moving audio source when the user moves their head.

[0137] Therefore, in some embodiments, "non-head-oriented" elements can be rendered on the user's headphones, where their position is adapted for each individual user based on the user's tracked head orientation. On the other hand, "head-oriented" elements can be rendered on the speaker and are not adapted to the user's head movement.

[0138] The advantage of this implementation is that the "head-oriented" elements, now primarily presented via speakers (rather than headphones), are mainly responsible for the acoustic isolation experienced when all elements are rendered via headphones. The inference here is that "head-oriented" sounds (primarily music and ambient sounds like crowds, wind, rain, thunder, etc.) are often continuous in nature and spatially ubiquitous, resulting in a sonic "blanket" that isolates the user from their physical environment. On the other hand, "non-head-oriented" elements are often more localized and sparser in space and time, and therefore less likely to "mask" the user's physical acoustic environment.

[0139] In some practical implementations, the user perception of "fixed-to-head orientation" sound rendered on speakers can be slightly different from that typically perceived when reproduced on headphones. However, this is usually not a problem because "fixed-to-head orientation" sound rendered by speakers is generally not directional or critical in terms of spatial localization.

[0140] Which audio elements are "non-head-oriented" and which are "head-oriented" can be clearly signaled using metadata in the audio content stream.

[0141] In the context of AR (and VR) audio reproduction, the term "story-driven sound" is also often used to describe whether audio elements should be "fixed to the head orientation." "Story-driven sound" describes elements that should remain in the same virtual position as the user moves their head (meaning their rendered position relative to the user's head must be modified). "Non-story-driven sound" describes elements whose position is not important to this, or even preferably, whose position does not take into account the user's head movement (meaning they will move with the user's head or be "attached" to the user's head).

[0142] In some embodiments, an audio rendering property indicator for an audio element can indicate the audio format of the audio element. Selector 209 can be configured to select whether a first renderer 205 or a second renderer 207 is used to render the audio element based on its audio format. The audio rendering property indicator can, for example, indicate that the audio element is an audio format from the group consisting of: audio object formats; high-order stereo reverb audio formats; and audio channel signal audio formats.

[0143] In some embodiments, selector 209 may be arranged to differentiate between elements to be rendered by headphones or by speakers based on the format of the audio element.

[0144] For example, channel-based or high-order stereo reverb (HOA) elements, often used to transmit background sounds like music and ambient sounds, can be rendered on speakers, while object elements, typically used to transmit the main audio elements of a scene (often representing audio sources with well-defined locations), can be rendered individually on headphones for each user. This also allows users to not only change their head orientation but also interact with individual audio objects (if the content creator wants the objects to be interactive).

[0145] This embodiment can be viewed as an alternative or addition to providing an audio rendering property indicator that directly defines which renderer should be used. For example, without explicitly issuing a signal that an audio element is a "non-fixed-to-head" or "fixed-to-head" element, selector 209 can evaluate the audio format to determine which renderer 205, 207 should be used.

[0146] Methods and different audio rendering property indicators can be combined, such as channels, HOAs, and elements explicitly signaled as "fixed to head orientation" to be rendered on speakers, while objects and "non-fixed to head orientation" elements are rendered on headphones.

[0147] In some embodiments, the audio rendering property indicator may indicate the audio source type for the first audio element. For example, the audio rendering property indicator may indicate whether the audio element is from an audio source type that includes one or more of the following: speech audio; music audio; foreground audio; background audio; narration audio; and narrator audio.

[0148] In some embodiments, the distribution of audio elements on speakers and headphones can be based on indications in the content stream for the source type of the audio element, such as metadata like “speech” or “music” or “foreground” or “background sound.” In this example, the “speech” source should be rendered on the headphones, while the “music” and “background” sources should be rendered on the speakers. A special instance could be speech tagged as “narration” or “narrator,” which is preferably rendered on the speakers (because it is not intended to have a specific location in space but is “ubiquitous”).

[0149] In some embodiments, as previously described, receiver 201 may also receive visual data indicating a virtual scene corresponding to an audio scene. This data may be fed to visual renderer 203 for rendering using, for example, a suitable rendering technique that generates a stereoscopic image corresponding to the current user pose.

[0150] In some embodiments, an audio rendering property indicator for an audio element can indicate whether the first audio element represents an audio source corresponding to a virtual scene object. The virtual scene object can be an object containing visual data, including a visual representation.

[0151] In examples where visual data provides visual data for the viewport, an audio rendering property indicator can indicate whether an audio element is linked to an object within the viewport.

[0152] If the audio rendering property indicator indicates that the object corresponding to the audio element is visible in the scene, selector 209 may decide to render it using headphones; otherwise, it may use speakers to render the audio element. In some embodiments, the audio rendering property indicator may directly indicate whether an object is visible. However, in other embodiments, the audio rendering property indicator may provide an indirect indication of whether an audio element corresponds to a visible scene object.

[0153] For example, the audio rendering property indicator may include an indication of a scene object represented by received visual data. Selector 209 can then proceed to evaluate whether the object linked to the audio element is visible to the current listener's pose. If visible, it can proceed to render it using headphones; otherwise, the object can be rendered using speakers.

[0154] In some embodiments, the distribution of audio elements on speakers and headphones may be based on an indication of whether an audio element in the received content stream is linked to a visual element / object in the content stream. If the indication indicates this, the audio element is rendered on the headphones. If the indication does not indicate this, the audio element is rendered on the speakers.

[0155] In the previous example, selector 209 has been configured to select the appropriate renderer 205, 207 based solely on the received data. However, it should be appreciated that in many embodiments, additional considerations and specifically other data may be taken into account.

[0156] In many embodiments, the apparatus may include a user input function capable of receiving user input. In such embodiments, selector 209 may be further arranged to select between a first renderer 205 and a second renderer 207 based on user input. User input may be, for example, a direct indication of preferred rendering, such as an explicit indication that a particular audio element should be rendered via headphones rather than speakers. In other embodiments, user input may be more indirect and may, for example, modify selection criteria or favor one of renderers 205, 207. For example, user input may indicate a desire for more audio elements to be rendered via headphones, and selector 209 may change its decision criteria to achieve this.

[0157] Therefore, in some embodiments, users can directly influence the distribution of elements on speakers and headphones. One example is giving users the possibility to manually specify individual elements for playback on headphones or speakers.

[0158] Another example of user-distributed control is providing the user with two or more modes from which they can choose; for example, "individual experience" and "shared experience" modes. In the case of the user selecting the "shared experience" mode, any of the embodiments described above for determining which audio elements should be rendered on the speakers and headphones respectively can be used in any combination.

[0159] In some embodiments, selector 209 itself may be configured to analyze one or more audio elements and determine, based on that analysis, which renderer 205, 207 to use. For example, if no audio rendering property indicator is received for a given audio element, selector 209 may proceed to analyze one or more audio elements to determine audio properties, such as the number of audio elements in the scene, the number of channels for each audio element, the position of the audio element, the distance of the audio element to one or more listeners (or to each speaker), or the movement of the audio element. Selector 209 may then proceed to decide, based on that audio property or based on a combination of these, which renderer 205, 207 to use.

[0160] In a specific paradigm configuration (hereafter referred to as Configuration X), selector 209 can select a renderer for each audio element to produce the most accurate spatial representation of the audio scene. For example, if an audio element is in a virtual position relatively close to one of the physical speakers, then it can be rendered on that particular speaker. Conversely, if an audio element falls in an area not covered by any speakers, then it can be rendered via headphones. The fact that audio elements have the same orientation as the speakers (from the listener's perspective) for a single listener, and also for multiple listeners (but with the case that they are all aligned with the speakers), can also be used in the same way. However, this is often impractical because users change position over time. In this specific Configuration X, the angular accuracy of the (binaural) headphone renderer 207 can be taken into account by selector 209 in making this decision.

[0161] Therefore, in some embodiments, the selection of appropriate renderers 205, 207 can be additionally based on the analysis of the audio signal. For example, an estimator of the acoustic properties of the audio signal can be used to determine properties such as the distance (or velocity) or reverberation time of the audio object / source (especially in the case of multi-channel signals). Audio signal classifiers can also be used, such as speech / music classifiers, music genre classifiers, or audio event classifiers. Specific types of classifiers can also be used to determine which type of microphone (HOA, lavalier, omnidirectional, XY, etc.) has been used to record a given signal. Analysis of the frequency distribution of the audio signal can also be used to determine which audio system (headphones or speakers) is more suitable for rendering the entire audio element.

[0162] In the previous example, selector 209 was arranged to select either the first renderer 205 or the second renderer 207 on an individual audio element basis. However, it should be appreciated that this is not necessary or required. For example, in some embodiments, selector 209 may be arranged to select which renderers 205, 207 to use for a group of audio elements.

[0163] Furthermore, in some embodiments, selector 209 can be arranged to individually select between renderers 205 and 207 for different parts of a single audio element. For example, for some audio elements, one part can be rendered by the first renderer 205, and another part can be rendered by the second renderer 207.

[0164] It should be understood that audio elements can be divided into different parts in different ways depending on the requirements and preferences of individual embodiments. For example, in some embodiments, audio elements can be received as a combination or set of different parts, and selector 209 can select renderer 207 separately for each part. For example, an audio element can represent a particular audio source by a first component representing an audio source with a well-defined location (e.g., corresponding to direct audio) and a second component representing a more diffuse and distributed sound (e.g., corresponding to reverberant sound). In such a scenario, selector 209 can be arranged to render the first component using headphones and the second component using speakers.

[0165] In other embodiments, selector 209 may be arranged to divide audio elements into different parts for rendering. For example, the received audio elements may correspond to audio signals that can be analyzed so that they can be divided into different parts that can then be rendered separately.

[0166] Specifically, in many embodiments, different portions of an audio element may correspond to different frequency ranges. For example, for a given first portion corresponding to a specific frequency range, selector 209 may be arranged to select which renderer 205, 207 to use. It may proceed to perform operations for different frequency ranges, and thus may result in different renderers 205, 207 being used for the first frequency range and the second frequency range.

[0167] In some embodiments, different audio rendering property indicators can be provided for different parts of an audio element, and for a given part, when deciding how to render it, selector 209 can consider a specific audio rendering property indicator. In other embodiments, an audio rendering property indicator can be provided for the entire audio element, but different decision criteria are used for different parts. For example, for the mid-to-high frequency range, the selection between headphones and speakers is based on the received audio rendering property indicator for the audio element, while for the very low frequency range, the first renderer 205 is used to indicate what signal to render on the speaker independently of the audio rendering property indicator (reflecting that low frequencies tend to provide much less significant spatial cues).

[0168] For example, the signal can be split into low-frequency and high-frequency components using low-pass and high-pass filtering, where, depending on the audio rendering property indicator, the low-frequency component is sent to the speaker and the high-frequency component is sent to the headphones. In some such embodiments, advanced audio source separation can be used (e.g., separating each time-frequency point between renderers).

[0169] The use of filtering that preserves the energy at each time point allows physically blended rendering systems to attenuate potential errors generated by filtering.

[0170] The described method can provide many beneficial effects, including, as previously described, allowing for perceptually accurate spatial rendering of the audio scene, while allowing / facilitating direct interaction between users in the same location.

[0171] In many cases, this approach can reduce complexity and resource usage due to the potentially reduced binaural processing required. Another advantage that can often be implemented is the reduction in energy used by the headphone reproduction system, for example, in terms of amplifier power and / or processing load for embedded renderers, which would be crucial in the case of cordless headphones (e.g., battery-powered headphones).

[0172] Another interesting property of hybrid audio reproduction systems for VR applications is their tendency to provide improved safety. In reality, unlike wearing closed headphones, attendees are not completely cut off from the potential dangers of their surrounding real-world environment. This can be a significant factor in many practical situations.

[0173] Another advantage of hybrid systems (such as those described) is the fact that audio content is rendered on a shared speaker setup that tends to enhance the user's sense of shared experience. This approach tends to provide an improved user experience.

[0174] It will be appreciated that, for clarity, the above description has referenced various functional circuits, units, and processors to describe embodiments of the invention. However, it will be apparent that any suitable functional distribution among different functional circuits, units, or processors can be used without departing from the invention. For example, a function illustrated as being performed by a separate processor or controller may be performed by the same processor. Therefore, references to specific functional units or circuits are to be considered merely as references to suitable devices for providing the described functions and do not indicate a strict logical or physical structure or organization.

[0175] This invention can be implemented in any suitable form, including hardware, software, firmware, or any combination thereof. Optionally, the invention can be implemented, at least in part, as computer software running on one or more data processors and / or digital signal processors. The elements and components of embodiments of the invention can be implemented physically, functionally, and logically in any suitable manner. In practice, functionality can be implemented in a single unit, in multiple units, or as part of other functional units. Thus, the invention can be implemented in a single unit or physically and functionally distributed among different units, circuits, and processors.

[0176] Although the invention has been described in conjunction with some embodiments, it is not intended to limit the invention to the specific forms set forth herein. Rather, the scope of the invention is limited only by the claims. Furthermore, while features may appear to have been described in conjunction with specific embodiments, those skilled in the art will recognize that various features of the described embodiments can be combined according to the invention. In the claims, the term "comprising" does not exclude the presence of other elements or steps.

[0177] Furthermore, although listed separately, multiple devices, elements, circuits, or method steps can be implemented, for example, by a single circuit, unit, or processor. Additionally, although individual features may be included in different claims, these features can be advantageously combined, and inclusion in different claims does not imply that the combination of features is infeasible and / or disadvantageous. Including a feature in one class of claims does not imply a limitation on that class, but rather indicates that the feature is equally applicable to other classes of claims where appropriate. Furthermore, the order of features in a claim does not imply any particular order in which the features must operate, and in particular, the order of steps in a method claim does not imply that the steps must be performed in that order. Rather, the steps can be performed in any suitable order. Additionally, singular references do not exclude plural. Therefore, references to “a,” “an,” “first,” “second,” etc., do not exclude plural. Reference numerals in the claims are provided merely for clarity of example and should not be construed as limiting the scope of the claims in any way.

Claims

1. An audio device, comprising: Receiver circuit, The receiver circuit is configured to receive data. The data describes the audio scene. The data includes audio data and metadata. The audio data refers to multiple audio elements. The plurality of audio elements correspond to the plurality of audio sources in the scene. The plurality of audio elements includes a first audio element. The metadata includes at least a first audio rendering property indicator. The first audio rendering property indicator is specific to the first audio element; A first renderer circuit, wherein the first renderer circuit is arranged to render audio elements by generating a first plurality of audio signals for a plurality of speakers; A second renderer circuit, wherein the second renderer circuit is arranged to render audio elements by generating a second plurality of audio signals for headphones; and Selector circuitry, wherein the selector circuitry is arranged to select between a first renderer circuitry and a second renderer circuitry, such that rendering of at least a first portion of the first audio element responds to the first audio rendering property indicator. The first audio rendering property indicator indicates the audio format of the first audio element.

2. The apparatus according to claim 1, in, The device is configured to generate audio signals for multiple listeners. The first renderer circuit is configured to generate the first plurality of audio signals as a set of common audio signals for the plurality of listeners. The second renderer circuitry is configured to generate the second plurality of audio signals for the headphones of the first listener among the plurality of listeners. The second renderer circuit is configured to generate a third plurality of audio signals for the headphones of the second of the plurality of listeners.

3. The apparatus according to claim 1, in, The first part is the frequency sub-range of the first audio element.

4. The apparatus according to claim 1, in, The selector circuit is configured to select between the first renderer circuit and the second renderer circuit for the first portion of the first audio element. The selector circuit is configured to select between the first renderer circuit and the second renderer circuit for a second portion of the first audio element.

5. The apparatus according to claim 1, in, The audio format is one of a set of audio formats including: audio object format; high-order stereo reverb audio format; and audio channel signal audio format.

6. The apparatus according to claim 5, in, The selector circuit is configured to select the first renderer circuit in response to the audio format when the audio format is the high-order stereo reverb audio format or the audio channel signal format.

7. The apparatus according to claim 5, in, The audio format is the audio object format.

8. The apparatus according to claim 6, in, The audio format is the high-order stereo reverb audio format.

9. The apparatus according to claim 6, in, The audio format is the audio format of the audio channel signal.

10. The apparatus of claim 1, further comprising a user input circuit. in, The user input circuit is configured to receive user input. The selector circuit is arranged to select between the first renderer circuit and the second renderer circuit in response to the user input to render at least the first portion of the first audio element.

11. An audio processing method, comprising: Receive data describing the audio scene. The data includes metadata and audio data for multiple audio elements. The plurality of audio elements correspond to the plurality of audio sources in the scene. The metadata includes at least a first audio rendering property indicator. Wherein, the first audio rendering property indicator is for the first audio element of the plurality of audio sources; Audio elements are rendered by generating a first plurality of audio signals for multiple speakers; Audio elements are rendered by generating a second, multiple audio signal for the headphones; and In response to the first audio rendering property indicator, a selection is made between rendering at least a first portion of the first audio element via the plurality of speakers and rendering at least a first portion of the first audio element via the headphones. The first audio rendering property indicator indicates the audio format of the first audio element.

12. The method of claim 11, further comprising: Generate audio signals for multiple listeners; Generate the first plurality of audio signals as a common plurality of audio signals for the plurality of listeners; Generate a second plurality of audio signals for the headphones of the first listener among the plurality of listeners; and Generate a third plurality of audio signals for the headphones of the second of the plurality of listeners.

13. The method according to claim 11, in, The first part is the frequency sub-range of the first audio element.

14. The method of claim 11, further comprising selecting different rendering for the first portion of the first audio element and for the second portion of the first audio element.

15. The method according to claim 11, in, The audio format is one of a set of audio formats including: audio object format; high-order stereo reverb audio format; and audio channel signal audio format.

16. The method according to claim 15, in, Selecting between rendering at least a first portion of the first audio element via the plurality of speakers and rendering at least a first portion of the first audio element via the headphones includes: selecting to render at least a first portion of the first audio element via the plurality of speakers in response to the audio format being the high-order stereo reverb audio format or the audio channel signal format.

17. The method according to claim 15, in, The audio format is the audio object format.

18. The method according to claim 15, in, The audio format is the high-order stereo reverb audio format.

19. The method according to claim 15, in, The audio format is the audio format of the audio channel signal.

20. A computer program stored on a non-transient medium, wherein, When run on a processor, the computer program performs the method according to claim 11.

Images