Audio device and method for audio processing

Abstract

To provide a method of audio processing and an audio apparatus for a virtual / augmented / mixed reality experience / application.SOLUTION: An audio apparatus comprises: a receiver 201 receiving data which includes audio data for a set of audio elements corresponding to audio sources in an audio scene and further includes metadata including at least an audio rendering property indicator for a first audio element of the set of audio elements; a first renderer 205 which generates a first set of audio signals for a set of loudspeakers; a second renderer 207 which generates a second set of audio signals for a headphone; and a selector 209 to select between the first renderer and the second renderer for rendering of at least a first part of the first audio element in response to the first audio rendering property indicator.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an audio device and method of audio processing, particularly but not exclusively to using the same to support augmented / virtual reality applications. [Background technology]

[0002] The variety and range of experiences based on audiovisual content has increased significantly in recent years, with the continued development and introduction of new services and ways of using and consuming such content. In particular, many spatial and interactive services, applications and experiences have been developed to give users more engaging and immersive experiences.

[0003] Examples of such applications are virtual reality (VR), augmented reality (AR), and mixed reality (MR) applications, which are rapidly becoming mainstream, with several solutions aimed at the consumer market. Several standards are also under development by several standardization bodies. Such standardization activities are actively developing standards for various aspects of VR / AR / MR systems, including, for example, streaming, broadcasting, rendering, etc.

[0004] VR applications tend to provide a user experience corresponding to the user being in a different world / environment / scene, while AR applications (including mixed reality MR) tend to provide a user experience corresponding to the user being in a current environment but with additional information or virtual objects or information added. Thus, VR applications tend to provide fully immersive, synthetically generated worlds / scenes, while AR applications tend to provide partially synthetic worlds / scenes overlaid on a real scene in which the user is physically present. However, the terms are often used interchangeably and have a high degree of overlap. In the following, the term virtual reality / VR is used to refer to both virtual reality and augmented reality.

[0005] As an example, an increasingly popular service is the provision of images and audio in a manner that allows the user to actively and dynamically interact with the system to change the parameters of the rendering, so that this adapts to movements and changes in the user's position and orientation. A highly attractive feature in many applications is the ability to change the effective viewing position and direction of the viewer, for example to enable the viewer to move and "look around" in the scene being presented.

[0006] Such features may in particular enable a virtual reality experience to be provided to the user, which allows the user to move around (relatively) freely in the virtual environment and dynamically change the user's position and where the user is looking. Typically, such virtual reality applications are based on a three-dimensional model of the scene, which is dynamically evaluated to provide a specific requested view. This approach is well known, for example, from gaming applications for computers and consoles, such as in the category of first-person shooters.

[0007] It is also desirable, particularly for virtual reality applications, that the images being presented are three-dimensional images. Indeed, to optimize the immersion of the viewer, it is generally preferred that the user experience the presented scene as a three-dimensional scene. Indeed, the virtual reality experience should preferably allow the user to select his or her own position, camera viewpoint, and moment in time relative to the virtual world.

[0008] Generally, virtual reality applications are inherently limited in being based on a predefined model of the scene, and generally an artificial model of the virtual world. In some applications, the virtual reality experience is provided based on a capture of the real world. Often, such approaches tend to be based on a virtual model of the real world, which is constructed from the capture of the real world. The virtual reality experience is then generated by evaluating this model.

[0009] Many current approaches tend to be sub-optimal, often having high computational or communication resource requirements and / or providing a sub-optimal user experience, for example with reduced quality or limited freedom.

[0010] As an example of an application, virtual reality glasses have entered the market that allow a viewer to experience captured 360° (panoramic) or 180° videos. These 360° videos are often pre-captured using a camera rig, with individual images stitched together into a single spherical mapping. Common stereo formats for 180° or 360° videos are top / bottom and left / right. Similar to non-panoramic stereo videos, the left and right eye pictures are compressed, for example, as part of a single H.264 video stream.

[0011] In addition to visual rendering, most VR / AR applications also provide a corresponding audio experience. In many applications, the audio preferably provides a spatial audio experience, where the audio sources are perceived to come from positions corresponding to the positions of the corresponding objects in the visual scene. Thus, the audio and video scenes are preferably perceived in unison, providing both a complete spatial experience.

[0012] For audio, the focus has so far mostly been on headphone playback using binaural audio rendering techniques. In many scenarios, headphone playback allows for a highly immersive and personalized experience for the user. Using head tracking, rendering can be done in response to the user's head movements, which greatly increases the sense of immersion.

[0013] Recently, both in the market and in standards discussions, use cases have started to be proposed that involve the "social" or "shared" aspect of VR (and AR), i.e. the possibility of sharing an experience with other people. These can be people in different locations, but also in the same location (or a combination of both). For example, several people in the same room share the same VR experience, where the projections (audio and video) of each participant are present in the VR content / scene. For example, in a game in which several people participate, each player has a different location in the game-scene, and as a consequence, a different projection of the audio-scene and the video-scene.

[0014] As a specific example, MPEG is attempting to standardize bitstreams and decoders for realistic immersive AR / VR experiences with six degrees of freedom. Social VR is a key feature, allowing users to interact in a shared environment (gaming, conference calls, online shopping, etc.). The concept of social VR also facilitates making the VR experience a more social activity for users who are physically in the same location, but where, for example, a head-mounted display or other VR headset provides a perceptual break from the physical surroundings.

[0015] A drawback of headphone playback in such "social" or "shared" AR (or VR) use cases is that users in the same location (e.g., a room) are at least partially acoustically isolated from each other when each user wears individual headphones, which reduces the "social" part of the experience (e.g., it becomes difficult or awkward for people standing next to each other to have a natural conversation). Summary of the Invention [Problem to be solved by the invention]

[0016] This is addressed by using loudspeakers instead of headphones for audio reproduction. However, this has the drawback that the audio reproduction cannot be adapted and customized as freely to the individual user. For example, it makes it difficult to dynamically adapt the audio reproduction to head movements, and in particular to the changes in head orientation of each individual user. Such effects are crucial for an immersive experience, and therefore loudspeakers tend to be suboptimal for generating an optimized user experience.

[0017] Therefore, improved techniques for audio processing, especially for virtual / augmented / mixed reality experiences / applications, would be advantageous, particularly techniques that allow for improved operation, increased flexibility, reduced complexity, ease of implementation, improved audio experience, a more consistent perception of audio and visual scenes, improved customization, improved personalization, improved virtual reality experience, and / or improved performance and / or operation.

[0018] SUMMARY OF THE DISCLOSURE Accordingly, the present invention seeks to preferably mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination. [Means for solving the problem]

[0019] According to an aspect of the invention, there is provided an audio device comprising: a receiver for receiving data representing an audio scene, the data including audio data for a set of audio elements corresponding to audio sources in the audio scene and metadata including at least a first audio rendering characteristic indicator for a first audio element of the set of audio elements; a first renderer for rendering the audio elements by generating a first set of audio signals for a set of loudspeakers and a second renderer for rendering the audio elements by generating a second set of audio signals for headphones; and a selector for selecting between the first renderer and the second renderer for rendering at least a first portion of the first audio element in response to the first audio rendering characteristic indicator, wherein the audio rendering characteristic indicator indicates whether the first portion of the first audio element is associated with a listener pose dependent position or a listener pose non-dependent position.

[0020] The present approach, in many embodiments, provides an improved user experience, particularly for many virtual reality applications (including augmented reality and mixed reality), particularly including social or shared experiences. The present approach uses hybrid rendering to provide improved performance. For example, in many embodiments, the present approach allows users in the same room to more easily talk directly, while still providing targeted and personalized rendering of the audio scene.

[0021] The audio rendering characteristic indicator indicates whether the first audio element represents an audio source with head orientation-fixed spatial characteristics (corresponding to listener pose-dependent and listener pose-independent positions, respectively) or an audio source with head orientation-independent spatial characteristics. The approach reduces complexity and resource requirements.

[0022] In some embodiments, the audio device comprises a first driver for driving a set of loudspeakers from a first set of audio signals and a second driver for driving headphones from a second set of audio signals, the first set of audio signals being in particular a set of surround signals and the second set of audio signals being in particular binaural stereo signals.

[0023] The first audio rendering characteristic indicator indicates a rendering characteristic to be applied to the first audio element or a characteristic of the first audio element.

[0024] According to an optional feature of the invention, the audio apparatus further comprises a listener posture receiver for receiving a listener posture indicative of a posture of a listener, the first renderer being configured to generate the first set of audio signals independent of the listener posture and the second renderer being configured to generate the second set of audio signals in response to the listener posture.

[0025] The audio device provides a highly advantageous and flexible user experience, for example allowing a close alignment between listener movement and the perceived audio scene. Pose refers to position and / or orientation data, also called configuration. Listener pose is a position indication for the listener, a direction indication for the listener, or a combined position and orientation indication for the listener. Pose / configuration is represented by one or more values ​​providing a position and / or direction indication.

[0026] According to an optional feature of the invention, the audio device is configured to generate audio signals for multiple listeners, and a first renderer is configured to generate a first set of audio signals as a common set of audio signals for the multiple listeners, and a second renderer is configured to generate a second set of audio signals for headphones for a first listener of the multiple listeners and to generate a third set of audio signals for headphones for a second listener of the multiple listeners.

[0027] The audio device provides advantageous support for multiple users, and in many applications, improved support is achieved with low complexity and resource usage, while often providing an engaging user experience with a consistent and natural perception of the audio stage.

[0028] A second set of audio signals is generated in response to a first listener pose for the first listener and a third set of audio signals is generated in response to a second listener pose for the second listener. The first set of audio signals is generated independent of listener pose.

[0029] According to an optional feature of the invention, the first portion is a frequency sub-range of the first audio element.

[0030] This provides improved performance in many embodiments.

[0031] According to an optional feature of the invention, the selector is configured to select different renderers from among the first renderer and the second renderer for a first portion of the first audio element and a second portion of the first audio element.

[0032] This provides an improved user experience in many embodiments.The selector is in particular configured to select different renderers for different frequency ranges of the first audio element.

[0033] This provides an efficient approach in many applications. The audio rendering characteristic indicator indicates whether the first audio element is diegetic or not.

[0034] According to an optional feature of the invention, the audio rendering characteristics indicator indicates an audio format of the first audio element.

[0035] This provides an improved user experience in many embodiments. The audio rendering characteristic indicator indicates an audio format from a set of audio formats including at least one audio format from the group of an audio object format, a high-order Ambisonics audio format, and an audio channel signal audio format.

[0036] In accordance with an optional feature of the invention, the audio rendering characteristics indicator indicates an audio source type for the first audio element.

[0037] This provides an improved user experience in many embodiments. The audio rendering characteristics indicator indicates an audio source type from a set of audio source types that includes at least one audio source type from the group of voice audio, music audio, foreground audio, background audio, voiceover audio, and narrator audio.

[0038] According to an optional feature of the invention, the audio rendering characteristic indicator indicates guidance rendering characteristics for the rendering of the first audio element.

[0039] This provides, in many embodiments, an improved user experience and / or performance.

[0040] According to an optional feature of the invention, the audio rendering characteristic indicator indicates whether the first portion of the first audio item is intended to be rendered over a loudspeaker or over headphones.

[0041] This provides, in many embodiments, an improved user experience and / or performance.

[0042] According to an optional feature of the invention, the receiver is further configured to receive visual data indicating a virtual scene corresponding to the audio scene, and the audio rendering characteristic indicator indicates whether the first audio element represents an audio source corresponding to the visual scene object.

[0043] This provides, in many embodiments, an improved user experience and / or performance.

[0044] In some embodiments, the audio rendering characteristics indicator indicates whether the first audio element represents an audio source that corresponds to a scene object that is within a viewport determined for the current listener pose.

[0045] According to an optional feature of the invention, the audio device further comprises a user input portion for receiving user input, and the selector is configured to select between the first renderer and the second renderer for rendering at least the first portion of the first audio element in response to the user input.

[0046] This provides an improved user experience in many embodiments.

[0047] According to an optional feature of the invention, the selector is configured to determine audio characteristics of the first audio element and, in response to the audio characteristics, select between the first renderer and the second renderer for rendering at least a first portion of the first audio element.

[0048] This provides, in many embodiments, an improved user experience and / or performance.

[0049] According to an aspect of the invention, there is provided a method of audio processing comprising the steps of receiving data representing an audio scene, the data including audio data for a set of audio elements corresponding to audio sources in the audio scene and metadata including at least a first audio rendering characteristic indicator for a first audio element of the set of audio elements; rendering the audio elements by generating a first set of audio signals for the set of loudspeakers and rendering the audio elements by generating a second set of audio signals for headphones; and selecting between rendering at least a first portion of the first audio element for the set of loudspeakers and rendering at least the first portion of the first audio element for the headphones in response to the first audio rendering characteristic indicator, wherein the audio rendering characteristic indicator indicates whether the first portion of the first audio element is associated with a listener posture dependent or listener posture independent position.

[0050] These and other aspects, features and advantages of the present invention will be apparent from and elucidated with respect to the embodiment(s) described hereinafter.

[0051] Embodiments of the present invention will now be described, by way of example only, with reference to the drawings in which: [Brief description of the drawings]

[0052] [Figure 1] FIG. 1 illustrates an example of a client-server based virtual reality system. [Diagram 2] FIG. 2 illustrates an example of elements of an audio device according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0053] Virtual reality (including augmented and mixed reality) experiences that allow users to move around in virtual or augmented worlds are becoming increasingly popular, and services are being developed to meet such demand. In many such approaches, visual and audio data are dynamically generated to reflect the user's (or an observer's) current pose.

[0054] In the art, the terms configuration and pose are used as general terms for position and / or orientation. A combination of position and orientation / orientation, for example of an object, camera, head, or view, is called a pose or configuration. Thus, a configuration or pose indication includes up to six values / components / degrees of freedom, each value / component generally representing an individual characteristic of the position / location or orientation / orientation of the corresponding object. Of course, in many situations, a configuration or pose is represented by fewer components, for example when one or more components are considered to be fixed or irrelevant (for example, when all objects are considered to be at the same height and have a horizontal orientation, four components provide a complete representation of the object's pose). In the following, the term pose is used to refer to a position and / or orientation represented by values ​​from 1 to 6 (corresponding to the maximum possible degrees of freedom).

[0055] Many VR applications are based on pose having maximum degrees of freedom, i.e., three degrees of freedom each of position and orientation, for a total of six degrees of freedom. Thus, a pose is represented by a set or vector of six values ​​that represent the six degrees of freedom, and thus the pose vector provides a three-dimensional position and / or a three-dimensional orientation indication. However, it will be appreciated that in other embodiments, a pose is represented by fewer values.

[0056] Systems or entities based on providing maximum degrees of freedom for an observer are commonly referred to as having six degrees of freedom (6DoF). Many systems and entities provide only orientation or position, and these are commonly known as having three degrees of freedom (3DoF).

[0057] Typically, virtual reality applications generate three-dimensional output in the form of separate view images for the left and right eyes. These are then typically presented to the user by suitable means, such as individual left and right eye displays in a VR headset. In other embodiments, one or more view images are presented, for example, on an autostereoscopic display, or indeed in some embodiments only a single two-dimensional image is generated (e.g., using a conventional two-dimensional display).

[0058] Similarly, for a given observer / user / listener pose, an audio representation of the scene is provided. The audio scene is generally rendered to provide a spatial experience, where the audio sources are perceived to originate from desired positions. Since the audio sources are static in the scene, a change in user pose results in a change in the relative position of the audio sources with respect to the user's pose. Thus, the spatial perception of the audio sources should change to reflect their new positions with respect to the user. Thus, the audio rendering is adapted depending on the user pose.

[0059] Observer or user pose input is determined in different ways in different applications. In many embodiments, the user's physical movements are tracked directly. For example, a camera looking over the user area detects and tracks the user's head (and even eyes (eye tracking)). In many embodiments, the user wears a VR headset that can be tracked by external and / or internal means. For example, the headset comprises an accelerometer and a gyroscope that provide information about the movement and rotation of the headset, and thus the head. In some examples, the VR headset transmits a signal or comprises an (e.g. visual) identifier that allows an external sensor to determine the position of the VR headset.

[0060] In some systems, observer pose is provided by manual means, e.g., by a user manually controlling a joystick or similar manual input, e.g., the user manually moves the virtual observer around in the virtual scene by controlling a first analog joystick with one hand, and manually controls the direction in which the virtual observer is looking by manually moving a second analog joystick with the other hand.

[0061] In some applications, a combination of manual and automatic techniques is used to generate the input observer pose: for example, a headset tracks the head orientation and the observer's movement / position in the scene is controlled by the user using a joystick.

[0062] In some systems, the VR application is provided locally to the observer by a standalone device, e.g., that does not use or even have access to remote VR data or processing, e.g., a device such as a game console, that includes a store for storing scene data, an input for receiving / generating observer poses, and a processor for generating corresponding images from the scene data.

[0063] In other systems, the VR application is implemented and performed remotely from the observer. For example, a device local to the user detects / receives movement / pose data, which is transmitted to a remote device, which processes the data to generate an observer pose. The remote device then generates suitable view images for the observer pose based on scene data representing the scene. The view images are then transmitted to a device local to the observer where they are presented. For example, the remote device directly generates a video stream (typically a stereo / 3D video stream) that is directly presented by the local device.

[0064] Similarly, the remote device generates an audio scene that reflects the virtual audio environment, in many embodiments by generating audio elements that correspond to the relative positions of different audio sources in the virtual audio environment, and these are rendered to be perceived at the corresponding positions.

[0065] For example, a remote device generates audio data representing an audio scene and transmits audio components / objects / signals or other audio elements corresponding to different audio sources in the audio scene along with position information indicating their location (which changes dynamically for example for moving objects). The audio elements include elements related to specific locations, but also elements for more distributed or diffuse audio sources. For example, audio elements are provided that represent general (non-localized) background sounds, ambient sounds, diffuse reverberations, etc.

[0066] The local VR device then renders the audio elements appropriately, for example by applying appropriate binaural processing that reflects the relative position of the audio source for the audio component.

[0067] For the audio side of the VR service, in some embodiments, a central server accordingly generates audio data representing an audio scene, and in particular represents this audio scene by several audio elements that can be rendered by the local client / device.

[0068] 1 shows an example of a VR system in which a central server 101 cooperates with several remote clients 103 over a network 105, such as the Internet. The central server 101 is configured to support a potentially large number of remote clients 103 simultaneously.

[0069] Such an approach provides an improved trade-off between, for example, complexity and resource demands, communication requirements, etc. for different devices in many scenarios. For example, the observer pose and corresponding scene data are transmitted at larger intervals, and the local device processes the observer pose and received scene data locally to provide a real-time low-lag experience. This significantly reduces the required communication bandwidth, while providing, for example, a low-latency experience, and allowing scene data to be stored, generated, and maintained centrally. It is suitable for applications, for example, where a VR experience is provided to multiple remote devices.

[0070] 2 illustrates elements of an audio device that provides improved audio rendering in many applications and scenarios. In particular, the audio device provides improved rendering for many VR applications, and is specifically configured to perform audio processing and rendering for the VR client 103 of FIG. 1.

[0071] The audio device of Fig. 2 is configured to render an audio scene by generating a hybrid set of output signals, a first (sub)set of output signals being generated for rendering by a set of loudspeakers and a second (sub)set of output signals being generated for rendering by headphones. The first set of audio signals is in particular a set of surround sound signals for rendering on a set of surround sound speakers. The second set of audio signals is in particular binaural stereo signals for rendering on a pair of headphones.

[0072] The audio device in FIG. 2 is part of a hybrid audio playback system for VR / AR that uses a combination of headphone and loudspeaker playback to present an audio scene.

[0073] Such an approach provides advantageous operation in many embodiments. For example, in many scenarios, using a combination of loudspeaker and headphone playback rather than either of the two provides an AR (or VR / MR) experience that is highly immersive for each individual user while at the same time not interfering with the "social" or "shared" aspects of the experience. For example, it allows the rendered audio to be customized to the individual user and the current context for that user. For example, it allows the position of the audio source to be accurately adapted to match the user's head movement / rotation. At the same time, it reduces the complexity required for, e.g., binaural processing, since a significant portion of the audio scene is rendered by a lower complexity audio channel / surround sound processing. Furthermore, it allows for example to be based on using headphones with low attenuation of external sounds, which facilitates direct interaction between users in the same environment / room, for example.

[0074] The following description focuses on embodiments in which the system renders an audio scene using a combination of a surround loudspeaker setup (e.g. a 5.1 or 7.1 system) that is common to all local users and individual (open or semi-open) headphones for individual users (where "individual headphones" means headphones that render a signal generated or adapted for the user wearing them).

[0075] The device is described in detail with respect to the use case of the "social" or "shared" aspect of VR / AR / MR applications, where several people share an experience. They can be in different locations, but more interesting for this example, they can also be in the same location (e.g., the same room). An example of a particular use case is several people in the same room sharing the same AR experience "projected" in a shared real environment. For example, a couple sitting together on a couch watching an immersive movie virtually projected on the living room wall. They wear see-through glasses that allow them to see each other and their environment, as well as open headphones that allow a dedicated personalized rendering of both, as well as allowing them to hear audio in the environment, including audio generated by a surround sound setup.

[0076] The apparatus of figure 2 in particular comprises a receiver 201 arranged to receive data representing a virtual scene, the data including data providing a visual description of the scene and including data providing an audio description of the scene, so that an audio scene description and a visual scene description are provided by the received data.

[0077] The receiver 201 is coupled to a visual renderer 203, which then renders an image corresponding to the observer's current viewing pose. For example, the data includes spatial 3D image data (e.g., an image and a depth or model description of a scene), from which the visual renderer 203 generates stereo images (images for the left and right eyes of the user), as known to those skilled in the art. The images are presented to the user, for example, via respective left and right eye displays of a VR headset.

[0078] The received data includes audio data representing a scene. The audio data includes, in particular, audio data for a set of audio elements corresponding to audio sources in the scene. Some audio elements represent localized audio sources in the scene that are associated with a particular position in the scene (positions are of course dynamically changing for moving objects). Often, the audio elements represent audio generated by a particular scene object in the virtual scene, and thus represent audio sources at positions corresponding to the position of a scene object (e.g., a person speaking).

[0079] Other elements represent more dispersed or diffuse audio sources, such as diffuse ambient or background noise. As another example, some audio elements fully or partially represent non-spatially localized components of audio from localized audio sources, such as diffuse reverberation from a spatially well-defined audio source.

[0080] An audio element is encoded audio data, such as an encoded audio signal. The audio elements are different types of audio elements, including different types of signals and components, and indeed in many embodiments, the first receiver 201 receives audio data that defines different types / formats of audio. For example, the audio data includes audio represented by audio channel signals, individual audio objects, Higher Order Ambisonics (HOA), etc.

[0081] The audio is represented, for example, as encoded audio for a given audio component to be rendered. The audio data further comprises position data indicating the position of a source of the audio component. The position data comprises, for example, absolute position data defining the position of the audio source in a scene.

[0082] The audio device further comprises two renderers 205 , 207 .

[0083] The first renderer 205 is configured to render audio elements on a set of loudspeakers. In particular, the first renderer 205 generates a first set of audio signals for the set of loudspeakers, where the first set of audio signals is, for example, a set of surround sound signals for a surround sound loudspeaker setup.

[0084] Thus, the first renderer 205 generates a set of audio signals that are intended to be rendered by a particular spatial loudspeaker configuration. The first renderer 205 generates signals for each loudspeaker in the surround sound configuration and therefore for rendering from specific locations corresponding to speaker positions in the surround sound configuration.

[0085] The first renderer 205 is configured to generate an audio signal such that a given audio element is rendered such that the composite effect leads to the impression that the audio element is being rendered from a desired position. Typically, the received data includes, for at least some audio elements, a specific position indication, and the first renderer 205 renders the audio element such that it is perceived to originate from the indicated position. Other audio elements are, for example, dispersed and diffused and rendered as such.

[0086] It will be appreciated that many algorithms and techniques for rendering spatial audio using loudspeakers, particularly in surround sound systems, are known to those skilled in the art and any suitable technique may be used without detracting from the invention.

[0087] For example, the first renderer 205 generates audio signals for five loudspeakers in a surround sound configuration having a center speaker, a left front speaker, a right front speaker, a left surround speaker, and a right surround speaker. The first renderer 205 generates a set of audio signals including an audio signal for each loudspeaker. The signals are then amplified to generate drive signals for the individual loudspeakers.

[0088] In some embodiments, audio elements being rendered using loudspeakers are received, for example, as a stereo downmix, and the first renderer 205 performs upmixing to generate surround signals, which in some cases are rendered directly. Such an approach is useful, for example, for audio elements representing diffuse sounds that are not directly related to the user's posture. For example, audio elements representing general diffuse ambient audio are provided as a stereo downmix, which is directly upmixed to provide the appropriate surround sound audio channels. Each of the resulting upmixed signals is combined with signals for the corresponding speakers generated from the other audio elements to generate a set of output signals.

[0089] Some audio elements to be rendered via the loudspeaker setup are provided, for example, in the form of audio objects. Such audio objects are represented by audio data representing a particular audio source and associated position data representing the position of the audio source. Based on the position data and the loudspeaker positions (whether actual positions or nominal positions for the surround sound speaker setup), the first renderer 205 determines coefficients for a matrix or vector that maps the audio signals to the different surround sound channels.

[0090] In some embodiments, the first renderer 205 is further configured to adapt the generated audio signals based on acoustic environment data. For example, if data is provided indicating that the current environment is a highly reflective environment (e.g., a bathroom or similar acoustic environment with a high degree of reflection), the first renderer 205 generates and applies a filter having an impulse response corresponding to a room transfer function for the environment (e.g., the first reflection). The filter is applied in some embodiments to each of the generated audio signals for the individual surround channels, or in some embodiments, to the audio elements before upmixing into different audio channels.

[0091] In some embodiments, the first renderer 205 is alternatively or additionally configured to add reverberation, in particular based on the environment data received together with the audio elements. For example, the first renderer 205 applies a synthesis reverberator, such as a Jot reverberator, with parameters set according to the acoustic environment data (e.g. with reverberation sustain indicated by the data). The reverberator is typically applied to the audio elements prior to upmixing or mapping to the surround channels. The second renderer 207 is configured to generate a second set of audio signals for headphones. The second set of audio signals is in particular a binaural stereo signal.

[0092] In many embodiments, the audio rendering by the second renderer 207 is a binaural rendering process that uses suitable binaural transfer functions to provide the desired spatial effect for a user wearing headphones. For example, the second renderer 207 is configured to use binaural processing to generate audio components that should be perceived to come from a particular position.

[0093] It is known that binaural processing is used to provide a spatial experience by virtual positioning of sound sources using individual signals for the listener's ears. In a suitable binaural rendering process, the signals required at the eardrum for the listener to perceive sounds from any desired direction can be calculated, and the signals can be rendered to provide the desired effect. These signals are then recreated at the eardrum using either headphones or crosstalk cancellation methods (suitable for rendering on closely spaced speakers). Binaural rendering can be considered as a technique for generating signals for the listener's ears, thereby tricking the human auditory system into thinking that sounds are coming from a desired position.

[0094] Binaural rendering is based on binaural transfer functions, which vary from person to person due to the acoustic properties of the head, ears, and reflecting surfaces such as shoulders. For example, binaural filters can be used to create binaural recordings that simulate multiple sources at various locations. This can be achieved, for example, by convolving each sound source with a pair of Head-Based Impulse Responses (HRIRs) that correspond to the sound source's location.

[0095] A well-known method for determining the binaural transfer function is binaural recording, which is a method of recording sounds using a dedicated microphone arrangement and which are then subject to replay using headphones. The recording is made either by placing microphones in the subject's ear canals or by using a dummy head with built-in microphones, a bust including the pinna (outer ear). The use of such a dummy head including the pinna provides an exact spatial impression as if the person listening to the recording was present during the recording.

[0096] For example, an appropriate binaural filter can be determined by measuring the response from a sound source at a particular location in 2D or 3D space to a microphone placed in or near a human ear. Based on such measurements, a binaural filter can be generated that reflects the acoustic transfer function to the user's ear. The binaural filters can be used to create binaural recordings that simulate multiple sources at various locations. This can be achieved, for example, by convolving each sound source with a pair of measured impulse responses for the desired position of the sound source. A large number of binaural filters with sufficient spatial resolution, for example 10 degrees, are generally required to create the illusion that the sound source is moving around the listener.

[0097] The head-based binaural transfer function is expressed, for example, as a head-based impulse response (HRIR), or equivalently, as a head-based related transfer function (HRTF) or a binaural room impulse response (BRIR) or a binaural room transfer function (BRTF). The (e.g., estimated or assumed) transfer function from a given position to the listener's ear (or eardrum) is given, for example, in the frequency domain, in which case it is generally called HRTF or BRTF, or in the time domain, in which case it is generally called HRIR or BRIR. In some scenarios, the head-based binaural transfer function is determined to include aspects or characteristics of the acoustic environment, in particular the room in which the measurement is performed, while in other examples, only user characteristics are taken into account. Examples of the first type of function are BRIR and BRTF.

[0098] Thus the second renderer 207 typically comprises a store with binaural transfer functions for a number of different positions, each providing information on how an audio signal should be processed / filtered in order to be perceived as originating from that position. Applying binaural processing to multiple audio signals / sources individually and combining the results is used to generate an audio scene with several audio sources at appropriate positions in the sound stage.

[0099] For a given audio element that should be perceived to originate from a given position relative to the user's head, the second renderer 207 selects and retrieves (or in some cases generates by interpolating between multiple close binaural transfer functions) the stored binaural transfer function that most closely matches the desired position. The second renderer 207 then applies the selected binaural transfer function to the audio signal of the audio element, thereby generating an audio signal for the left ear and an audio signal for the right ear.

[0100] The generated output stereo signals in the form of left-ear and right-ear signals are then suitable for headphone rendering and are amplified to generate drive signals that are fed to the user's headset, so that the user will then perceive the audio elements as originating from the desired positions.

[0101] It will be appreciated that the audio elements may also be processed in some embodiments, for example to add acoustic ambience effects. For example, as described for the first renderer 205, the audio elements may be processed to add reverberation or, for example, decorrelation / diffusion. In many embodiments, this processing is performed on the generated binaural signals, rather than on the direct audio element signals.

[0102] Thus, the second renderer 207 is configured to generate an audio signal such that a given audio element is rendered such that a user wearing headphones would perceive the audio element as being received from a desired position. In general, the second renderer 207 renders an audio element such that it is perceived to originate from a position indicated in the position data included in the audio data. Other audio elements are, for example, possibly dispersed and diffused and rendered as such.

[0103] The device is thus part of a client 103 that receives data including audio data representing an audio scene from a central server 101. In many applications, the central server 101 provides several audio elements in the form of audio objects, audio channels, audio components, HOAs, audio signals, etc. In many situations, some of the audio elements correspond to a single audio source having a specific location. Other audio elements correspond to more diffuse, less clear and more dispersed audio sources.

[0104] It will be appreciated that many algorithms and techniques for rendering spatial audio using headphones, and in particular binaural rendering, are known to those skilled in the art and any suitable technique may be used without detracting from the invention.

[0105] 2 is then used at the client 103 to process the received audio data to render the desired audio scene. In particular, the device processes each audio element based on the desired position data (when appropriate) and then combines the results.

[0106] Thus, the apparatus of Fig. 2 uses two different rendering techniques for generating audio representing a scene. The different rendering techniques have different characteristics, and the apparatus of Fig. 2 comprises a selector 209 configured to select which audio element is rendered by the first renderer 205 and which audio element is rendered by the second renderer 207. In particular, for a given first audio element, the selector 211 selects which renderer 205, 207 should be used for rendering. Thus, the selector 209 receives the first audio element and supplies it to the first renderer 205 or the second renderer 207 depending on the selection.

[0107] In the system, the receiver 201 is configured to receive, in addition to the audio data (and possibly the visual data), metadata including an audio rendering characteristic indicator for at least one of the audio elements, and often for most or indeed all of the audio elements, in particular at least a first audio rendering characteristic indicator for a first audio element.

[0108] The selector 209 is configured to select which renderer to use depending on the received metadata and the audio rendering characteristic indicator. In particular, the selector 209 is configured to take into account the first audio rendering characteristic indicator and to determine whether the first audio element should be rendered by the first renderer 205 or by the second renderer 207, i.e. whether the first audio element should be rendered using a loudspeaker or using headphones.

[0109] As a low-complexity example, the data includes, for each audio element, encoded audio data as well as metadata including a position indication (typically the position of the audio source corresponding to the audio element) and an audio rendering characteristic indicator for the audio element, where the audio rendering characteristic indicator in the particular example is simply a binary indication of whether the audio element should be rendered by the first renderer 205 or the second renderer 207. The selector 209 then evaluates this binary indication and selects the indicated renderer 205, 207. The renderers 205, 207 then generate appropriate output signals for the loudspeakers and headphones, respectively, such that the audio element(s) is perceived to come from the position indicated by the position indicator. Then, the instructions are that the audio elements should be rendered using a first renderer 205, the contributions from each audio element are combined to generate a first set of audio signals for the loudspeakers, and then the instructions are that the audio elements should be rendered using a second renderer 207, the contributions from each audio element are combined to generate a second set of audio signals for the headphones.

[0110] In this way, the audio device of FIG. 2 renders the audio scene on a hybrid audio rendering system that includes both loudspeakers and headphones. Furthermore, the distribution of audio elements across headphones and loudspeakers can be controlled / guided remotely. For example, the provider of the VR experience also controls and determines how the audio elements should be rendered. Since the provider generally has additional information of the specific nature of the audio source for each audio element, this allows the selection of how to render each audio element to be controlled based on additional information and knowledge that is not available at the client. The present technique provides improved rendering in many situations and provides an improved user experience in many scenarios. The present technique provides accurate and natural rendering of the audio scene while allowing, for example, people in the same room to talk to each other more naturally.

[0111] Thus, in many embodiments, the audio rendering characteristic indicator provides guidance to the client and the audio device as to how the received audio data should be rendered. The audio rendering characteristic indicator indicates guidance rendering characteristics for the rendering of the first audio element. The guidance rendering characteristics, in many embodiments, are preferred, suggested, or nominal rendering characteristics that are recommended to be used by the local renderer. Thus, the guidance rendering characteristics are control data that can be used by the client to set the rendering parameters of the rendering.

[0112] In some embodiments, the guidance rendering properties are intended as mandatory rendering properties that must be used when rendering an audio element, while in other embodiments, the guidance rendering properties are suggested properties that may or may not be used by the client. Thus, in many embodiments, the audio device chooses whether to adapt its rendering to match the guidance rendering properties or to adopt different values. However, the present approach provides a technique that allows the audio device to adapt its behavior under the guidance of a remote server / provider. This achieves improved performance in many embodiments because the remote server / provider has additional information. It also allows the client to retain freedom and flexibility in the rendering, while still allowing, for example, centralized manual optimization or analysis to potentially improve the rendering.

[0113] In the particular example described above, the audio rendering characteristic indicator indicates whether the first audio item is intended for rendering on a loudspeaker or whether the first audio item is intended for rendering on headphones. The selector 209 is configured to select, for the first audio element, the first renderer 205 for rendering if the first rendering indicator for the first audio element indicates that the first audio element is intended for rendering by a loudspeaker, and to select the second renderer 207 for rendering the first audio element if the first rendering indicator indicates that the first audio element is intended for rendering by headphones. The selector 209 then provides the first audio element to the selected renderer 205, 207 for rendering.

[0114] Thus, in many embodiments, the audio rendering characteristic indicator indicates the rendering characteristics to be applied to the first audio element, and in particular the rendering indicator for the audio element indicates whether the audio element is intended to be rendered through loudspeakers or headphones.

[0115] In some embodiments, accordingly, when a hybrid playback system is used, whether an audio element should be rendered over loudspeakers or headphones is explicitly signaled by metadata in the content stream. This is an explicit artistic choice made by the content producer, thus providing improved control / guidance for the rendering.

[0116] In the device of Fig. 2, audio rendering (as well as visual rendering) depends on the observer pose. In particular, the device comprises a listener pose receiver 211 configured to receive a listener pose indicative of the listener's pose. The listener pose is represented in particular by a headset pose, for example determined by tracking of a VR headset worn by a user / listener. It will be understood that any suitable method for generating, estimating, receiving and providing the listener pose may be used without detracting from the invention.

[0117] The listener pose receiver 211 is connected to the visual renderer 203 and is used to generate visual outputs corresponding to particular poses. Additionally, the listener pose receiver 211 is coupled to the second renderer 207 and is used in rendering audio elements for headphones. Thus, the second renderer 207 is configured to generate a second set of audio signals in response to the listener pose.

[0118] The second renderer 207 performs binaural rendering, in particular such that the audio elements are rendered to be perceived as occurring at the appropriate position relative to the listener's current orientation and position. For example, for a first audio element, the second renderer 207 first determines the position in the scene space indicated by the position indication received for the first audio element in the data stream. The relative position of the first audio element with respect to the user is then determined by analyzing the current listener pose and the corresponding pose in the scene space. The second renderer 207 then retrieves the HRTF corresponding to this relative position and filters the first audio signal using the retrieved HRTF to generate binaural stereo signal components for the first audio element. These components are then added to the corresponding components generated from the other audio elements to generate the output binaural stereo signal.

[0119] Many different techniques are known for generating headphone signals (and, in particular, binaural signals) corresponding to audio sources at spatial locations, and it will be appreciated that any suitable such technique or algorithm may be used by the second renderer 207.

[0120] In contrast to the second renderer 207, rendering by the first renderer 205 (i.e., rendering for the loudspeakers) is independent of the listener pose and therefore, in the example of FIG. 2, the first renderer 205 is configured to generate a first set of audio signals independent of the listener pose.

[0121] The first renderer 205 in particular considers a position indication for an audio element to be rendered by the first renderer 205 and maps this to a position in the rendering space of the loudspeakers. The first renderer 205 then generates signals for the loudspeakers to provide a spatial perception of the audio element corresponding to the determined position.

[0122] Many different techniques are known for generating loudspeaker signals (and, in particular, surround sound signals) corresponding to audio sources at spatial locations, and it will be appreciated that any suitable such technique or algorithm may be used by the first renderer 205.

[0123] Thus, in this example, the headphone signal is continuously generated to reflect the movement and rotation of the listener's head, thereby providing a continuous and consistent user experience. At the same time, the rendering using the loudspeakers is invariant to the movement and rotation of the listener's head, again providing a consistent approach. This approach provides a way in which different rendering approaches provide a consistent representation of the audio scene for a non-static listener.

[0124] The previous examples have focused on situations where a device generates a representation of an audio scene for a single user, however, in many embodiments the device generates a representation of an audio scene for multiple users, such as for two or more users located in the same room.

[0125] In such a case, the first renderer 205 is configured to generate a common set of audio signals for multiple users, and the second renderer 207 is configured to generate individual headphone signals for each user.

[0126] Thus, for audio elements selected to be rendered by the first renderer 205, only a single set of output signals is generated for all users, e.g., only a single loudspeaker signal is generated for each loudspeaker in the configuration, and these are generally independent of user-specific characteristics. In particular, the first set of audio signals generated for rendering by the loudspeakers is generated without taking into account the listener pose. The same rendering of the audio scene is generated for all users.

[0127] However, for the audio elements rendered by the second renderer 207, a different set of audio signals is generated for each user. In particular, a binaural stereo signal is generated for each user. These individual signals are generated to reflect characteristics or specific attributes for the individual listeners, in particular to reflect the listener pose of the individual listeners. Thus, binaural signals are generated that reflect the current position and orientation of the users.

[0128] The device therefore provides particularly efficient support of multi-user scenarios: the required audio processing to support multiple listeners is significantly reduced: for example, binaural processing is generally relatively complex and resource intensive, and the number of audio signals that need to be generated using binaural processing is significantly reduced, thereby reducing the complexity and computational burden in many embodiments.

[0129] Thus, in one example where the device supports two users in the same room, the first renderer 205 is configured to generate a common first set of audio signals for rendering using loudspeakers, and the second renderer 207 is configured to generate a second set of audio signals for headphones for the first listener and a third set of audio signals for headphones for the second listener, where the first set of audio signals is generated independent of listener poses of the first and second listeners, the second set of audio signals is generated in response to a listener pose of the first listener, and the third set of audio signals is generated in response to a listener pose of the second listener.

[0130] The audio rendering characteristic indicators provided in the received data stream represent different data in different embodiments.

[0131] The audio rendering characteristic indicator indicates whether a first portion of a first audio element is associated with a listener posture-dependent or listener posture-independent position. The audio rendering characteristic indicator in particular indicates whether the first audio element is diegetic or not.

[0132] As a particular example, in some embodiments, the selector 209 is configured to distribute audio elements across the first renderer 205 and the second renderer 207 based on whether an audio rendering characteristic indicator for an audio element indicates that the audio element is “fixed to head orientation” or “not fixed to head orientation,” according to MPEG terminology.

[0133] Audio elements indicated by the audio rendering characteristic indicator as "head-fixed" are audio elements that are intended to have a fixed location relative to the user's head. Such audio elements are rendered using the second renderer 207 and are rendered independent of the listener pose. Thus, the rendering of such audio elements does not take into account (changes in) the user's head orientation; in other words, such audio elements are audio elements whose relative position does not change when the user turns his or her head (e.g., non-spatial audio such as ambient noise, or, for example, music that is intended to follow the user without changing its relative position).

[0134] Audio elements indicated by the audio rendering characteristic indicator as "not head-fixed" are audio elements that are intended to have a fixed location in the (virtual or real) environment and therefore their rendering is dynamically adapted to (changes in) the user's head orientation. In many embodiments, this is more realistic when such audio elements are rendered as binaural headphone signals that are adapted based on the current listener pose. For example, the perception of the position of an audio source rendered by a surround sound loudspeaker setup depends on the user position and orientation, and thus the rendering of audio elements indicated as "not head-fixed" by such a loudspeaker setup results in audio sources that are perceived to move when the user moves their head.

[0135] Thus, in some embodiments, "non-head-orientation-locked" elements are rendered on the user's headphones and their position is adapted for each individual user according to his or her tracked head orientation, whereas "head-orientation-locked" elements are rendered on the loudspeakers and do not adapt to the user's head movements.

[0136] The advantage of such an embodiment is that the "head-fixed" elements, which currently exist mostly via loudspeakers (and not via headphones), primarily serve as the sound insulation experienced when all elements are rendered via headphones. The reasoning here is that "head-fixed" sounds (mostly music and atmospheric sounds, e.g., crowds, wind, rain, thunder, etc.) are often continuous and essentially spatially omnipresent, resulting in a "blanket" of sound that insulates the user from his or her physical surroundings. "Non-head-fixed" elements, on the other hand, are often more localized and sparse in space and time, and therefore do little to "mask" the user's physical acoustic surroundings.

[0137] In some practical implementations, the user perception of "head-locked" sounds rendered on loudspeakers is somewhat different compared to how they are typically perceived when played on headphones, however this is usually not an issue since "head-locked" sounds rendered by loudspeakers are typically omnidirectional or unimportant with respect to spatial localization.

[0138] Which audio elements are "non-head-locked" and which are "head-locked" are explicitly signaled by metadata in the audio content stream.

[0139] In the context of AR (and VR) audio playback, the term "diegetic" is also typically used to describe whether an audio element should be "fixed to the head orientation" or not. "Diegetic" describes elements that should stay in the same virtual position when the user moves their head (meaning that their rendered position relative to the user's head must be modified). "Non-diegetic" describes elements where this is not important, or even better, where the element's position does not take into account the user's head movement (meaning that the element will move with the user's head or is "attached" to the user's head).

[0140] In some embodiments, an audio rendering characteristic indicator for an audio element indicates an audio format of the audio element. The selector 209 is configured to select whether the first renderer 205 or the second renderer 207 is used to render the audio element based on the audio format of the audio element. The audio rendering characteristic indicator indicates, for example, that the audio element is in an audio format from the group of an audio object format, a high-order Ambisonics audio format, and an audio channel signal audio format.

[0141] In some embodiments, the selector 209 is configured to distinguish between elements that should be rendered by headphones and elements that should be rendered by loudspeakers based on the format of the audio elements.

[0142] For example, channel-based or Higher Order Ambisonics (HOA) elements, often used to transmit background sounds such as music and ambience, are rendered on loudspeakers, while object elements, typically used to transmit the main audio elements of a scene (which often represent audio sources with well-defined positions), are rendered on headphones for each user individually. This also allows users to interact with individual audio objects (if the content creator intended the objects to be interactive), as well as change their head orientation.

[0143] This embodiment is considered as an alternative or addition to providing an audio rendering characteristic indicator that directly defines which renderer should be used. For example, in situations where no explicit signaling is included whether an audio element is a "non-head-locked" / "head-locked" element, the selector 209 evaluates the audio format to determine which renderer 205, 207 should be used.

[0144] The approach is combined with different audio rendering characteristic indicators, e.g., channel elements, HOA elements, and elements explicitly signaled as "fixed to head orientation" are rendered on the loudspeakers, and objects and "non-fixed to head orientation" elements are rendered on the headphones.

[0145] In some embodiments, the audio rendering characteristics indicator indicates an audio source type for the first audio element, e.g., the audio rendering characteristics indicator indicates whether the audio element is of an audio source type from a set that includes, e.g., one or more of voice audio, music audio, foreground audio, background audio, voiceover audio, and narrator audio.

[0146] In some embodiments, the distribution of audio elements across loudspeakers and headphones is based on an indication in the content stream of the source type for the audio element, e.g., metadata such as "voice", "music", "foreground", or "background sound". In this example, a "voice" source might be rendered on headphones, and a "music" source and a "background" source might be rendered on loudspeakers. A special case might be a voice marked as being a "voiceover" or "narrator", which would be best rendered on loudspeakers (as it is not intended to have a specific location in space, but rather is intended to be "omnipresent").

[0147] In some embodiments, the receiver 201 also receives visual data representing a virtual scene corresponding to the audio scene, as previously described. This data is provided to a visual renderer 203 for rendering using a suitable rendering technique, for example generating a stereo image corresponding to the current user pose.

[0148] In some embodiments, the audio rendering characteristic indicator for the audio element indicates whether the first audio element represents an audio source that corresponds to a visual scene object. A visual scene object is an object whose visual data includes a visual representation.

[0149] In one example where visual data provides visual data for a viewport, the audio rendering characteristic indicator indicates whether an audio element is linked to an object in the viewport.

[0150] If the audio rendering characteristic indicator indicates that an object corresponding to the audio element is visible in the scene, the selector 209 decides to render it using headphones, otherwise the selector 209 renders the audio element using loudspeakers. In some embodiments, the audio rendering characteristic indicator directly indicates whether an object is visible. However, in other embodiments, the audio rendering characteristic indicator provides an indirect indication of whether the audio element corresponds to a visible scene object.

[0151] For example, the audio rendering characteristic indicator includes an indication of the scene object represented by the received visual data. The selector 209 then subsequently evaluates whether the object linked to the audio element is visible to the current listener pose. If it is visible, the selector 209 renders the object using headphones, otherwise the object is rendered by the loudspeakers.

[0152] In some embodiments, the distribution of an audio element across loudspeakers and headphones is based on an indication in the received content stream of whether the audio element is linked to a visual element / object in that content stream. If the indicator indicates that the audio element is linked to a visual element / object in the received content stream, the audio element is rendered on the headphones. If the indicator indicates that the audio element is not linked to a visual element / object in the received content stream, the audio element is rendered on the loudspeakers.

[0153] In the previous example, the selector 209 was configured to select the appropriate renderer 205, 207 based solely on the received data, however, it will be appreciated that in many embodiments further considerations, and in particular other data, are taken into account.

[0154] In many embodiments, the device includes a user input feature capable of receiving user input. The selector 209, in such embodiments, is further configured to select between the first renderer 205 and the second renderer 207 based on the user input. The user input is, for example, a direct indication of a preferred rendering, such as, for example, an explicit indication that a particular audio element should be rendered through headphones rather than loudspeakers. In other embodiments, the user input is more indirect, for example modifying the selection criteria or biasing the selection towards one of the renderers 205, 207. For example, the user input indicates that more audio elements are desired to be rendered through headphones, and the selector 209 changes the criteria to achieve this.

[0155] Thus, in some embodiments, the user can directly influence the distribution of elements across loudspeakers and headphones, One example is to give the user the possibility to manually specify individual elements for playback on headphones or loudspeakers.

[0156] Another example of distributed control by the user is to offer the user two or several modes that the user can choose from, for example a "personal experience" mode and a "shared experience" mode. If the user selects the "shared experience" mode, any of the embodiments described above are used in any combination for determining which audio elements should be rendered on the loudspeakers and headphones, respectively.

[0157] In some embodiments, the selector 209 is itself configured to analyze the audio element(s) and, based on this analysis, determine which renderer 205, 207 to use. For example, if no audio rendering characteristic indicator is received for a given audio element, the selector 209 subsequently analyzes the audio element(s) to determine audio characteristics, such as, for example, the number of audio elements in the scene, the number of channels per audio element, the position of the audio element, the distance of the audio element to the listener(s) (or to each loudspeaker), or the movement of the audio element. The selector 209 then subsequently determines which renderer 205, 207 to use based on this audio characteristic or based on a combination of these.

[0158] In a particular exemplary configuration, hereafter referred to as configuration X, the selector 209 selects a renderer for each audio element to create the most accurate spatial representation of the audio scene. For example, if an audio element as a virtual location is relatively close to the location of one of the physical loudspeakers, the audio element is rendered on that particular loudspeaker. Conversely, if the audio element falls in an area not covered by a loudspeaker, the audio element is rendered through headphones. Also, the same manner can be used for a single listener, and also for multiple listeners, provided that the audio element has the same orientation (from the listener's point of view) as the loudspeaker, provided that the multiple listeners are all aligned with the speakers. However, this is generally not practical since the user changes position over time. In this particular configuration X, the angular accuracy of the (binaural) headphone renderer 207 is taken into account by the selector 209 to make this decision.

[0159] Thus, in some embodiments, the selection of the appropriate renderer 205, 207 is further based on an analysis of the audio signal. For example, estimators of the acoustic properties of the audio signal may be used to determine characteristics such as the distance (or speed) of the audio object / audio source (especially in the case of multi-channel signals), or the reverberation time. Audio signal classifiers may also be used, such as a speech / music classifier, a music genre classifier, or an audio event classifier. Classifiers of a specific type may also be used to determine which type of microphone (HOA, Lavalier mic, Omnidirectional, XY...) was used to record a given signal. An analysis of the frequency distribution of the audio signal is also used to determine which audio system (headphones or loudspeakers) is more suitable for rendering the entire audio element.

[0160] In the previous examples, the selector 209 was configured to select either the first renderer 205 or the second renderer 207 for each audio element. However, it will be appreciated that this is not necessary or required. For example, in some embodiments, the selector 209 is configured to select which renderer 205, 207 to use for a group of audio elements.

[0161] Also, in some embodiments, the selector 209 is configured to select between the renderers 205, 207 separately for different parts of a single audio element. For example, for some audio elements, one part is rendered by the first renderer 205 and another part is rendered by the second renderer 207.

[0162] It will be appreciated that the audio element may be divided into different parts in different ways depending on the requirements and preferences of a particular embodiment. For example, in some embodiments, the audio element is received as a combination or collection of different parts, and the selector 209 selects the renderer 207 for each part individually. For example, the audio element may represent a particular audio source by a first component representing an audio source with a well-defined position (e.g. corresponding to direct audio) and a second component representing a more diffuse, distributed sound (e.g. corresponding to a reverberant sound). The selector 209 may be configured in such a scenario to render the first component using headphones and the second component using loudspeakers.

[0163] In other embodiments, the selector 209 is configured to split the audio element into different parts for rendering, for example, the received audio element corresponds to an audio signal, the audio signal is analyzed to be split into different parts, and the different parts are then rendered separately.

[0164] In particular, in many embodiments, different parts of an audio element correspond to different frequency ranges. For example, the selector 209 is configured to select which renderer 205, 207 to use for a given first part corresponding to a particular frequency range. The selector 209 then does the same for a different frequency range, thus leading to different renderers 205, 207 being used for the first and second frequency ranges.

[0165] In some embodiments, different audio rendering characteristic indicators are provided for different parts of an audio element, and the selector 209 takes the particular audio rendering characteristic indicator into account when deciding for a given part how it should be rendered. In other embodiments, audio rendering characteristic indicators are provided for the entire audio element, but different criteria are used for different parts. For example, for the mid-to-high frequency range, the choice between headphones and loudspeakers is made depending on the received audio rendering characteristic indicator for the audio element, while for the very low frequency range, the first renderer 205 is used to render the signal on the loudspeakers, independent of what the audio rendering characteristic indicator indicates (reflecting that low frequencies tend to provide few significant spatial cues).

[0166] For example, the signal is separated into low and high frequency parts using low pass and high pass filtering, where the low frequency part is sent to loudspeakers and the high frequency part is sent to headphones depending on the audio rendering characteristic indicator. In some such embodiments, advanced audio source separation is used (e.g., splitting each time-frequency point between renderers).

[0167] The use of filtering that preserves the energy at each time-frequency point allows the physical hybrid rendering system to attenuate possible errors produced by the filtering process.

[0168] The described approach provides several advantageous effects, including enabling / facilitating direct interaction between co-located users while enabling a perceived accurate spatial rendering of the audio scene, as previously described.

[0169] The approach reduces complexity and resource usage in many scenarios, potentially requiring a reduced amount of binaural processing. Another advantage that can often be achieved is a reduction in the energy used by the headphone playback system, e.g., in terms of amplifier power and / or processing load for an embedded renderer, which can be significant in the case of untethered headphones (e.g., battery-powered headphones).

[0170] Another interesting property of hybrid audio reproduction systems for VR applications is that they tend to offer improved safety: indeed, contrary to wearing closed headphones, the participant is never completely isolated from potential dangers in the real environment around him, which is an important factor in many practical situations.

[0171] Yet another advantage of a hybrid system such as the one described is that some of the audio content is rendered on a common loudspeaker set, which tends to enhance the users' sense of a shared experience. This approach tends to provide an improved user experience.

[0172] It will be appreciated that in the above description, for clarity, embodiments of the invention have been described in terms of different functional circuits, units and processors. However, it will be apparent that any suitable distribution of functionality between different functional circuits, units or processors may be used without detracting from the invention. For example, functionality shown as being performed by separate processors or controllers may be performed by the same processor or controller. Thus, references to specific functional units or circuits should not be construed as indicative of a strict logical or physical structure or organization, but only as references to suitable means for providing the described functionality.

[0173] The invention may be implemented in any suitable form including hardware, software, firmware or any combination of these. The invention is optionally implemented at least partly as computer software running on one or more data processors and / or digital signal processors. The elements and components of an embodiment of the invention may be physically, functionally and logically implemented in any suitable way. Indeed, functionality may be implemented in a single unit, in several units or as part of other functional units. Thus, the invention may be implemented in a single unit or may be physically and functionally distributed between different units, circuits and processors.

[0174] Although the present invention has been described with respect to several embodiments, the present invention is not limited to the specific form described herein. Rather, the scope of the present invention is limited only by the appended claims. Moreover, although features appear to be described with respect to certain embodiments, those skilled in the art will recognize that various features of the described embodiments may be combined in accordance with the present invention. In the claims, the term "comprises, includes, has" does not exclude the presence of other elements or steps.

[0175] Moreover, although individually listed, a plurality of means, elements, circuits or method steps may be implemented by, for example, a single circuit, unit or processor. Moreover, although individual features are included in different claims, they may, in some cases, be advantageously combined, and the inclusion in different claims does not imply that the combination of features is not feasible and / or advantageous. Moreover, the inclusion of a feature in one category of claims does not imply a limitation of this category, but rather indicates that the feature is equally applicable to other claim categories, as appropriate. Moreover, the order of features in the claims does not imply a particular order in which the features must be operated, and in particular the order of individual steps in a method claim does not imply that the steps must be performed in this order. Rather, the steps may be performed in any suitable order. Moreover, singular references do not exclude a plurality. Thus, references to "first", "second", etc. do not exclude a plurality. Reference signs in the claims are provided merely as a clarifying example and should not be construed as limiting the scope of the claims in any way.

Claims

1. 1. An audio device comprising a receiver circuit, a first renderer circuit, a second renderer circuit, and a selector circuit, The receiver circuit receives data; the data represents an audio scene; the data includes audio data; the audio data is for a set of audio elements corresponding to audio sources in the audio scene; the set of audio elements includes a first audio element; the first renderer circuitry renders an audio element by generating a first set of audio signals for a set of loudspeakers; the second renderer circuit renders the audio element by generating a second set of audio signals for headphones; the selector circuit performs an analysis of the first audio element and determines an audio characteristic of the first audio element as a result of the analysis; the selector circuitry selects between the first renderer circuitry and the second renderer circuitry such that rendering of at least a first portion of the first audio element is responsive to the audio characteristics. Audio equipment.

2. the audio device generating audio signals for a plurality of listeners; the first renderer circuit generates the first set of audio signals as a common set of audio signals for the multiple listeners; the second renderer circuit generates the second set of audio signals for headphones for a first listener of the plurality of listeners; The audio device of claim 1 , wherein the second renderer circuit generates a third set of audio signals for headphones for a second listener of the plurality of listeners.

3. The audio device of claim 1 , wherein the first portion is a frequency subrange of the first audio element.

4. the selector circuitry selecting between the first renderer circuitry and the second renderer circuitry for the first portion of the first audio element; 2. The audio device of claim 1, wherein the selector circuit selects between the first renderer circuit and the second renderer circuit for a second portion of the first audio element.

5. The audio device of claim 1 , wherein analyzing the first audio element comprises analyzing the audio data for the first audio element to determine the audio characteristics.

6. 2. The audio device of claim 1, wherein the audio characteristics include at least one of: a number of audio elements in the set of audio elements, a number of channels for the first audio element, a position of a first audio source in the audio scene corresponding to the first audio element, a distance between the first audio source and a listener, a distance between the first audio source and the loudspeaker, a movement of the first audio source, or a reverberation time for the first audio element.

7. 2. The audio device of claim 1, wherein the audio characteristics include at least one of an audio signal classifier for the first audio element, a type of microphone used to record the first audio element, and a frequency distribution of an audio signal corresponding to the first audio element.

8. the data includes metadata about the audio data; The audio device of claim 1 , wherein analyzing the first audio element comprises analyzing the metadata to determine the audio characteristics.

9. the audio device further comprising a user input circuit; the user input circuitry receives a user input; the selector circuit selects between the first renderer circuit and the second renderer circuit; 2. The audio device of claim 1, wherein a selected rendering circuit is responsive to the user input to render at least the first portion of the first audio element.

10. receiving data representative of an audio scene, the data includes audio data for a set of audio elements; the set of audio elements corresponds to a set of audio sources in the audio scene; receiving the set of audio elements, the set of audio elements including a first audio element; Rendering the audio element by generating a first set of audio signals for a set of loudspeakers; Rendering the audio element by generating a second set of audio signals for headphones; performing an analysis of the first audio element to determine audio characteristics of the first audio element; selecting, as a result of the analysis, between rendering at least a first portion of the first audio element through the set of loudspeakers and rendering at least the first portion of the first audio element through the set of headphones; 13. A method of audio processing comprising:

11. A computer program stored on a non-transitory medium, the computer program being adapted to perform the method of claim 10 when executed on a processor.

12. generating audio signals for a plurality of listeners; generating the first set of audio signals as a common set of audio signals for the multiple listeners; generating the second set of audio signals for headphones for a first listener of the plurality of listeners; and generating a third set of audio signals for headphones for a second listener of the plurality of listeners.

13. The method of claim 10 , wherein the first portion is a frequency subrange of the first audio element.

14. selecting between a first renderer and a second renderer for the first portion of the first audio element; and selecting between the first renderer and the second renderer for a second portion of the first audio element.

15. The method of claim 10 , wherein analyzing the first audio element comprises analyzing the audio data for the first audio element to determine the audio characteristics.

16. 11. The method of claim 10, wherein the audio characteristics include at least one of: a number of audio elements in the set of audio elements, a number of channels for the first audio element, a position of a first audio source in the audio scene corresponding to the first audio element, a distance between the first audio source and a listener, a distance between the first audio source and the loudspeaker, a movement of the first audio source, or a reverberation time for the first audio element.

17. 11. The method of claim 10, wherein the audio characteristics include at least one of an audio signal classifier for the first audio element, a type of microphone used to record the first audio element, and a frequency distribution of an audio signal corresponding to the first audio element.

18. the data includes metadata about the audio data; The method of claim 10 , wherein analyzing the first audio element comprises analyzing the metadata to determine the audio characteristics.

19. 20. The method of claim 18, wherein the audio characteristics include at least one of an audio signal classifier for the first audio element and a type of microphone used to record the first audio element.

20. receiving a user input; and selecting, in response to the user input, a rendering of at least the first portion of the first audio element.

Images