Apparatus and method for generating an audio signal

Abstract

An apparatus comprises a first receiver (103) receiving a plurality of audio capture signals comprising audio of a scene captured by one or more audio capture devices. A second receiver (101) receives an image capture signal comprising at least one image representing a capture frustum of the scene. A third receiver (105) receives metadata comprising capture region indications for at least some of the audio capture signals where a capture region indication for an audio capture signal is indicative of a region of audio capture of the scene for the audio capture signal. An image generator (107) generates a view image of the scene for a view frustum from the image capture signal, and an audio generator (109) generates an output audio signal representing the scene by mixing the plurality of audio capture signals in dependence on the view frustum and the capture region indications.

Description

FIELD OF THE INVENTION

[0001] The invention relates to generation of an audio signal from a plurality of audio capture signals, and in particular, but not exclusively, to generation of an audio signal capturing audio for an event, such as e.g. a sports event.BACKGROUND OF THE INVENTION

[0002] Capturing live real world scenes has become increasingly important in the last decades. For example, capturing audio and video for large events, such as sport or concert events, has become widespread. Further, the demands and requirements for provided services, and thus for the audio and video capture, have increased substantially. For example, spatial audio capture and rendering is becoming prevalent and there is need for accurate and efficient capture, distribution, and rendering of spatial audio.

[0003] In many practical applications, audio is captured by a plurality of microphones at different positions. For example, a plurality of microphones is often used to capture audio in an environment. In particular, when audio of a large environment / scene is to be captured, such as when capturing audio in a sports arena or stadium, it is necessary to deploy a substantial number of widely distributed microphones in order to ensure sufficient capture across the entire scene.

[0004] Audio capture is often associated with and linked to video / image capture of the scene. It is desired for rendering of audiovisual data that a high quality is provided and in particular that the rendering of the audio and video is suitably coordinated and consistent. It is further desired that the rendering can provide a flexible and adaptive rendering of a scene.

[0005] However, current approaches for rendering of audiovisual content tend to be suboptimal in many scenarios. In particular, current approaches tend to provide a suboptimal quality in many scenarios, and specifically the coordination and consistency between the audio and video rendering of a scene tends to be suboptimal. In many scenarios, the rendering further tends to not be as flexible and adaptable as is desired.

[0006] Hence, an improved generation of audio would be advantageous, and in particular an approach allowing reduced complexity, increased flexibility, facilitated implementation, reduced cost, improved audio capture, improved consistency / coordination between rendered audio and video, improved adaptation and / or flexibility, and / or improved performance would be advantageous.SUMMARY OF THE INVENTION

[0007] Accordingly, the Invention seeks to preferably mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination.

[0008] According to an aspect of the invention, there is provided an apparatus for generating an output audio signal providing an audio representation of a scene, the apparatus comprising: a first receiver arranged to receive a plurality of audio capture signals, each audio capture signal comprising audio of the scene captured by one or more audio capture devices; a second receiver arranged to receive at least one image capture signal, each image capture signal comprising at least one image representing a capture frustum of the scene; a third receiver arranged to receive metadata comprising capture region indications for at least some of the audio capture signals, a capture region indication for an audio capture signal being indicative of a region of audio capture of the scene for the audio capture signal; an image generator arranged to generate a view image of the scene for a view frustum from the at least one image capture signal; and an audio generator arranged to generate the output audio signal including mixing the plurality of audio capture signals in dependence on the view frustum and the capture region indications.

[0009] The invention may provide improved generation of an audio signal in many embodiments and scenarios. In particular, it may provide an improved audio representation of a scene, and in particular it may in many scenarios provide an output audio signal more consistent and coordinated with a visual representation provided by the generated view image. The approach may provide an improved user experience in many scenarios. It may in particular allow flexible adaptation and may in many scenarios provide audio effects resulting in an improved user perception of the scene. For example, the sensation of immersion and presence for a typical user is may in many scenarios be enhanced by improved audio mixing.

[0010] In some embodiments, the audio mixing may be a non-coherent mixing / combination / summation of the weighted audio capture signals. In many embodiments, the mixing weights may be scalar values. In many embodiments, mixing weights may be gains for the audio capture signals. The mixing may be a weighted combination / summation.

[0011] A pose may be a position and / or direction / orientation. The audio capture devices may be microphones. The capture frustum for an image capture signal may be a viewing frustum for a video capture device capturing the scene to provide the image capture signal. A view frustum may be a geometrical frustum of the pyramid of vision for a view pose. In the extreme, a view frustum may be the pyramid of vision for a given pose. The view frustum may be represented by one or more of a capture / view pose, a field of view (one or more viewing angles), and / or a depth / distance range / value. An audio capture signal may be an audio signal representing audio captured in the (typically real world) scene. An image capture signal may be an image signal representing visual data / information captured in the (typically real world) scene.

[0012] In accordance with an optional feature of the invention, the audio generator is arranged to increase a mixing weight for a first audio capture signal for an increasing proximity of a capture region of the first audio capture signal to the view frustrum.

[0013] This may provide improved performance and / or operation in many embodiments and may in particular provide a highly advantageous adaptation of the audio signal to the generated image.

[0014] In accordance with an optional feature of the invention, the audio generator is arranged to select a subset of audio capture signals included in the mixing, the audio generating circuit being arranged to select between including and excluding a given audio capture signal in the mixing in dependence on a spatial relationship (e.g. a distance) between the view frustum and a region of audio capture for the given audio capture signal (as indicated by the capture region indication).

[0015] This may provide improved performance and / or operation in many embodiments and may in particular provide a highly advantageous adaptation of the audio signal to the generated image. For example, in many embodiments and scenarios, a user may be provided with an effect where sound of the visible scene elements is audible while sound of non-visible scene elements may be further attenuated to reduce the risk of such sounds becoming distracting.

[0016] In accordance with an optional feature of the invention, the metadata comprises a property indication for at least one audio capture signal, the property indication being indicative of at least one of: a directional capture sensitivity for the at least one audio capture signal; a capture pose for the at least one audio capture signal; a capture device property for an audio capture device capturing the at least one audio capture signal; or a capture range for the at least one audio capture signal; and the audio generator is arranged to adapt the mixing in dependence on the property indication.

[0017] This may provide improved performance and / or operation in many embodiments. The property indication may for example in particular allow or facilitate the apparatus making a more accurate assessment of which spatial regions are audible in the audio capture signal.

[0018] In accordance with an optional feature of the invention, the apparatus comprises a first circuit for determining a viewer attention region in the scene; and the audio generator is arranged to adapt the mixing in dependence on the viewer attention region.

[0019] This may provide improved performance and / or operation in many embodiments. It may provide an improved user experience.

[0020] In many embodiments, the audio generator is arranged to adapt the mixing in dependence on the viewer attention region relative to the view frustum, and specifically dependent on a spatial relationship (such as a distance) between the viewer attention region and the view frustum.

[0021] In accordance with an optional feature of the invention, the apparatus comprises a gaze detector arranged to detect a gaze of a viewer, and the first circuit is arranged to determine the viewer attention region in dependence on the gaze of the viewer.

[0022] This may provide improved performance and / or operation in many embodiments. It may allow the output audio signal to be generated to reflect that it is likely that the user is more interested in sound of scene elements that the viewer is looking at, than in other sounds.

[0023] The gaze of the viewer may include / consist in the focus of the gaze of the viewer. For example, if the viewer zooms his view to a specific object or area in the scene, the sound corresponding to that object or area may be emphasized (with the area possibly being close by, further away or restricted to a narrower angle).

[0024] In accordance with an optional feature of the invention, the apparatus comprises a view circuit arranged to provide a desired view of the scene and the image generator is arranged to dynamically select the view frustum as a subset of the capture frustum for the at least one image capture signal in response to changes in the desired view.

[0025] This may provide improved performance and / or operation in many embodiments. It may provide an improved user experience. It may in many embodiments and scenarios increase the sensation of immersion and presence for a typical user.

[0026] In accordance with an optional feature of the invention, the metadata includes an indication of the capture frustum for the at least one image capture signal, and the image generator is arranged to determine the view frustum in dependence on the capture frustum.

[0027] This may be particularly advantageous in many embodiments and scenarios.

[0028] In accordance with an optional feature of the invention, the third receiver is further arranged to receive dynamically varying capture pose data for at least one audio capture signal of the plurality of audio capture signals; and the audio generating circuit is arranged to dynamically adapt a mixing weight for the at least one audio capture signal in dependence on the dynamically varying capture pose data.

[0029] This may be particularly advantageous in many embodiments and scenarios. For example, the sensation of immersion and presence for a typical user is may in many scenarios be enhanced by improved audio mixing.

[0030] In accordance with an optional feature of the invention, the second receiver is arranged to receive dynamically varying image capture pose data for the at least one image capture signal; and the image generator is arranged to determine the view frustum in dependence on the dynamically varying image capture pose data.

[0031] This may provide improved performance and / or operation in many embodiments.

[0032] In accordance with an optional feature of the invention, the third receiver is arranged to receive a synchronization indication, the synchronization indication indicating a set of capture signals of the plurality of audio capture signals and the at least one image capture signal being time synchronized with each other; and the audio generator is arranged to adapt the mixing dependent on the synchronization indication.

[0033] This may provide improved performance and / or operation in many embodiments. The capture signals of the set of capture signals may be time synchronized with a given relative time offset between the signals.

[0034] In accordance with an optional feature of the invention, the third receiver is arranged to receive an accuracy indication indicative of a synchronization accuracy for the set of capture signals, and the audio generating circuit is arranged to adapt the mixing dependent on the synchronization accuracy.

[0035] This may provide improved performance and / or operation in many embodiments.

[0036] In some embodiments, the mixing of the plurality of audio capture signals may include an equalization / frequency dependent attenuation / gain and the equalization / frequency dependent attenuation / gain in dependence on the view frustum and the capture region indications. Specifically, the equalization / frequency dependent attenuation / gain for at least one audio capture signal may be dependent on the view frustum and the capture region indication for that audio capture signal.

[0037] In accordance with an optional feature of the invention, the metadata comprises a playlist file comprising the capture region indications (e.g., indicated as pose information) and the first receiver is arranged to retrieve audio capture signals from a remote source in dependence on the capture region indications.

[0038] This may provide improved performance and / or operation in many embodiments.

[0039] According to another aspect of the invention, there is provided a method of generating an output audio signal providing an audio representation of a scene, the method comprising: receiving a plurality of audio capture signals, each audio capture signal comprising audio of the scene captured by one or more audio capture devices; receiving at least one image capture signal, each image capture signal comprising at least one image representing a capture frustum of the scene; receiving metadata comprising capture region indications for at least some of the audio capture signals, a capture region indication for an audio capture signal being indicative of a region of audio capture of the scene for the audio capture signal; generating a view image of the scene for a view frustum from the at least one image capture signal; and generating the output audio signal including mixing the plurality of audio capture signals in dependence on the view frustum and the capture region indications.

[0040] These and other aspects, features and advantages of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter.BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Embodiments of the invention will be described, by way of example only, with reference to the drawings, in which FIG. 1 illustrates an example of elements of a render apparatus in accordance with some embodiments of the invention; FIG. 2 illustrates an example of capture scenario for a sports event; and FIG. 3 illustrates some elements of a possible arrangement of a processor for implementing elements of an audio apparatus in accordance with some embodiments of the invention. DETAILED DESCRIPTION OF SOME EMBODIMENTS OF THE INVENTION

[0042] FIG. 1 illustrates example elements of a rendering apparatus which is arranged to generate an output audio signal associated with an image, and often with a video signal. The output audio signal may be adapted based on the image / video which may also be generated by the rendering apparatus. The audio and image / video signals represent a scene captured by suitable audio and image capture devices, such as specifically by microphones and cameras capturing a scene. The processing of the apparatus is particularly suitable for providing a flexible and adaptive processing of captured audio to generate an audio signal representing audio in a large scene while maintaining low complexity and resource usage and adapting the audio to match the generate image(s).

[0043] The audio apparatus of FIG. 1 comprises a video signal receiver 101 (also referred to as the second receiver 101) which is arranged to receive one or more image capture signals, and specifically video capture signals, from one or more cameras capturing a scene. The video signal receiver 101 may specifically receive image capture signals from one or more video signals, and thus the image signal will typically be a video capture signal. The following description will focus on such a scenario and thus will focus on the video signal receiver 101 receiving video capture signals and a stream of images, but it will be appreciated that in many other embodiments the video signal receiver 101 may receive an image capture signal comprising only one or more static images for the same time instant. In many cases the received image capture signal(s) may provide 3D images in accordance with a suitable representation (such as an image+depth representation).

[0044] Each image capture signal provides a capture of a scene, and specifically a real world scene. Each image capture signal is linked with a capture frustum. The capture frustum for an image capture signal may be a viewing frustum for a video capture device that is represented by the image capture signal. The capture frustum corresponds to the frustum that is captured by the image capture signal if no occlusion occurs. The capture frustum may typically be a view frustum of a camera capturing the scene to generate the image capture signal.

[0045] A view frustum may be a region of space in the scene / world that may be viewed from a given pose. It may typically correspond to what will appear on a screen representing the view. A capture pose may correspond to the part of the scene / world that would appear on a screen presenting the image capture signal (and assuming no occlusion).

[0046] The view frustum may specifically be a field of view for a camera. A view frustum may be a geometrical frustum, which may be a truncation with parallel planes of the pyramid of vision. This may be an adaptation of an (idealized) cone of vision that a camera or eye would have to rectangular viewports. The view / capture frustum may in many cases be a synonym for pyramid of vision / capture. Specifically, the truncation may in the extreme case be at a zero and / or infinite distance respectively.

[0047] A view / capture frustum may include a view / capture pose from which the scene, and specifically the real world, is viewed / captured.

[0048] A view / capture frustum may be represented by one or more of a view / capture pose (for a camera), a field of view (e.g., for a camera), and / or none, one, or two of a depth / distance range / value. The depth / distance range / value may specifically be values for the truncating planes. In many embodiments and scenarios, such values may be respectively zero and infinite corresponding to no truncation of the frustum.

[0049] Thus, the image capture signal(s) may represent / capture views of the scene from a specific capture pose(s) and capture frustum.

[0050] In the field, the terms placement and pose are used as a common term for position and / or direction / orientation. The combination of the position and direction / orientation of e.g., an object, a camera, a microphone, a head, or a view may be referred to as a pose or placement. Thus, a placement or pose indication may comprise six values / components / degrees of freedom with each value / component typically describing an individual property of the position / location or the orientation / direction of the corresponding object. Of course, in many situations, a placement or pose may be considered or represented with fewer components, for example if one or more components is considered fixed or irrelevant (e.g., if all objects are considered to be at the same height and have a horizontal orientation, four components may provide a full representation of the pose of an object). In the following, the term pose is used to refer to a position and / or orientation which may be represented by one to six values (corresponding to the maximum possible degrees of freedom). A pose may be a position and / or orientation.

[0051] A pose having the maximum degrees of freedom, i.e., three degrees of freedom of each of the position and the orientation resulting in a total of six degrees of freedom. A pose may thus be represented by a set or vector of six values representing the six degrees of freedom and thus a pose vector may provide a three-dimensional position and / or a three-dimensional direction indication. However, it will be appreciated that in other embodiments, the pose may be represented by fewer values. Similarly, in some embodiments, more than six values may be used to represent a pose. For example, rotation / orientation may be given in quaternions which means a pose may be represented by seven values (three for position and four for rotation).

[0052] A pose may also be represented by more than six values, which would be an overcomplete representation. An example is when a rotation matrix is used to represent the orientation.

[0053] A pose may be at least one of an orientation and a position. A pose value may be indicative of at least one of an orientation value and a position value. A pose may be a position and / or orientation.

[0054] A system or entity based on providing the maximum degree of freedom for the viewer is typically referred to as having 6 Degrees of Freedom (6DoF). Many systems and entities provide only an orientation or position and these are typically known as having 3 Degrees of Freedom (3DoF).

[0055] The audio apparatus further comprises an audio signal receiver 103, also referred to as the first receiver 103, arranged to receive audio capture signals, where each audio capture signal comprises audio of the scene captured by one or more audio capture devices.

[0056] In many embodiments, the audio capture devices are each arranged to generate an audio capture signal from audio captured by one or more microphones capturing the scene. Indeed, one or more of the audio capture signals may be microphone signals, and may specifically each be from a single microphone. Each such microphone may for example be a directional microphone or may be an omnidirectional microphone. In many scenarios, one or more audio capture devices may comprise a plurality of microphones and these may be combined to generate a single audio capture signal. In particular, in many embodiments, one or more of the audio capture devices may comprise a microphone array and may include audio beamforming processing to generate one or more audio capture signals each corresponding to a directional, and often adaptive, beam shape / sensitivity pattern. In some embodiments, an audio capture device may comprise a microphone array that is used to beamform multiple audio signals. For instance, one microphone array could track each of the players in a game and provide that as multiple audio tracks. In such a scenario, a single hardware device may provide multiple audio capture signals with different capture regions and thus a single hardware device may equivalently be considered to correspond to multiple audio capture devices.

[0057] The render apparatus further comprises a metadata receiver 105, also referred to as the third receiver 105, which is arranged to receive metadata that includes capture region indications for at least some, and typically all, of the audio capture signals. A capture region indication for an audio capture signal is indicative of the region of audio capture of the scene for the audio capture signal.

[0058] Each of the audio capture devices (specifically microphones) has a capture pose corresponding to the position and / or orientation of the audio capture device (microphone) in the scene. Thus, each of the audio capture signals is linked with a capture pose and represents a capture of audio of the scene from that capture pose. In some embodiments, only two microphones / capture positions may be employed but in many embodiments the plurality of microphones may include substantially more microphones, and indeed in some embodiments 5,10, or even more microphones may be considered.

[0059] The audio capture devices may be distributed throughout the scene and typically at least some of the audio capture devices have different capture positions. Alternatively, at least some of the audio capture signals may represent audio with different directional sensitivities, such as audio from different audio beams or with different directional sensitivities. Accordingly, the audio capture of the scene may differ between the audio capture signals and each audio capture signal may be associated with a capture region. A capture region may for example be given as a region for which an acoustic attenuation to the capture pose is less than a given value. Each audio capture signal may represent a capture of part of the scene.

[0060] The capture region indication may in some cases be data describing a particular region or part of the scene. However, in many embodiments, the capture region indication may be an indication of a capture pose for the audio capture signal. In many cases, a capture pose for a given audio capture signal may, typically together with information of the microphone sensitivity (including directional sensitivity), describe / define a capture region. In some embodiments, the microphone properties may be implicitly used / assumed, and indeed in many embodiments a capture pose may in itself be considered a capture region indication.

[0061] In the case of a beamforming microphone array, the capture region indication may e.g. describe: the coverage of a beam, for example which (3D) angles the beam is able to point to. The width of a beam, e.g., the range of the angular widths, or a collection of beam patterns. These properties may depend on the (3D) angle. For example, for a linear array, the available range of beam patterns may be different for broad-side (perpendicular to the linear array), end-fire (in line with the linear array) or any other angle.

[0062] The render apparatus comprises an image generating circuit / image generator 105 which generates at least one view image of the scene for a view frustum. The view frustum may in many embodiments be represented / determined by a view pose and a viewing angle or viewport / field of view. For example, the image generator 107 may receive a desired view pose and may proceed to generate the view image to represent a view of the scene from the desired view pose.

[0063] The image generator 107 may in many embodiments be arranged to generate a sequence / stream of images and specifically may generate a video signal. The view frustum for the generated image may dynamically change.

[0064] The image generator 107 may for example be arranged to generate the image by selecting a section of the image represented by the received image / video signal, and specifically the image generator 107 may be arranged to perform a zooming in on the received image from the image capture device / camera.

[0065] In many embodiments, the image generator 107 may be arranged to generate the image to be a view of the scene from a different view pose than the capture pose, i.e., it may different than the pose for the received image signal. In many embodiments, the received image signal may be a 3D image signal, such as e.g., an image / video with associated depth. The image generator 107 may, based on this information be arranged to synthesize the image / video from a different view pose. It will be appreciated that many algorithms and techniques for such image synthesis and for performing view shifting will be known to the skilled person, and that any suitable approach may be used.

[0066] The image generator 107 may e.g., be arranged to provide the generated image / video signal to a display which may present the image(s).

[0067] The render apparatus additionally comprises an audio generating circuit / audio generator 109 which generates the output audio signal. The audio generator 109 includes a mixer which is arranged to mix the audio capture signals to generate the output audio signal. The generation of the output audio signal may include other signal processing such as filtering etc. A suitable filtering could be an equalization. Other signal processing applied could comprise speech enhancement and / or noise reduction to e.g., further enhance a speech signal or another distinct sound.

[0068] The mixing is dependent on the view frustum and the capture region indications.

[0069] The audio mixing may typically be a weighted combination / summation where each audio capture signal is weighted by weights. The weights may in many embodiments be scalar values and thus may in many embodiments be gains that are applied to the audio capture signals.

[0070] If we weigh the audio capture signals p i with weights w i , then the mixing may be performed as: p mix = ∑ i = 1 N mics w i p i where the summation may be over all audio capture signals p i and the weights w i may specifically be scalar (and typically positive) values.

[0071] The audio generator 109 may specifically be arranged to adapt the mixing weights in dependence on the view frustum and the capture region indications, and in particular in many embodiments, the (relative) mixing weight for a given audio capture signal may be determined as dependent on a distance between the capture region as indicated by the capture region and the view frustum.

[0072] In most embodiments, the audio generating circuit 109 is arranged to increase a mixing weight for a given audio capture signal for an increasing proximity of the capture region of the first audio capture signal to the view frustrum. The capture region is dependent on the capture region indication for the given audio capture signal.

[0073] Depending on the view frustum, the mixing may in some embodiments and scenarios also attenuate or enhance capture regions that are outside of the view frustum, e.g., sound of an audience or object (football) that is in the opposite direction of the view frustum may be attenuated or amplified to provide for an enhanced (typically artistically motivated) experience.

[0074] Thus, the render apparatus is arranged to dynamically adapt the output audio signal dependent on the view frustum and specifically the audio mix may be adapted to correspond to the view frustum, and thus to the view image.

[0075] Specifically, the render apparatus may generate a spatial audio representation for a scene. The spatial representation is generated on the basis of two or more audio capture devices which capture a sound field in the scene and one or more image / video capture devices organized for capturing a field of view in the scene. The render apparatus may generate the spatial audio representation based on associated metadata representing a capturing area / region in the scene. The render apparatus may generate a first field of view in the scene from the image signals from the camera device. The render apparatus may then generate the spatial audio representation (in the form of the output audio signal) based on the metadata and the first field of view, and specifically based on the capture region indications and the view frustum. The spatial audio representation may specifically correspond to / be representative of the first field of view / the view frustum.

[0076] As an example of an application, FIG. 2 illustrates an example setup for capturing a sports match consisting of a single camera C generating the image signal and two microphones M1 and M2 generating the audio capture signals. The camera (C) and first microphone (M 1 ) are directed such that the camera (C) images / captures the sports action in a sport area 201 (e.g., a football pitch) while the microphone M 1 captures sound that is generated by the action on the field / sports area 201 (e.g., player's speech, ball kicks). The second microphone (M 2 ) is directed towards the audience 203 and will pick up sounds from the audience.

[0077] Knowledge of this configuration (via the metadata) allows for new client application functionality. For example, when a user visually zooms in on the scene, an audio mix can gradually give more weight to the audio of the forward-facing microphone and less weight to the audio from the audience microphone. The user / director / algorithm can also mute the audio from the audience.

[0078] To allow this and other functionality, meta data related to microphone positions and orientations and their relations to the camera / capture frustum may be transported all the way from the capture system to the client application.

[0079] In some embodiments, the audio generator 109 is arranged to select / deselect audio capture signals to be included in the audio mix dependent on the capture region indication(s) and the view frustum. In particular, the audio generator 109 may for a given audio capture signal decide to include the audio capture signal in the audio mix or exclude it from the audio capture signal depending on the spatial relationship between the view frustum and the region of audio capture for the given audio capture signal as indicated by the capture region indication for the audio capture signal.

[0080] The audio generator 109 may specifically be arranged to select / deselect a given audio capture signal depending on the distance in the scene from the capture region to the view frustum. As a low complexity example, the audio generator 109 may determine a minimum distance between the capture region and the view frustum using any suitable distance measure (such as e.g., a straight Euclidean distance). If the distance measure is below a threshold, the audio capture signal may be included in the mix and otherwise it may be excluded. As another example, the audio generator 109 may determine a distance for all audio capture signals and select a predetermined number of audio capture signals. For example, the audio generator 109 may be arranged to mix four audio capture signals, and the audio generator 109 may select the four audio capture signals for capture regions closest to the view frustum.

[0081] In many embodiments, the distance measures may include a consideration of, and dependency on, the sensitivity of the microphones in different directions, e.g., it may include the directionality of the capture of the audio capture signal.

[0082] The selection and deselection of a given audio capture signal may be considered to correspond to setting mixing weights for the given audio capture signal to a non-zero or zero value respectively.

[0083] In some embodiments, the audio generator 109 may be arranged to determine an attenuation measure / indication for audio sources in the view frustum to a given audio capture signal, and may be arranged to set a mixing weight for a given audio capture signal in dependence on the attenuation measure / indication. In some embodiments, the audio generator 109 may be arranged to select / deselect audio capture signals to be included in the audio mix dependent on the attenuation measure / indication. The attenuation measure / indication may be determined to include a contribution from, or consist in, an acoustic attenuation from (a position in) the view frustum to the capture pose. Such an acoustic attenuation may be dependent on the distance between the view frustum and the capture pose, and indeed such a distance measure may directly be considered an attenuation measure / indication in some embodiments.

[0084] The attenuation measure / indication may include a contribution from the sensitivity / gain of the audio capture device, such as the gain of a microphone or microphone array in the direction of the view frustum. The attenuation measure / indication may in some embodiments be calculated as the combination of a capture sensitivity / attenuation of the audio capture device in the direction of the view frustum and the acoustic attenuation from the view frustum to the capture position / pose. The attenuation measure / indication may take into account the sensitivity / beampattern of the audio capture device in the direction of the view frustum, and this may be adaptive properties, e.g., for audio capture devices that employ microphone arrays and adaptive beamforming.

[0085] The audio generator 109 may typically be arranged to increase a weight for a given audio capture signal to lower the attenuation measure, and specifically the weight may be a monotonically decreasing function of the attenuation measure. This may bias the output audio signal towards audio capture signal that are more likely to capture audio of the view frustum.

[0086] Thus, in many embodiments, the audio generator 109 may, based on the received metadata and capture region indications, determine mixing weights for the audio capture signals, and may specifically select a subset of audio capture signals based thereupon. The weights / selection may be dependent on the coverage of the audio capture range relative to the view frustum. The determination may include consideration of e.g., beampattern, sensitivity etc.

[0087] In some embodiments, the metadata may provide further information than the capture region indications and these may be used to adapt the mixing and specifically may be used in determining the weights.

[0088] In some embodiments, the metadata may include a directional capture sensitivity for one or more of the audio capture signals. For example, the beam pattern for a given audio capture device may be transmitted in the metadata and may be used by the audio generator 109 to determine an attenuation measure for the audio capture signal which may then be used to determine the weight and / or to select audio capture signals.

[0089] In some embodiments, the metadata may include a capture pose for one or more of the audio capture signals. The capture pose may include a capture position that allows e.g., a path loss attenuation to be determined which may be used to determine the attenuation measure / indication which is then used to determine the weight and / or to select audio capture signals. The capture pose may include a direction / orientation of the capture device, such as e.g., a direction / orientation of a directional microphone. Knowledge of the beam pattern / directional sensitivity for the microphone may then allow the sensitivity / gain in the direction of the view frustum to be determined. This may then be used to determine the attenuation measure / indication which is then used to determine the weight and / or to select audio capture signals.

[0090] In many embodiments, the metadata may include a capture device property for one or more of the audio capture signals, such as specifically a property of the audio capture device that captured the audio capture signal. For example, the metadata may indicate a type or manufacturer of the audio capture device, a reference beam pattern of the audio capture device etc. The audio generator 109 may determine the attenuation measure / indication based on such properties and may then determine the weights / select the audio capture signals.

[0091] In some embodiments, the metadata may indicate a capture range for one or more of the audio capture signals / audio capture devices. For example, a nominal or reference capture range may directly be indicated for the audio capture device and the audio generator 109 may proceed to determine the capture region based on the capture pose and the given audio capture range. The audio capture range may be a nominal or reference value for the given audio capture device / microphone and the capture region indication may indicate a capture pose which together with the nominal or reference audio capture range may directly allow the capture region for the audio capture signal to be determined.

[0092] The render apparatus may in many embodiments include a view processor 111 which is arranged to receive a desired view pose, and the image generator 107 may be arranged to dynamically determine the view frustum based on the view pose.

[0093] As a low complexity embodiment, the view processor 111 may receive a user input from which the view pose is determined. For example, a joystick may by a viewer / user be used to select the desired view of the scene. The image generator 107 may then proceed to generate the view image to be a view of the scene from the desired view pose. The user may in such an embodiment e.g., manipulate the joystick to change the desired view of the scene, including both the orientation and / or position of the view of the scene.

[0094] In some embodiments, the view processor 111 may receive a zoom input and the image generator 107 may be arranged to adapt the size of the viewport of the scene that is presented by the image based on the zoom input. In particular, a user may zoom in or out on the scene as represented in the generated image.

[0095] Thus, the view processor 111 may provide an indication of a desired view, including typically a view pose and / or a view zoom / viewing angle(s), for a desired view. The image generator 107 may then proceed to generate the image(s) to represent a view of the scene from the desired view pose and with the desired view zoom / viewing angle(s).

[0096] The image is thus generated to represent a given view frustum, which for example may be represented by a view pose and viewing angle(s) (e.g., by a horizontal and vertical viewing angle and / or from a given position and in a given direction).

[0097] The view frustum may at least in some cases / scenarios during use be generated to be a subset of the capture frustum. In particular, the image capture signal may include a view of the scene which corresponds to a relatively large viewing angle / viewport / field of view. The image(s) generated by the image generator 107 may represent a typically smaller viewing angle / viewport / field of view. For example, the generated image may be a zoomed in image compared to the image captured by the image capture device / camera. The view frustum, and thus the viewing angle / field of view, for the generated image may be a subset of the capture frustum, and thus the viewing angle / field of view, of e.g., a camera providing the image capture signal.

[0098] As the view frustum is dynamically varied, e.g., by changing the view pose and / or the view angle, the audio may be automatically adapted to better reflect audio of the selected view frustum. For example, if the viewer / producer zooms in on a specific person in the scene, the level of the audio for that person may be increased in the audio mix relative to other audio sources of the scene which are not present in the current image.

[0099] In some images, the received metadata may include an indication of the capture frustum for the image capture signal / image capture device. For example, the metadata may include a capture pose and field of view / viewing angle for the image capture device / camera capturing the image signal.

[0100] The image generator 107 may use this data in determining the view frustum for the generated image. For example, in cases where there is only a zoom operation / change in the viewing angle, the view frustum may be determined as a frustum having the same origin / view pose as the capture pose indicated by the metadata. The view frustum may however be determined to have a smaller extension, and specifically a smaller viewing angle / field of view. In other embodiments, the view frustum may be determined to have an origin / view pose that is determined from a geometric transformation of the capture pose indicated in the metadata.

[0101] In some applications, the image capture signal may be generated by a static camera that has a static capture pose. However, in many embodiments and applications, the image capture signal may be associated with a dynamically varying capture pose and the image signal receiver 101 may receive dynamically varying image capture pose data for the image capture signal. For example, the image capture signal may be generated by a camera that can be moved along e.g., the side of a sports field and therefore the capture frustum may dynamically change. The position and / or orientation of the camera may be measured / tracked, e.g., based on a global positioning tracking device, and together with the image capture signal may be provided to the render apparatus.

[0102] The image generator 107 may in such a case be arranged to dynamically determine the view frustum for the image dependent on the dynamically varying image capture pose data. For example, the dynamically varying image capture pose data may be used to determine a capture frustum for the image capture signal, and the view frustum may be determined with the constraint that it must be contained in the capture frustum.

[0103] As another example, in many embodiments, the view frustum may be determined to correspond to a frustum having an origin pose corresponding to the current capture pose and with a zoom factor / viewing angle(s) that is determined e.g., based on a user input. For example, if the camera is positioned along the straight of a running track, the view frustum may follow the movement of the camera along the track but with the zoom determined manually by a user.

[0104] In some embodiments, the render apparatus further comprises a viewer attention circuit 113 which is arranged to detect a viewer attention region in the scene. The viewer attention region may be a region in the scene that is estimated to be the focus of attention for the viewer. The viewer attention region is determined as a region within the view frustum, and typically is a subset thereof.

[0105] The audio generator 109 may be arranged to adapt the mixing in dependence on the viewer attention region. Specifically, one or more of the mixing weights (including selection by setting the weights to zero to deselect an audio capture signal) may be set depending on the viewer attention region. As a specific example, the approaches described previously with respect to determining the mixing weights dependent on a distance to the view frustum and / or on an attenuation from the view frustum may be used but with the calculations being specifically based on the viewer attention region rather than on the view frustum as a whole.

[0106] In some embodiments, the viewer attention region may be determined based on image characteristics. For example, for a video signal / application, the viewer attention circuit 113 may be arranged to detect motion in the video and may determine the viewer attention region in dependence on the motion. For example, the viewer attention region may be set to correspond to a region of the scene for which the video signals reflect the highest amount of motion and change (in accordance with any suitable metric and requirement). In such cases, the viewer attention region may be determined as a region of high movement. This may reflect that in many practical applications, viewers will tend to focus on the parts of the scene where there is substantial change, typically corresponding to action. For example, for a football game, the viewer will tend to focus on the ball which is fast moving and with increased player movement typically occurring around the ball. Accordingly, detecting a region of high movement as a viewer attention region is likely to reflect the focus of typical viewers.

[0107] In some embodiments, the viewer attention circuit 113 is arranged to determine the viewer attention region based on detecting viewer properties. In particular, in many embodiments, the viewer attention circuit 113 may comprise a gaze detector 115 which is arranged to detect the gaze of viewer and to determine the viewer attention region from the viewer.

[0108] Such gaze detection may for example be achieved by cameras detecting the eyes and pupil positions for a viewer and therefrom estimating where the viewer is looking. It will be appreciated that a large number of gaze detection algorithms and approaches are known to the skilled person and that any suitable approach may be used without detracting from the invention.

[0109] Thus, in some embodiments, the render apparatus may be arranged to employ some saliency modelling / estimation to detect a viewer attention region. This may for example be based on analyzing motion, e.g., by analyzing motion vectors, to identify areas of the image that are likely to have the attention of the viewer. The viewer attention region in the scene can then be determined by geometric calculations based on the capture pose for the images. Alternatively or additionally, an eye gaze tracker may be used to more directly identify focus areas.

[0110] Combining such attention information with e.g., depth information, may allow the viewer attention circuit 113 to identify a volumetric region of attention in the scene. Thus, the viewer attention region may be a three dimensional region in scene space and may be within the view frustum. The render apparatus may then automatically adapt the audio to allow the user to e.g., better identify audio sources and captured sound from this volumetric region of attention. In particular, audio sources of the viewer attention region can be mixed in more, giving the impression that the viewer "zooms in" on this semantically relevant region of space.

[0111] For instance, when the viewport captures a running player over a backdrop of other players and the public, then the sound of this player can be amplified and the sound of the audience, and if the player has a close microphone, then it is mixed in.

[0112] In some embodiments, the mixing of the plurality of audio capture signals may include an equalization / frequency dependent attenuation / gain and the equalization / frequency dependent attenuation / gain in dependence on the view frustum and the capture region indications. Specifically, the equalization / frequency dependent attenuation / gain for at least one audio capture signal may be dependent on the view frustum and the capture region indication for that audio capture signal.

[0113] For example, in some embodiments, low pass filtering may be applied only to audio capture signal having capture regions with a distance to the view frustum which exceeds a given threshold. Such low pass filtering may remove high frequency components that are typically more distracting than low frequency components and may thus result in an increased emphasis on audio sources within the image while still retaining some audio from audio sources at further distances.

[0114] As another example, for audio capture signal having capture regions within the view frustum, a bandwidth filter emphasizing the main frequencies of human speech may be applied in order to allow a further emphasis on the dialogue speech of humans in the region of the scene that is currently focuses upon. Alternatively or often additionally, audio capture signal having capture regions outside the view frustum may be filtered by a filter that has an inverse filter shape, i.e. a filter that attenuates the main typical frequencies of human speech. This may attenuate speech and dialogue from speakers outside of the image thereby reducing the potential distraction and further emphasizing speakers within the image.

[0115] In some embodiments, one or more of the audio capture devices may be a mobile or moveable audio capture device. In such a case, the capture pose may dynamically change and specifically the capture position may in many cases dynamically change. For example, the audio capture device may be a microphone that can be moved around in the scene such that both the position and orientation of the microphone changes, and accordingly both the capture position and orientation may change dynamically.

[0116] As an example, a microphone may be attached to a movable object / subject, and the position and / or orientation of the microphone may be tracked, e.g., by tracking the position and / or orientation of the object / subject. As a specific example, the microphone may be attached to a person, such as a referee or a player, in the scene and the local audio of that person may be captured. The audio capture signal may in such case be generated by an audio transducer that is attached e.g., to the neck of a person so that its speech is captured through the vibrations in the throat or via bone conduction. The position and / or orientation of the person may be tracked, e.g., using a global positioning system based tracking device, using an RF signal, using video analytics, etc.

[0117] Such an example is often in the field referred to as close miking.

[0118] In many embodiments, the metadata receiver receives dynamically varying capture pose data for one or more audio capture signals / audio capture devices. The audio generator 109 may then dynamically adapt a mixing weight for the audio capture signal in dependence on the dynamically varying capture pose data, and specifically may dynamically adapt the weight based on a spatial relationship between the indicated capture pose and the view frustum. For example, the render apparatus may be arranged to dynamically determine a capture region for the current capture pose and may then proceed to dynamically determine a mixing weight as previously described. In particular, the mixing weight may be increased for a decreasing distance between the capture pose and / or capture region and the view frustum.

[0119] Such an approach may for example result in a scenario where a closed miked player or referee becomes increasingly audible as he moves towards and into the view frustum and less audible as he moves away from the view frustum. In many embodiments, the audio of a moveable microphone may be mixed in or out based on the position with respect to the field of view / view frustum of the generated image.

[0120] There is a current trend towards more close miking in e.g., sports, with the goal of presenting a relevant and pleasant experience that is quite different from the experience of an on-site fan, for example: Racing sports capture the communications of the driver and team members. Formula 1 captures the sound of the engine with microphones at multiple positions and exhaust locations to mix an engine sound that sounds better than in real life. There are plans to outfit soccer players with wearable microphones.

[0121] This is a step beyond collimated microphones. In a soccer stadium it is impossible for the audience to hear the ball clearly over the general noise, but the sound of the ball is still expected by viewers of soccer on TV.

[0122] The render apparatus may be arranged to consider the position (and possibly also the orientation) of a microphone that is attached to a subject (player, driver, etc.) or object (car, ski, etc.) in relation to the view frustum of the generated image when performing the audio mixing to generate the output audio signal.

[0123] Close microphones are typically small and are shaking and can be occluded by clothing or body parts such as hands, so the objective audio quality will generally be much lower than that of fixed microphones, and the audio quality is also fluctuating over time. The close microphones are therefore advantageously only mixed-in when the subject / object is close to the center of the user's attention.

[0124] In some embodiments, the metadata receiver 105 may further be arranged to receive a synchronization indication that is indicative of a set of capture signals which are time synchronized. The set of capture signals may include one or more image capture signals and / or may include one or more audio capture signals. In some cases, the set of capture signals may include capture signals that are synchronized but which may have a fixed time offset between them. For example, one of the synchronized capture signals may be delayed by a fixed amount relative to another one of the synchronized capture signals. In such cases, the synchronization indication may further provide an indication of the time offset.

[0125] The audio generator 109 may be arranged to adapt the mixing operation based on the grouping. For example, in some cases, the mixing may include a timing adjustment to time align different audio capture signals and the audio generator 109 may ensure that the exact same timing adjustment is applied to all audio capture signals.

[0126] In some cases, the synchronization indication may also be used to e.g., select or identify audio capture signals that are included in the audio mixing. For example, all audio capture signals that are indicated to be synchronized to the image capture signal may be included in the mixing as this may be done with low complexity and without requiring any time compensation.

[0127] The synchronization indication may also be used for other purposes, such as adapting the weights in order to achieve a beamforming between different audio capture signals.

[0128] For example, in some scenarios, multiple microphones and cameras may be connected to the same hardware clock and thus may be synchronized. This may be relevant for video processing and audio processing such as beamforming and may advantageously be indicated in the metadata. In particular, the metadata may indicate whether or not a given set of microphones or cameras form a synchronized group, i.e., a group of synchronized capture signals.

[0129] In some embodiments, the metadata may further include an accuracy indication which is indicative of a synchronization accuracy for the set of capture signals. For example, for each set of synchronized capture signals, an accuracy indication may be provided. This may for example indicate whether time stamps are based on a global positioning system, whether synchronization is achieved over wires or RF, whether it is achieved with triggers over a multi-hop local area network, etc.

[0130] The audio mixing may be adapted depending on the accuracy indication. For example, audio capture signals of a set of synchronized signals may only be combined without time compensation if the accuracy is sufficiently high.

[0131] In some embodiments, the metadata may comprise a playlist file which includes the capture region indications (e.g., indicated as pose information). In such cases, the first receiver 101 may be arranged to retrieve audio capture signals from a remote source in dependence on the capture region indications. For example, the render apparatus may determine the view frustum, and in some cases further a viewer attention region, and may from this select audio capture signals that by the capture region indications of the playlist are indicated to be close to the current view frustum. When received, the audio capture signals may be mixed together, e.g., using any of the previously described principles or approaches.

[0132] As an example, the approach may in some cases include adding pose information to playlists for live streaming over Internet.

[0133] Live streaming over the Internet is typically done via HTTP Live Streaming (HLS) or Dynamic Adaptive Streaming over HTTP (DASH). Both standards work with a playlist on a server that can specify different flavors of video and audio streams represented as small segments (chunks) such that the data can be easily retrieved via HTTP requests. In DASH this playlist is an XML format called the Media Presentation Description (MPD).

[0134] In some embodiments, HLS and DASH playlists may be supplemented by spatial information in the form of capture region indications for audio capture signals / audio capture devices. For example, in an HLS Multivariant Playlist, capture region indications may be included by adding capture position and orientation as a pose field (POSE), e.g:

[0135] Each playlist points to separate *.m3u8 playlist file that each correspond with an audio / video playlist for a different pose as parametrized by the POSE variable. The POSE variable can be used by the client to retrieve the correct streams for rendering and to also perform rendering. The POSE variable can for example specify a translation vector and an orientation quaternion.

[0136] For example, the live streaming (sliding window) playlist for the first pose can be specified as: #EXTM3U #EXT-X-TARGETDURATION: #EXT-X-VERSION:4 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:5.0, fileSequence500.ts #EXTINF:5.0, fileSequence501.ts #EXTINF:5.0, fileSequence502.ts #EXTINF:5.0, fileSequence503.ts #EXTINF:5.0, fileSequence504.ts This is a standard HLS playlist that specifies the most recent five transport stream files (video and audio). Note that such a syntax for pose (position and orientation) also applies to the similar DASH standard. Further, the pose can be specified for audio and video streams separately.

[0137] It may in many embodiments be advantageous to represent a pose as a rotation quaternion. Advantages may include: The definition is with respect to the spatial coordinates x 0 , x 1 , x 2 , whatever they are, making the orientations unambiguous, (unlike Euler angles) There is no gimbal point (similar to Euler angles) A 3-D rotation has three freedoms which is evident from a unit quaternion or Euler angles, but not from a rotation matrix

[0138] In the described approach a textual format may be used. An approach is to use a scaled unit quaternion sq = sq w + isq x + jsq y + ksq z , quantized to be represented by four integers. The sq w component is an unsigned integer, and the other components are signed. The orientation can be decoded by parsing the string and normalizing the quaternion:Example 1:

[0139] Forward (neutral) orientation: "1+0+0+0" decodes to q = 1.Example 2:

[0140] Some arbitrary rotation: "12-34+56-79" decodes to q = 12 − 34 i + 56 j − 79 k 12 2 + 32 2 + 56 2 + 79 2 ≈ 0.12 + 0.33 i + 0.65 j + 0.76 k

[0141] A standard may dictate a maximum string length for orientations.

[0142] Since the q = 1 angle is common, it may be defined that trailing zeros can be omitted, giving only " 1" to represent q = 1.

[0143] To indicate that an audio stream is from a close mike, an indication can be used in the playlist: e.g., CLOSE =1, and the pose of the microphone in the playlist may be the last known or predicted location.

[0144] In some cases, there may be a timed metadata track that streams the position or pose of one or more non-stationary microphones such that the client can request upcoming segments of the audio track when the microphones come close to the view frustum. For example: when a player is about to walk into the view frustum, the audio of the microphone of the player needs to be available in the client. When the viewport moves away, or the player moves out of the viewport, no additional segments of that audio track are needed by the client.

[0145] FIG. 3 is a block diagram illustrating an example processor 300 according to embodiments of the disclosure. Processor 300 may be used to implement one or more processors implementing an apparatus as previously described or elements thereof (including in particular the beamformers as described). Processor 300 may be any suitable processor type including, but not limited to, a microprocessor, a microcontroller, a Digital Signal Processor (DSP), a Field ProGrammable Array (FPGA) where the FPGA has been programmed to form a processor, a Graphical Processing Unit (GPU), an Application Specific Integrated Circuit (ASIC) where the ASIC has been designed to form a processor, or a combination thereof.

[0146] The processor 300 may include one or more cores 302. The core 302 may include one or more Arithmetic Logic Units (ALU) 304. In some embodiments, the core 302 may include a Floating Point Logic Unit (FPLU) 306 and / or a Digital Signal Processing Unit (DSPU) 308 in addition to or instead of the ALU 304.

[0147] The processor 300 may include one or more registers 312 communicatively coupled to the core 302. The registers 312 may be implemented using dedicated logic gate circuits (e.g., flip-flops) and / or any memory technology. In some embodiments the registers 312 may be implemented using static memory. The register may provide data, instructions and addresses to the core 302.

[0148] In some embodiments, processor 300 may include one or more levels of cache memory 310 communicatively coupled to the core 302. The cache memory 310 may provide computer-readable instructions to the core 302 for execution. The cache memory 310 may provide data for processing by the core 302. In some embodiments, the computer-readable instructions may have been provided to the cache memory 310 by a local memory, for example, local memory attached to the external bus 316. The cache memory 310 may be implemented with any suitable cache memory type, for example, Metal-Oxide Semiconductor (MOS) memory such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), and / or any other suitable memory technology.

[0149] The processor 300 may include a controller 314, which may control input to the processor 300 from other processors and / or components included in a system and / or outputs from the processor 300 to other processors and / or components included in the system. Controller 314 may control the data paths in the ALU 304, FPLU 306 and / or DSPU 308. Controller 314 may be implemented as one or more state machines, data paths and / or dedicated control logic. The gates of controller 314 may be implemented as standalone gates, FPGA, ASIC or any other suitable technology.

[0150] The registers 312 and the cache 310 may communicate with controller 314 and core 302 via internal connections 320A, 320B, 320C and 320D. Internal connections may be implemented as a bus, multiplexer, crossbar switch, and / or any other suitable connection technology.

[0151] Inputs and outputs for the processor 300 may be provided via a bus 316, which may include one or more conductive lines. The bus 316 may be communicatively coupled to one or more components of processor 300, for example the controller 314, cache 310, and / or register 312. The bus 316 may be coupled to one or more components of the system.

[0152] The bus 316 may be coupled to one or more external memories. The external memories may include Read Only Memory (ROM) 332. The external memory may include Random Access Memory (RAM) 333. RAM 333 may be a static RAM, battery backed up static RAM, Dynamic RAM (DRAM) or any other suitable technology. The external memory may include Electrically Erasable Programmable Read Only Memory (EEPROM) 335. The external memory may include Flash memory 334. The External memory may include a magnetic storage device such as disc 336. In some embodiments, the external memories may be included in a system.

[0153] It will be appreciated that the above description for clarity has described embodiments of the invention with reference to different functional circuits, units and processors. However, it will be apparent that any suitable distribution of functionality between different functional circuits, units or processors may be used without detracting from the invention. For example, functionality illustrated to be performed by separate processors or controllers may be performed by the same processor or controllers. Hence, references to specific functional units or circuits are only to be seen as references to suitable means for providing the described functionality rather than indicative of a strict logical or physical structure or organization.

[0154] The invention can be implemented in any suitable form including hardware, software, firmware or any combination of these. The invention may optionally be implemented at least partly as computer software running on one or more data processors and / or digital signal processors. The elements and components of an embodiment of the invention may be physically, functionally and logically implemented in any suitable way. Indeed the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the invention may be implemented in a single unit or may be physically and functionally distributed between different units, circuits and processors.

[0155] Although the present invention has been described in connection with some embodiments, it is not intended to be limited to the specific form set forth herein. Rather, the scope of the present invention is limited only by the accompanying claims. Additionally, although a feature may appear to be described in connection with particular embodiments, one skilled in the art would recognize that various features of the described embodiments may be combined in accordance with the invention. In the claims, the term comprising does not exclude the presence of other elements or steps.

[0156] Furthermore, although individually listed, a plurality of means, elements, circuits or method steps may be implemented by e.g. a single circuit, unit or processor. Additionally, although individual features may be included in different claims, these may possibly be advantageously combined, and the inclusion in different claims does not imply that a combination of features is not feasible and / or advantageous. Also the inclusion of a feature in one category of claims does not imply a limitation to this category but rather indicates that the feature is equally applicable to other claim categories as appropriate. Furthermore, the order of features in the claims do not imply any specific order in which the features must be worked and in particular the order of individual steps in a method claim does not imply that the steps must be performed in this order. Rather, the steps may be performed in any suitable order. In addition, singular references do not exclude a plurality. Thus, references to "a", "an", "first", "second" etc. do not preclude a plurality. Reference signs in the claims are provided merely as a clarifying example shall not be construed as limiting the scope of the claims in any way.

Claims

1. An apparatus for generating an output audio signal providing an audio representation of a scene, the apparatus comprising: a first receiver (103) arranged to receive a plurality of audio capture signals, each audio capture signal comprising audio of the scene captured by one or more audio capture devices; a second receiver (101) arranged to receive at least one image capture signal, each image capture signal comprising at least one image representing a capture frustum of the scene; a third receiver (105) arranged to receive metadata comprising capture region indications for at least some of the audio capture signals, a capture region indication for an audio capture signal being indicative of a region of audio capture of the scene for the audio capture signal; an image generator (107) arranged to generate a view image of the scene for a view frustum from the at least one image capture signal; and an audio generator (109) arranged to generate the output audio signal including mixing the plurality of audio capture signals in dependence on the view frustum and the capture region indications.

2. The apparatus of claim 1 wherein the audio generator (109) is arranged to increase a mixing weight for a first audio capture signal for an increasing proximity of a capture region of the first audio capture signal to the view frustrum.

3. The apparatus of claim 1 or 2 wherein the audio generator (109) is arranged to select a subset of audio capture signals included in the mixing, the audio generating circuit being arranged to select between including and excluding a given audio capture signal in the mixing in dependence on a spatial relationship between the view frustum and a region of audio capture for the given audio capture signal.

4. The apparatus of any previous claim wherein the metadata comprises a property indication for at least one audio capture signal, the property indication being indicative of at least one of: a directional capture sensitivity for the at least one audio capture signal; a capture pose for the at least one audio capture signal; a capture device property for an audio capture device capturing the at least one audio capture signal; or a capture range for the at least one audio capture signal; and the audio generator (109) is arranged to adapt the mixing in dependence on the property indication.

5. The apparatus of any previous claim further comprising: a first circuit (113) arranged to determine a viewer attention region in the scene; and wherein the audio generator (109) is arranged to adapt the mixing in dependence on the viewer attention region.

6. The apparatus of claim 5 further comprising a gaze detector (115) arranged to detect a gaze of viewer, and wherein the first circuit is arranged to determine the viewer attention region in dependence on the gaze of the viewer.

7. The apparatus of any previous claim comprising a view circuit (111) arranged to provide a desired view of the scene and wherein the image generator (107) is arranged to dynamically select the view frustum as a subset of the capture frustum for the at least one image capture signal in response to changes in the desired view.

8. The apparatus of any previous claim wherein the metadata includes an indication of the capture frustum for the at least one image capture signal, and the image generator (107) is arranged to determine the view frustum in dependence on the capture frustum.

9. The apparatus of any previous claim wherein the first receiver (103) is further arranged to receive dynamically varying capture pose data for at least one audio capture signal of the plurality of audio capture signals; and the audio generator (109) is arranged to dynamically adapt a mixing weight for the at least one audio capture signal in dependence on the dynamically varying capture pose data.

10. The apparatus of any previous claim wherein the second receiver (101) is arranged to receive dynamically varying image capture pose data for the at least one image capture signal; and wherein the image generator (107) is arranged to determine the view frustum in dependence on the dynamically varying image capture pose data.

11. The apparatus of any previous claim wherein the third receiver (105) is arranged to receive a synchronization indication, the synchronization indication indicating a set of capture signals of the plurality of audio capture signals and the at least one image capture signal being time synchronized with each other; and wherein the audio generator (109) is arranged to adapt the mixing dependent on the synchronization indication.

12. The apparatus of claim 11 wherein the third receiver (105) is arranged to receive an accuracy indication indicative of a synchronization accuracy for the set of capture signals, and the audio generating circuit is arranged to adapt the mixing dependent on the synchronization accuracy.

13. The apparatus of any previous claim wherein the metadata comprises a playlist file comprising the capture region indications and the first receiver is arranged to retrieve audio capture signals from a remote source in dependence on the capture region indications.

14. A method of generating an output audio signal providing an audio representation of a scene, the method comprising: receiving a plurality of audio capture signals, each audio capture signal comprising audio of the scene captured by one or more audio capture devices; receiving at least one image capture signal, each image capture signal comprising at least one image representing a capture frustum of the scene; receiving metadata comprising capture region indications for at least some of the audio capture signals, a capture region indication for an audio capture signal being indicative of a region of audio capture of the scene for the audio capture signal; generating a view image of the scene for a view frustum from the at least one image capture signal; and generating the output audio signal including mixing the plurality of audio capture signals in dependence on the view frustum and the capture region indications.

15. A computer program product comprising computer program code means adapted to perform all the steps of the method of claim 14 when said program is run on a computer.

Images