Generate audio output signal

Abstract

An apparatus, method, and computer program are described, including: capturing spatial audio data during an image capture process; determining the orientation of an image capture device during the capture of the spatial audio data; generating an audio focus signal from the captured spatial audio data (wherein the audio focus signal is focused in the image capture direction of the image capture device); generating modified spatial audio data (e.g., by modifying the captured spatial audio data to compensate for changes in orientation during the capture of the spatial audio data); and generating an audio output signal from a combination of the audio focus signal and the modified spatial audio data.

Description

Technical Field

[0001] The present description relates to audio output signals associated with spatial audio. Background Art

[0002] Arrangements for capturing spatial audio are known. However, there is still a need for further development in this area. Summary of the invention

[0003] In a first aspect, the present specification provides an apparatus (e.g., an imaging device, such as a mobile phone including a camera), comprising: a component for capturing spatial audio data during an image capture process; a component for determining the orientation of the apparatus during spatial audio data capture; a component for generating an audio focus signal (e.g., a mono audio signal) from the captured spatial audio data, wherein the audio focus signal is focused on the image capture direction of the apparatus; a component for generating modified spatial audio data, wherein generating the modified spatial audio data includes: modifying the captured spatial audio data to compensate for one or more changes in the orientation of the apparatus during spatial audio data capture; and a component for generating an audio output signal from a combination of the audio focus signal and the modified spatial audio data. Some examples include: a component for capturing a visual image (e.g., a still image or a moving image) of an object or scene.

[0004] In some examples, spatial audio data is captured from a start time (e.g., when a photo application is launched) that is at or before the start of the image capture process to an end time that is at or after the end of the image capture process.

[0005] In some examples, the means for generating modified spatial audio data may be configured to compensate for the one or more changes in orientation of the apparatus by rotating the captured spatial audio data to counteract the determined change in orientation of the apparatus.

[0006] In some examples, the spatial audio data may be parameterized audio data.The means for generating modified spatial audio data may be configured to generate the modified spatial audio data by modifying parameters of the parameterized audio data.

[0007] In some examples, the means for generating the audio focus signal may include one or more beamforming arrangements.

[0008] In some examples, the means for generating the audio focus signal may be configured to emphasize audio (eg, captured spatial audio data) in an image capture direction of the device.

[0009] In some examples, the means for generating the audio focus signal may be configured to attenuate audio (eg, captured spatial audio data) in directions other than an image capture direction of the apparatus.

[0010] In some examples, the means for generating the audio output signal may be configured to generate the audio output signal based on a weighted sum of the audio focus signal and the modified spatial audio data.

[0011] In some examples, components for determining orientation of the device include one or more sensors (eg, one or more accelerometers and / or one or more gyroscopes).

[0012] The component may include: at least one processor; and at least one memory including computer program code, the at least one memory and the computer program code configured to: with the at least one processor, cause execution of the apparatus.

[0013] In a second aspect, the present specification describes a method comprising: capturing spatial audio data during an image capture process; determining the orientation of an image capture device during the capture of the spatial audio data; generating an audio focus signal (e.g., a mono audio signal) from the captured spatial audio data, wherein the audio focus signal is focused on the image capture direction of the image capture device; generating modified spatial audio data, wherein generating the modified spatial audio data comprises: modifying the captured spatial audio data to compensate for one or more changes in the orientation of the image capture device during the capture of the spatial audio data; and generating an audio output signal from a combination of the audio focus signal and the modified spatial audio data.

[0014] In some examples, the method may also include capturing a visual image of the object or scene.

[0015] In some examples, spatial audio data is captured from a start time (e.g., when a photo application is launched) that is at or before the start of the image capture process to an end time that is at or after the end of the image capture process.

[0016] In some examples, the modified spatial audio data may be generated by compensating for the one or more changes in orientation of the image capture device. Compensating for the changes in orientation of the image capture device may include rotating the captured spatial audio data to counteract the determined changes in orientation of the apparatus.

[0017] In some examples, the spatial audio data may be parametric audio data. The modified spatial audio data may be generated by modifying parameters of the parametric audio data.

[0018] In some examples, the audio focus signal may be generated using one or more beamforming arrangements.

[0019] In some examples, generating the audio focus signal may include emphasizing audio (eg, captured spatial audio data) in an image capture direction of the image capture device.

[0020] In some examples, generating the audio focus signal may include attenuating audio (eg, captured spatial audio data) in directions other than an image capture direction of the image capture device.

[0021] In some examples, the audio output signal may be generated based on a weighted sum of the audio focus signal and the modified spatial audio data.

[0022] In some examples, the orientation of the image capture device is determined using one or more sensors (eg, one or more accelerometers and / or one or more gyroscopes).

[0023] In a third aspect, the specification describes an apparatus configured to perform any method as described with reference to the second aspect.

[0024] In a fourth aspect, the present specification describes computer readable instructions which, when executed by a computing device, cause the computing device to perform any of the methods as described with reference to the second aspect.

[0025] In a fifth aspect, the present specification describes a computer program comprising instructions for causing a device to perform at least the following operations: capture spatial audio data during an image capture process; determine the orientation of an image capture device during spatial audio data capture; generate an audio focus signal (e.g., a mono audio signal) from the captured spatial audio data, wherein the audio focus signal is focused on the image capture direction of the image capture device; generate modified spatial audio data, wherein generating the modified spatial audio data comprises: modifying the captured spatial audio data to compensate for one or more changes in the orientation of the image capture device during spatial audio data capture; and generating an audio output signal from a combination of the audio focus signal and the modified spatial audio data.

[0026] In a sixth aspect, the present specification describes a computer-readable medium (such as a non-transitory computer-readable medium) comprising program instructions stored thereon to perform at least the following operations: capture spatial audio data during an image capture process; determine the orientation of an image capture device during spatial audio data capture; generate an audio focus signal (e.g., a mono audio signal) from the captured spatial audio data, wherein the audio focus signal is focused on the image capture direction of the image capture device; generate modified spatial audio data, wherein generating the modified spatial audio data comprises: modifying the captured spatial audio data to compensate for one or more changes in the orientation of the image capture device during spatial audio data capture; and generating an audio output signal from a combination of the audio focus signal and the modified spatial audio data.

[0027] In a seventh aspect, the present specification describes an apparatus comprising: at least one processor; and at least one memory comprising computer program code, which, when executed by the at least one processor, causes the apparatus to: capture spatial audio data during an image capture process; determine the orientation of an image capture device during spatial audio data capture; generate an audio focus signal (e.g., a mono audio signal) from the captured spatial audio data, wherein the audio focus signal is focused on the image capture direction of the image capture device; generate modified spatial audio data, wherein generating the modified spatial audio data comprises: modifying the captured spatial audio data to compensate for one or more changes in the orientation of the image capture device during spatial audio data capture; and generate an audio output signal from a combination of the audio focus signal and the modified spatial audio data.

[0028] In an eighth aspect, the present specification describes an apparatus comprising: a first audio module configured to capture spatial audio data during an image capture process; a first control module configured to determine the orientation of an image capture device during spatial audio data capture; a second control module configured to generate an audio focus signal (e.g., a mono audio signal) from the captured spatial audio data, wherein the audio focus signal is focused on the image capture direction of the image capture device; a second audio module configured to generate modified spatial audio data, wherein generating the modified spatial audio data comprises modifying the captured spatial audio data to compensate for one or more changes in the orientation of the image capture device during spatial audio data capture; and an audio output module configured to generate an audio output signal from a combination of the audio focus signal and the modified spatial audio data. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Example embodiments will now be described by way of non-limiting example with reference to the following schematic diagrams, in which:

[0030] Figures 1 to 4 is a block diagram of a system according to an example embodiment;

[0031] Figure 5A , Figure 5B and Figure 5C is a block diagram of a system according to an example embodiment;

[0032] Figure 6 is a flow chart illustrating an algorithm according to an example embodiment;

[0033] Figure 7 , Figure 8 , Fig. 9A , Fig. 9B , Fig. 9C and Figures 10 to 12 is a block diagram of a system according to an example embodiment; and

[0034] Fig.13A and Fig. 13B Tangible media storing computer readable codes, which when executed by a computer, perform operations according to the embodiments, are shown, respectively, as a removable memory unit and a compact disk (CD). DETAILED DESCRIPTION

[0035] In the present specification and drawings, like reference numerals refer to like elements throughout.

[0036] Figure 1 1 is a block diagram of a system according to an example embodiment, which is generally indicated by reference numeral 10. System 10 includes a focus object 12, an image capture device 14, and background objects 16. Focus object 12 can move, for example, in a left direction as shown by a dashed arrow. Focus object 12 can be any one or more objects in an image capture direction of image capture device 14, so that image capture device 14 can be used to capture one or more images and / or videos of focus object 12. Background object 16 can represent any one or more background objects that may exist around image capture device 14 and / or focus object 12.

[0037] It is to be understood that the focused object 12 moving in the left direction is merely an example at any instant in time, such that the focused object 12 may be moving in any direction, or may also be stationary. Furthermore, the "image capture direction" of the image capture device 14 may be any direction in which the image capture device 14 is visible (not just in front of the device, as in the case of a Figure 1 shown in ).

[0038] In an example embodiment, when the image capture device 14 is being used to capture an image, the image capture device 14 also captures spatial audio data. The spatial audio data may include focused audio from the focused object 12 and background audio from the background objects 16. If the focused object 12 is moving, the orientation (e.g., image capture direction) of the image capture device 14 may be changed to focus the focused object 12 as the focus of the image capture (e.g., in the center of the image capture scene). As the orientation changes, the captured spatial audio data may also change according to the change in distance or direction of the focused object 12 and / or background objects 16 relative to the image capture device 14.

[0039] In an example embodiment, the focused object 12 is a moving car, such as a car in a race, and the image capture device 14 is a camera or mobile device for capturing images and / or video of the car. The image capture device 14 can be, for example, held by a viewer or can be attached to a wall or a tripod. The background object 16 can represent a crowd watching the race. Therefore, the spatial audio data can include sounds from the car as well as the crowd. However, when capturing images and / or video of the car, the sounds from the crowd will be considered background audio, while the sounds from the car will be considered focused audio.

[0040] It is to be understood that the focused object 12 and the background object 16 are example representations and are not limited to a single object, such that they may be any one or more objects or scenes. The focused object 12 may be any object and / or scene in the image capture direction. The background object 16 may be any object and / or scene in any direction.

[0041] Figures 2 to 4 is a block diagram of example systems, generally indicated by reference numerals 20, 30, and 40, respectively. Systems 20, 30, and 40 include focus object 12, image capture device 14, and background object 16 as described above.

[0042] System 20( Figure 2 ) includes a focused object 12, an image capture device 14, and background objects 16 moving in a left direction shown by dashed arrow 22. The orientation of image capture device 14 relative to background objects 16 at a first time instance (e.g., at a start time) may be shown by angle 21. The image capture direction may be shown by direction 26, and any (multiple) directions different from the image capture direction (for purposes of modifying spatial audio) may be shown by direction 27 (by way of example). As focused object 12 moves in the direction of dashed arrow 22, the orientation of image capture device 14 may be changed (e.g., by rotation) in the direction of dashed arrow 23 so that focused object 12 maintains focus of the image capture scene.

[0043] System 30( Figure 3 ) includes focused object 12, image capture device 14, and background objects 16 still moving in a left direction (as indicated by dashed arrow 32). The orientation of image capture device 14 at a second time instance relative to background objects 16 may be shown by angle 34. The image capture direction may be shown by direction 36 (by way of example), and any direction(s) different from the image capture direction may be shown by direction 37. As focused object 12 moves in the direction of dashed arrow 32, the orientation of image capture device 14 may be changed (e.g., by rotating) in the direction of dashed arrow 33 so that focused object 12 remains in focus of the image capture scene.

[0044] System 40( Figure 4 ) includes focus object 12, image capture device 14, and background object 16. The orientation of image capture device 14 relative to background object 16 at a third time instance (e.g., end time) may be shown by angle 44. The image capture direction may be shown by direction 46, and any (multiple) directions different from the image capture direction may be shown by direction 47 (by way of example).

[0045] Figure 5A , Figure 5B and Figure 5C 1 is a block diagram of a system, generally indicated by reference numerals 50A, 50B, and 50C, respectively, according to an example embodiment. Systems 50A, 50B, and 50C illustrate the manner in which the apparent direction of background audio may change when the orientation of image capture device 14 is changed so as to focus on focus object 12. The change in the apparent direction of the background audio may give the listener the impression that background object 16 is moving, which may not be desirable (e.g., if background object 16 is stationary and focus object 12 is moving).

[0046] At a first time instance (e.g., at a start time) shown by system 50A, the positions of the focused object, image capture device, and background objects are illustrated by focused object 12a, image capture device 14a, and background object 16a. This is the system 20 described above ( Figure 2 ) arrangement.

[0047] When the focused object moves in the left direction, the orientation of the image capture device may change (e.g., rotate to the left). At a second time instance shown by system 50B, the positions of the focused object, image capture device, and background objects are illustrated by focused object 12b, image capture device 14b, and background object 16b. This is system 30 (described above). Figure 3It can be seen that the direction of the background object 16b relative to the image capture device 14b is different at the first time instance and the second time instance.

[0048] At a third time instance shown by system 50C (the focused object continues to move in the left direction), the positions of the focused object, image capture device, and background objects are illustrated by focused object 12c, image capture device 14c, and background object 16c. This is system 40 (described above) Figure 4 It can be seen that the direction of the background object 16c relative to the image capture device 14c is different at the first time instance, the second time instance and the third time instance.

[0049] Figure 6 is a flow chart of an algorithm according to an example embodiment, generally indicated by reference numeral 60. Figures 2 to 4 as well as FIG. 5A to FIG. 5C To describe Figure 6 .

[0050] At operation 61, spatial audio data is captured during an image capture process, such as using image capture device 14. Spatial audio data may be captured from both focus object 12 and background objects 16.

[0051] At operation 62, an orientation of an apparatus, such as image capture device 14, is determined during spatial audio data capture. Orientation may be determined using one or more sensors, such as accelerometer(s) or gyroscope(s). For example, in systems 20, 30, and 40, the orientation of image capture device 14 is shown to change in a counterclockwise direction (from direction 26 (angle 21) to direction 36 (angle 34), and then to direction 46 (angle 44)).

[0052] At operation 63, an audio focus signal is generated. An audio focus signal is generated from the captured spatial audio data and is focused on the image capture direction. For example, the audio focus signal is focused on direction 26 at a first time instance, on direction 36 at a second instance, and on direction 46 at a third instance. As further described below, operation 63 may be implemented using a beamforming arrangement.

[0053] At operation 64, modified spatial audio data is generated. The modified spatial audio data is generated by modifying the spatial audio data for compensating for a change in orientation during capture of the spatial audio data.

[0054] At operation 65, an audio output signal is generated from the combination of the audio focus signal and the modified spatial audio data.

[0055] In one example embodiment, during the image capture process, in addition to capturing spatial audio data, a visual image of an object or scene may also be captured.

[0056] In an example embodiment, in operation 65, an audio output signal is generated based on a weighted sum of the audio focus signal (generated in operation 63) and the modified spatial audio data (generated in operation 64).

[0057] In an example embodiment, the audio focus signal can be focused in the image capture direction by panning the audio focus signal in the direction of the focus object, the direction of the focus object being the same as the direction from which the focus object is heard in the spatial audio data. Likewise, in the audio output signal, audio from a moving focus object is perceived as coming from a moving object and changes based on the actual direction of movement of the focus object. In the audio output signal, any audio from background objects is perceived as coming from stationary objects and is configured to be perceived as remaining constant throughout the image capture process.

[0058] In an exemplary embodiment, spatial audio data is captured in operation 61 from a start time (e.g., at a first time instance) to an end time, the start time being at or before the start of the image capture process, and the end time being at or after the end of the image capture process. For example, in a mobile phone with a camera, the image capture process and spatial audio data capture may begin when the camera application is active. The image capture process may end when the user takes a photo. Spatial audio data may not be captured, for example, until a set time after the photo is taken, until the camera application is closed, or until the phone screen is turned off. In another example, the image capture process and spatial audio data capture may begin when video capture is started on the camera application, and the image capture process and spatial audio data capture may end when video capture is ended.

[0059] In an example embodiment, at operation 64, the spatial audio data is modified to compensate for the change in orientation by rotating the captured spatial audio data to counteract the determined change in orientation. For example, in system 20, the direction (relative to image capture device 14) of spatial audio data corresponding to background object 16 (i.e., any spatial audio data that does not include an audio focus signal) may be shown by direction 27. Figure 7 9 describes in more detail the manner in which the captured spatial audio data may be rotated to counteract the determined change in orientation.

[0060] Figure 7is a block diagram of a system according to an example embodiment, which is generally indicated by reference numeral 70. System 70 is similar to system 30 described above. In system 70, the direction (relative to image capture device 14) of spatial audio data corresponding to background object 16 (i.e., any spatial audio data that does not include an audio focus signal) can be shown by direction 77. However, the change in orientation compared to system 20 is compensated (shown by angle 74) by rotating the direction from direction 77 to direction 78 to offset the determined change in orientation. This can allow a listener to perceive the modified spatial audio data as coming from direction 78, and to perceive the location of background object 16 as being at background object representation 75. The captured spatial audio data can be rotated so that angle 71 between image capture device 14 and background object representation 75 is substantially the same as angle 21 of system 20 described above. Thus, a listener will perceive the background object as being stationary because angle 71 is the same as angle 21.

[0061] Figure 8 8 is a block diagram of a system according to an example embodiment, which is generally indicated by reference numeral 80. System 80 is similar to system 40 described above. In system 80, the direction (relative to image capture device 14) of spatial audio data corresponding to background object 16 (i.e., any spatial audio data that does not include an audio focus signal) can be shown by direction 87. However, the orientation change (shown by angle 84) is compensated by rotating the direction from direction 87 to direction 88 to offset the determined change in orientation. This can allow a listener to perceive the modified spatial audio data as coming from direction 88, and to perceive the location of the background object at background object representation 85. The captured spatial audio data can be rotated so that the angle 81 between image capture device 14 and background object representation 85 is substantially the same as angle 21 described above. Therefore, the listener will perceive the background object as being stationary because angle 81 is the same as angle 21.

[0062] Fig. 9A , Fig. 9B and Fig. 9Cis a block diagram of a system according to an example embodiment, generally indicated by reference numerals 90A, 90B, and 90C. Systems 90A, 90B, and 90C illustrate modified spatial audio data and audio focus signals at a first time instance, a second time instance, and a third time instance, respectively, from respective perspectives, such that a focused object is at the center of an image capture scene. Similar to systems 50A, 50B, and 50C, the positions of a focused object, an image capture device, and a background image are illustrated by focused objects 12a to 12c, image capture devices 14a to 14c, and background objects 16a to 16c at the first time instance, the second time instance, and the third time instance. At a first time instance (e.g., at a start time) illustrated by system 90A, the positions of a focused object, an image capture device, and background objects are illustrated by focused object 12a, image capture device 14a, and background object 16a. This is system 20( described above). Figure 2 ) and System 50A( Figure 5A ). In a second time instance, shown by system 90B, the direction of the spatial audio data is rotated so that the background object is perceived (by the listener) as being at position 91 (the same position as position 16a). In a third time instance, shown by system 90C, the direction of the spatial audio data is rotated so that the background object is perceived (by the listener) as being at position 92 (again, the same position as position 16a). The audio focus signal is focused on the image capture direction (e.g., the example direction of focus object 12 and image capture device 14) shown by arrows 93a, 93b, and 93c.

[0063] Fig.10 is a block diagram of a system according to an example embodiment, which is generally indicated by reference numeral 100. System 100 includes an image capture module 101, a spatial audio capture module 102, a controller 103, an audio modification module 104, and a memory module 105.

[0064] The image capture module 101 is used to capture images (eg, photographic images and / or video images). During the image capture process, the spatial audio capture module 102 captures spatial audio data. The captured image data and the captured audio data are provided to the controller 103.

[0065] The controller 103 determines the orientation of the device during the spatial audio data capture, and uses the audio modification module 104 to modify the captured audio based on the orientation data (as described in detail above) to generate modified spatial audio data by modifying the captured spatial audio to compensate for the change in orientation during the spatial audio data capture. Similarly, the audio modification module 104 generates an audio focus signal from the captured spatial audio data under the control of the controller 103, wherein the audio focus signal is focused on the image capture direction of the image capture module 101.

[0066] The memory 105 may be used to store one or more of the captured spatial audio data, the modified spatial audio data, and the audio focus signal.

[0067] Finally, the controller 103 is used to generate an audio output signal from the combination of the audio focus signal and the modified spatial audio data (eg by retrieving said data from the memory 105).

[0068] In an example embodiment, the spatial audio data captured in operation 61 of algorithm 60 is parameterized audio data. For example, the parameterized audio data can be DirAC or Nokia's OZO Audio. When capturing parameterized audio data, multiple spatial parameters (representing multiple properties of the captured audio) can be analyzed for each time-frequency block of the captured multi-microphone signal. One or more parameters may include, for example, a direction of arrival (DOA) parameter and / or a ratio parameter (such as diffusion for each time-frequency block). The spatial audio data can be represented by spatial metadata and a transmitted audio signal. The transmitted audio signal and the spatial metadata can be used to synthesize a sound field. The sound field can produce an auditory perception such that a listener will perceive that his / her head / ear is located at a certain position of the image capture device.

[0069] In an example embodiment, modified spatial audio data may be generated in operation 64 by modifying one or more parameters in the parameterized audio data to rotate the captured spatial audio data to offset the determined change in the orientation of the device. For example, the one or more parameters may be modified by rotating the sound field of the spatial audio data. The sound field may be rotated by rotating one or more DOA parameters accordingly.

[0070] In an example embodiment, the spatial audio data captured in operation 61 of algorithm 60 is Ambisonics audio, such as first-order Ambisonics (FOA) or higher-order Ambisonics (HOA). The spatial audio data may be represented by a transmitted audio signal. The transmitted audio signal may be used to synthesize a sound field. The sound field may produce an auditory perception such that a listener would perceive that his / her head / ear is located at a certain position of the image capture device.

[0071] In an example embodiment, modified spatial audio data may be generated by modifying the Ambisonics audio data using a rotation matrix in operation 64. The Ambisonics audio may be modified using a rotation matrix so that a sound field synthesized from the modified audio data causes a listener to perceive that the sound sources have rotated around the listener.

[0072] In an example embodiment, one or more beamforming arrangements may be used to generate an audio focus signal in operation 63. For example, a beamformer (such as a delay and beamformer) may be used for one or more beamforming arrangements. Alternatively or additionally, parametric spatial audio processing may be used to generate an audio focus signal (beamformed output) by emphasizing (or extracting) audio from a focused object from the complete spatial audio data.

[0073] In an example embodiment, generating the audio focus signal may be configured to emphasize audio (e.g., captured spatial audio data) in the image capture direction of the device. The audio focus signal may also be configured to attenuate audio (e.g., captured spatial audio data) in directions other than the image capture direction. For example, in systems 90A, 90B, and 90C, the audio focus signal may be configured to emphasize audio in the image capture direction (such as directions 93a, 93b, and / or 93c). Any audio received from a direction other than the image capture direction (e.g., from background objects) may be attenuated.

[0074] By way of example, Fig.11 is a block diagram of a system according to an example embodiment, which is generally indicated by reference numeral 110. System 110 includes focus object 12 and image capture device 14 as described above. System 110 also shows a beamforming arrangement 112 showing an audio focus direction of image capture device 14.

[0075] For the sake of completeness, Fig.12 is a schematic diagram of components of one or more of the previously described example embodiments, which are hereinafter collectively referred to as a processing system 300. The processing system 300 may be, for example, an apparatus as recited in the following claims.

[0076] The processing system 300 may have a processor 302, a memory 304 tightly coupled to the processor and consisting of a RAM 314 and a ROM 312, and optionally a user input 310 and a display 318. The processing system 300 may include one or more network / device interfaces 308 to connect to a network / device, such as a modem, which may be wired or wireless. The interface 308 may also operate as a connection with other devices (such as devices / devices that are not network-side devices). Therefore, direct connections between devices / devices without network involvement are possible.

[0077] The processor 302 is connected to each of the other components to control the operation thereof.

[0078] The memory 304 may include non-volatile memory, such as a hard disk drive (HDD) or a solid state drive (SSD). The ROM 312 of the memory 304 also stores an operating system 315, among other things, and may store software applications 316. The RAM 314 of the memory 304 is used by the processor 302 to temporarily store data. The operating system 315 may contain code that implements various aspects of the algorithm 60 described above when executed by the processor. Note that in the case of a small device / apparatus, the memory will be best suited for small size usage, i.e., a hard disk drive (HDD) or a solid state drive (SSD) is not always used.

[0079] Processor 302 may take any suitable form. For example, it may be a microcontroller, multiple microcontrollers, a processor, or multiple processors.

[0080] The processing system 300 may be a stand-alone computer, server, console, or a network thereof. The processing system 300 and the required structural parts may be completely within a device / apparatus, such as an IoT device / apparatus, i.e., embedded in a very small size.

[0081] In some example embodiments, the processing system 300 may also be associated with external software applications. These external software applications may be applications stored on a remote server device / apparatus and may be run partially or exclusively on the remote server device / apparatus. These applications may be referred to as cloud-hosted applications. The processing system 300 may communicate with the remote server device / apparatus to utilize the software applications stored on the remote server device / apparatus.

[0082] Fig.13A and Fig. 13B Tangible media storing computer readable code are shown, respectively a removable memory unit 365 and a compact disk (CD) 368, which can perform the method according to the example embodiments described above when run by a computer. The removable memory unit 365 can be a memory stick, such as a USB memory stick, with an internal memory 366 storing computer readable code. The internal memory 366 can be accessed by the computer system via a connector 367. The CD 368 can be a CD-ROM or a DVD or the like. Other forms of tangible storage media can be used. The tangible medium can be any device / apparatus capable of storing the following data / information: These data / information can be exchanged between devices / apparatus / networks.

[0083] Embodiments of the present invention may be implemented in software, hardware, application logic, or a combination of software, hardware, and application logic. The software, application logic, and / or hardware may reside in memory or on any computer medium. In an example embodiment, the application logic, software, or instructions are maintained on any of a variety of conventional computer-readable media. In the context of this document, "memory" or "computer-readable medium" may be any of the following non-transitory media or components: These non-transitory media or components may contain, store, convey, propagate, or transmit instructions for use by or in connection with an instruction execution system, device, or apparatus (such as a computer).

[0084] Where relevant, references to "computer-readable medium", "computer program product", "tangibly embodied computer program", etc. or "processor" or "processing circuit system", etc. should be understood to encompass not only computers with different architectures (such as single processor / multi-processor architecture and sequencer / parallel architecture), but also specialized circuits such as field programmable gate arrays FPGA, application specific circuits ASIC, signal processing devices / apparatus and other devices / apparatus. References to computer programs, instructions, codes, etc. should be understood to express software for programmable processor firmware, such as the programmable content of hardware devices / apparatus as instructions for a processor or configured settings or configuration settings for fixed function devices / apparatus, gate arrays, programmable logic devices / apparatus, etc.

[0085] If desired, the different functions discussed in this article may be performed in different orders and / or simultaneously with each other. In addition, if desired, one or more of the above functions may be optional or may be combined. Similarly, it is also understood that: Figure 6 The flowcharts are merely examples, and various operations depicted therein may be omitted, re-ordered, and / or combined.

[0086] It is to be understood that the above exemplary embodiments are merely illustrative and do not limit the scope of the present invention. Other variations and modifications will be apparent to those skilled in the art after reading this specification.

[0087] In addition, the disclosure of the present application should be understood to include any novel feature or any novel combination of features or any generalization thereof disclosed herein explicitly or implicitly, and during the filing of the present application or any application derived from the present application, new claims may be formulated to cover any such feature and / or combination of such features.

[0088] Although various aspects of the invention are set out in the independent claims, further aspects of the invention comprise other combinations of features from the described example embodiments and / or the dependent claims with features from the independent claims and not just the combinations explicitly set out in the claims.

[0089] It should also be noted herein that although various examples are described above, these descriptions should not be viewed in a limiting sense. Rather, several variations and modifications may be made without departing from the scope of the present invention as defined in the appended claims.

Claims

1. A device comprising: means for capturing spatial audio data during an image capture process; means for determining an orientation of said apparatus during capture of said spatial audio data; means for generating an audio focus signal from the captured spatial audio data, wherein the audio focus signal is focused on one or more objects in an image capture direction of the apparatus; means for generating modified spatial audio data, wherein generating the modified spatial audio data comprises: excluding the audio focus signal from the captured spatial audio data to obtain spatial audio data corresponding to one or more background objects, and modifying the spatial audio data corresponding to the one or more background objects to compensate for one or more changes in the orientation of the device during the capture of the spatial audio data, wherein the means for generating the modified spatial audio data is configured to: offset the determined change in the orientation of the device by rotating the captured spatial audio data corresponding to the one or more background objects to compensate for the one or more changes in the orientation of the device; as well as Means for generating an audio output signal from a combination of the audio focus signal and the modified spatial audio data.

2. The apparatus according to claim 1, wherein the spatial audio data is captured from a start time to an end time, the start time being at or before the start of the image capturing process, and the end time being at or after the end of the image capturing process.

3. The apparatus according to any one of claims 1 to 2, wherein the spatial audio data is parametric audio data. 4 . The apparatus of claim 3 , wherein the means for generating modified spatial audio data is configured to generate the modified spatial audio data by modifying parameters of the parameterized audio data.

5. An apparatus according to any preceding claim, wherein the means for generating the audio focus signal comprises one or more beamforming arrangements.

6. An apparatus according to any preceding claim, wherein the means for generating the audio focus signal is configured to emphasise audio in the image capture direction of the apparatus.

7. An apparatus according to any one of the preceding claims, wherein the means for generating the audio focus signal is configured to attenuate the captured spatial audio data in directions other than the image capture direction of the apparatus.

8. The apparatus according to any of the preceding claims, wherein the means for generating the audio output signal is configured to generate the audio output signal based on a weighted sum of the audio focus signal and the modified spatial audio data.

9. The device according to any one of the preceding claims, further comprising: A component used to capture a visual image of an object or scene.

10. An apparatus according to any preceding claim, wherein the means for determining the orientation of the apparatus comprises one or more sensors.

11. The device according to any one of the preceding claims, wherein the component comprises: at least one processor; as well as At least one memory including computer program code, the at least one memory and the computer program being configured to, together with the at least one processor, cause execution of the apparatus.

12. A method comprising: capturing spatial audio data during an image capture process; determining an orientation of an image capture device during said spatial audio data capture; generating an audio focus signal from the captured spatial audio data, wherein the audio focus signal is focused on one or more objects in an image capture direction of the image capture device; Generating modified spatial audio data, wherein generating the modified spatial audio data comprises: excluding the audio focus signal from the captured spatial audio data to obtain spatial audio data corresponding to one or more background objects, and modifying the spatial audio data corresponding to the one or more background objects to compensate for one or more changes in the orientation of the image capture device during the capture of the spatial audio data, wherein generating the modified spatial audio data comprises: offsetting the determined change in the orientation of the image capture device by rotating the captured spatial audio data corresponding to the one or more background objects to compensate for the one or more changes in the orientation of the image capture device; and An audio output signal is generated from a combination of the audio focus signal and the modified spatial audio data.

13. The method of claim 12, wherein generating the audio focus signal comprises: Audio is emphasized in the image capturing direction of the image capturing device.

Images