Apparatus, system, method and medium for processing spatial audio information

Abstract

An apparatus, method, and computer program product for receiving spatial audio information captured by a plurality of microphones, receiving captured audio objects from an audio device wirelessly connected to the apparatus, determining an audio audibility value related to the audio device, determining whether the audio audibility value satisfies at least one criterion, and activating inclusion of audio objects captured by the audio device in the spatial audio information captured by the plurality of microphones in response to determining that the audio audibility value satisfies the at least one criterion.

Description

TECHNICAL FIELD

[0001] The present application relates generally to spatial audio information. More specifically, the present application relates to adding an audio object to spatial audio information. BACKGROUND

[0002] The amount of multimedia content is constantly increasing. Users create and consume multimedia content and it plays an important role in modern society. SUMMARY

[0003] Various aspects of examples of the application are set out in the claims. The scope of the protection sought to be afforded by the various embodiments of the present application, sought by the Applicant, is defined by the independent claims. Examples and features that are not within the scope of the independent claims are to be interpreted as examples useful for understanding the various embodiments of the present application and are not to be interpreted as limiting the scope of the inventive concept.

[0004] According to a first aspect of the application, there is provided an apparatus comprising means for receiving spatial audio information captured by a plurality of microphones, receiving a captured audio object from an audio device wirelessly connected to the apparatus, determining an audio audibility value related to the audio device, determining whether the audio audibility value meets at least one criterion, and in response to determining that the audio audibility value meets the at least one criterion, activating inclusion of the audio object captured by the audio device in the spatial audio information captured by the plurality of microphones.

[0005] According to a second aspect of the application, there is provided a method comprising receiving spatial audio information captured by a plurality of microphones, receiving a captured audio object from an audio device wirelessly connected to the apparatus, determining an audio audibility value related to the audio device, determining whether the audio audibility value meets at least one criterion, and in response to determining that the audio audibility value meets the at least one criterion, activating inclusion of the audio object captured by the audio device in the spatial audio information captured by the plurality of microphones.

[0006] According to a third aspect of the application, there is provided a computer program comprising instructions for causing an apparatus to perform at least the following: receiving spatial audio information captured by a plurality of microphones, receiving a captured audio object from an audio device wirelessly connected to the apparatus, determining an audio audibility value related to the audio device, determining whether the audio audibility value meets at least one criterion, and in response to determining that the audio audibility value meets the at least one criterion, activating inclusion of the audio object captured by the audio device in the spatial audio information captured by the plurality of microphones.

[0007] According to a fourth aspect of the application, there is provided an apparatus comprising at least one processor and at least one memory including a computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to receive spatial audio information captured by a plurality of microphones, receive a captured audio object from an audio device wirelessly connected to the apparatus, determine an audio audibility value related to the audio device, determine whether the audio audibility value fulfils at least one criterion, and in response to determining that the audio audibility value fulfils the at least one criterion, activate inclusion of the audio object captured by the audio device in the spatial audio information captured by the plurality of microphones.

[0008] According to a fifth aspect of the application, there is provided a non-transitory computer readable medium comprising program instructions for causing an apparatus at least to perform the following: receive spatial audio information captured by a plurality of microphones, receive a captured audio object from an audio device wirelessly connected to the apparatus, determine an audio audibility value related to the audio device, determine whether the audio audibility value fulfils at least one criterion, and in response to determining that the audio audibility value fulfils the at least one criterion, activate inclusion of the audio object captured by the audio device in the spatial audio information captured by the plurality of microphones.

[0009] According to a sixth aspect of the application, there is provided a computer readable medium comprising program instructions for causing an apparatus at least to perform the following: receive spatial audio information captured by a plurality of microphones, receive a captured audio object from an audio device wirelessly connected to the apparatus, determine an audio audibility value related to the audio device, determine whether the audio audibility value fulfils at least one criterion, and in response to determining that the audio audibility value fulfils the at least one criterion, activate inclusion of the audio object captured by the audio device in the spatial audio information captured by the plurality of microphones. BRIEF DESCRIPTION OF DRAWINGS

[0010] Some example embodiments will now be described with reference to the drawings:

[0011] Figure 1 A block diagram of an example apparatus in which examples of the disclosed embodiments can be applied is shown;

[0012] Figure 2 A block diagram of another example apparatus in which examples of the disclosed embodiments can be applied is shown;

[0013] Figure 3A 、 Figure 3B and Figure 3C An example system in which examples of the disclosed embodiments can be applied is shown;

[0014] Figure 4A 、 Figure 4B andFigure 4C Another example system showing an example in which the disclosed embodiments can be applied is shown;

[0015] Figure 5 A in Figure 5 B in shows an example user interface;

[0016] Figure 6 An example method is shown; and

[0017] Figure 7A A and Figure 7B Example audio audibility values and thresholds are shown. DETAILED DESCRIPTION

[0018] The following embodiments are examples. Although the specification can

[0019] Example embodiments relate to an apparatus configured to activate inclusion of audio signals captured by an audio device in audio information received by the apparatus. The audio signals captured by the audio device can include, for example, audio captured by a single or multiple microphones.

[0020] Some example embodiments relate to an apparatus configured to receive spatial audio information captured by a plurality of microphones, receive a captured audio object from an audio device wirelessly connected to the apparatus, determine an audio audibility value related to the audio device, determine whether the audio audibility value satisfies at least one criterion, and in response to determining that the audio audibility value satisfies the at least one criterion, activate inclusion of the audio object captured by the audio device in the spatial audio information captured by the plurality of microphones.

[0021] Some example embodiments relate to activating distributed audio or audiovisual capture. Distributed audio / audiovisual capture includes utilizing audio objects received from separate devices.

[0022] Some example embodiments relate to an apparatus comprising an audio codec. An audio codec is a codec configured to encode and / or decode audio signals. The audio codec can include, for example, a speech codec configured to encode and / or decode speech signals. In practice, an audio codec comprises a computer program implementing an algorithm that compresses and decompresses digital audio data. The purpose of this algorithm is to represent a high-fidelity audio signal with the least number of bits while maintaining quality for transmission purposes. In this way, the storage space and bandwidth required for the transmission of audio files can be reduced.

[0023] Different audio codecs can have different bitrates. Bitrate refers to the number of bits processed or transmitted per unit of time. Typically, bitrate is expressed in bits per second or kilobits (e.g., kbps or kbits / second). Bitrates can include constant bitrate (CBR) or variable bitrate (VBR). CBR files allocate a constant amount of data for a time period, while VBR files allow higher bitrates, i.e., more storage space, to be allocated to more complex media file segments, and VBR files allow lower bitrates, i.e., less storage space, to be allocated to less complex media file segments. VBR operations can include discontinuous transmission (DTX), which can be used in combination with CBR or VBR operations. In DTX operations, parameters can be selectively updated to describe background noise levels and / or spectral noise characteristics during inactive periods, such as silence, while regular coding can be used during active periods, such as speech.

[0024] There are different kinds of audio / speech codecs, for example, an enhanced voice service (EVS) codec suitable for improved telephony and teleconferencing, audiovisual conferencing services, and streaming audio. Another example codec is an immersive voice and audio services (IVAS) codec. One purpose of the IVAS codec is to provide support for real-time conversational spatial voice, multi-stream teleconferencing, virtual reality (VR) conversational communications, and / or user-generated live and on-demand content streaming. Conversational communications can include, for example, real-time two-way audio between multiple users. The IVAS codec provides support for, for example, encoding, decoding, and / or rendering from mono to stereo to fully immersive audio. Immersive services can include, for example, immersive voice and audio for virtual reality (VR) or augmented reality (AR), and the codec can be configured to handle encoding, decoding, and rendering of speech, music, and general audio. The codec can also support channel-based audio, object-based audio, and / or scene-based audio.

[0025] For example, channel-based audio can include creating soundtracks by recording separate audio tracks (channels) for each speaker or panning and mixing selected audio tracks between at least two speaker channels. Common speaker arrangements for channel-based surround sound systems are 5.1 and 7.1, which use five and seven surround channels, respectively, along with a low-frequency channel. One disadvantage of channel-based audio is that each soundtrack is created for a specific speaker configuration, such as 2.0 (stereo), 5.1, and 7.1.

[0026] Object-based audio addresses this shortcoming by representing an audio scene as a plurality of individual audio objects, each audio object comprising one or more audio signals and associated metadata. An audio object can be associated with metadata defining a position or trajectory of the object in the audio scene. Object-based audio rendering comprises rendering the audio objects to loudspeaker signals to reproduce the audio scene. In addition to specifying the position and / or movement of an object, the metadata can also define a type of the object (e.g., acoustic properties of the object), and / or a category of a renderer to be used to render the object. For example, an object can be identified as a diffuse object or a point source object. An object-based renderer can use the positioning metadata with rendering algorithms specific to a particular object type to direct the sound object based on knowledge of the loudspeaker positioning of the loudspeaker configuration.

[0027] Scene-based audio combines the advantages of object-based audio and channel-based audio and is suitable for implementing a truly immersive VR audio experience. Scene-based audio comprises encoding and representing a three-dimensional (3D) soundfield of fixed points in space. For example, scene-based audio can comprise Ambisonics and parametric immersive audio. Ambisonics comprises a spherical surround sound format that includes sound sources above and below the listener in addition to the horizontal plane. For example, Ambisonics can comprise first order Ambisonics (FOA) with four channels or higher order Ambisonics (HOA) with more than four channels, such as 9, 16, 25, 36, or 49 channels. Parametric immersive audio can comprise, for example, metadata assisted spatial audio (MASA).

[0028] Spatial audio can comprise a complete spherical surround sound to mimic the way people perceive audio in real life. Spatial audio can comprise audio that is positioned from the user to be assigned to a certain direction and / or distance. Thus, the perceived audio can change as the user moves or the user turns. Spatial audio can comprise audio created by a sound source, ambient audio, or a combination thereof. For example, ambient audio can comprise audio that can not be identifiable in terms of a sound source, such as traffic hum, wind, or waves. Spherical surround sound can comprise a spatial audio scene, and the position of the user or the position of a capture device can be considered a reference point in the spatial audio scene. According to example embodiments, the reference point comprises a center of the audio scene.

[0029] Devices comprising multiple microphones can be used to capture spatial audio information. For example, a user can capture spatial audio or video information including spatial audio while watching a performance of a choir. However, the positioning of the user capturing the spatial audio information can not be optimal in terms of positioning away from the choir. If the distance between the capturing device and the sound source is long, the signal-to-noise ratio (SNR) is deteriorated compared to when the distance between the capturing device and the sound source is short. Another problem is that it can not be possible to distinguish the performance of a particular person in the choir from the overall capture. It can be very challenging to separate a particular sound source from multiple sound sources, especially in the presence of multiple spatially overlapping sound sources.

[0030] Figure 1 is a block diagram depicting an apparatus 100 operating in accordance with an example embodiment of the application. The apparatus 100 can be an electronic device such as a chip or a chipset, for example. The apparatus 100 comprises one or more control circuitries, such as at least one processor 110, and at least one memory 160 comprising one or more algorithms, such as computer program code 120, wherein the at least one memory 160 and the computer program code 120 are configured to, with the at least one processor 110, cause the apparatus 100 to perform any of the example functions described below.

[0031] In Figure 1 example, the processor 110 is a control unit operably connected to read from and write to the memory 160. The processor 110 can also be configured to receive control signals received via an input interface, and / or the processor 110 can be configured to output control signals via an output interface. In example embodiments, the processor 110 can be configured to convert received control signals into appropriate commands to control the functioning of the apparatus 100.

[0032] The at least one memory 160 stores computer program code 120 which, when loaded into the processor 110, controls the operation of the apparatus 100 as described below. In other examples, the apparatus 100 can comprise more than one memory 160 or different kinds of storage devices.

[0033] The computer program code 120 for implementing the example embodiments of the application, or parts of such computer program code, can be loaded onto the device 100 by the manufacturer of the device 100, by a user of the device 100, or by the device 100 itself based on a download program, or the code can be pushed to the device 100 by an external device. The computer program code 120 can arrive at the device 100 via an electromagnetic carrier signal, or be copied from a physical entity such as a computer program product, a memory device, or a record medium such as a compact disc (CD), a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), or a Blu-ray disc.

[0034] Figure 2 is a block diagram depicting a device 200 according to an example embodiment of the application. The device 200 can be an electronic device such as a handheld device, a mobile phone or a personal digital assistant (PDA), a personal computer (PC), a laptop, a desktop, a tablet, a wireless terminal, a communication terminal, a game console, a music player, an e-book reader, a positioning device, a digital camera, a household appliance, a CD-, DVD, or Blu-ray player, or a media player. In the following examples, it is assumed that the device 200 is a mobile computing device or a part thereof.

[0035] In Figure 2 In an example embodiment, the device 200 is shown to comprise the device 100, a plurality of microphones 210, one or more loudspeakers 230, and a user interface 220 for interacting with the device 200, e.g., a mobile computing device. The device 200 can also comprise a display configured to act as the user interface 220. For example, the display can be a touch screen display. In an example embodiment, the display and / or the user interface 220 can be external to the device 200, but in communication with the device 200.

[0036] Additionally or alternatively, the user interface 220 can also comprise manually operable controls such as buttons, keys, touch pads, joysticks, styluses, pens, scroll wheels, rocker switches, keypads, keyboards, or any suitable input mechanism for inputting and / or accessing information. Other examples include cameras, voice recognition systems, eye movement recognition systems, acceleration, tilt, and / or motion based input systems. Thus, the device 200 can also comprise different kinds of sensors such as one or more gyroscopic sensors, accelerometers, magnetometers, positioning sensors, and / or tilt sensors.

[0037] According to an example embodiment, the device 200 is configured to establish radio communication with another device using, e.g., Bluetooth, WiFi, radio frequency identification (RFID), or near field communication (NFC) connection. For example, the device 200 can be configured to establish radio communication with wireless earphones, augmented / virtual reality devices, etc.

[0038] According to example embodiments, the apparatus 200 is operatively connected to an audio device 250. According to example embodiments, the apparatus 200 is wirelessly connected to the audio device 250. For example, the apparatus 200 can be connected to the audio device 250 through a Bluetooth connection or the like.

[0039] The audio device 250 can comprise at least one microphone for capturing audio signals and at least one loudspeaker for playing back received audio signals. The audio device 250 can further be configured to filter out background noise and / or to detect an in-ear position. The audio device 250 can comprise a single audio device 250 or a first audio device and a second audio device configured to work as a pair. The audio device 250 comprising the first audio device and the second audio device can be configured such that the first audio device and the second audio device can be used separately and / or independently from each other.

[0040] According to example embodiments, the audio device 250 comprises a wireless earpiece. The wireless earpiece can be used independently from other wireless earpieces and / or together with at least one other wireless earpiece. For example, assuming that the audio device 250 comprises a pair of wireless earpieces, the same or different audio information can be directed to each wireless earpiece or the audio information can be directed to a single wireless earpiece and the other wireless earpiece can act as a microphone.

[0041] According to example embodiments, the audio device 250 is configured to receive audio information from the apparatus 200. The apparatus 200 can be configured to control the provision of audio information to the audio device 250 based on characteristics of the audio device 250 or characteristics of the apparatus 200. For example, the apparatus 200 can be configured to adjust one or more settings in the apparatus 200 and / or the audio device 250 when providing audio information to the audio device 250. For example, the one or more settings can relate to the playback of the audio information, the number of available loudspeakers, or the like.

[0042] For example, the audio information can comprise a speech signal representing speech of a caller or streaming audio information. According to example embodiments, the audio device 250 is configured to render the audio information received from the apparatus 200 by causing the output of the received audio information via the at least one loudspeaker.

[0043] According to example embodiments, the audio device 250 is configured to transmit audio information to the apparatus 200. For example, the audio information can comprise a speech signal representing speech or some other type of audio information.

[0044] According to an example embodiment, the apparatus 200 is configured to receive spatial audio information captured by a plurality of microphones. The spatial audio information comprises at least one audio signal and at least one audio parameter for controlling the at least one audio signal. For example, the at least one audio parameter can comprise an audio parameter corresponding to a direction and / or a positioning of the audio relative to a reference point in a spatial audio field.

[0045] According to an example embodiment, the apparatus 200 is configured to capture spatial audio information using a plurality of microphones 210. The plurality of microphones 210 can be configured to capture audio signals around the capturing device. The plurality of microphones 210 can be comprised by the apparatus 200 or the plurality of microphones 210 can comprise separate microphones operatively connected to the apparatus 200.

[0046] According to an example embodiment, the spatial audio information comprises spatial audio information captured during a voice or video call.

[0047] According to an example embodiment, the apparatus 200 is configured to receive captured audio objects from an audio device wirelessly connected to the apparatus 200. For example, the captured audio objects can comprise audio objects captured by at least one microphone comprised by the audio device 250.

[0048] According to an example embodiment, the audio objects comprise audio data associated with metadata. The metadata associated with the audio objects provides information about the audio data. The information about the audio data can comprise, for example, one or more properties of the audio data, one or more characteristics of the audio data, and / or identification information related to the audio data. For example, the metadata can provide information about a positioning associated with the audio data in a spatial audio field, a movement of the audio objects in the spatial audio field, and / or a function of the audio data.

[0049] According to an example embodiment, the audio objects comprise spatial audio objects comprising one or more audio signals and associated metadata defining a positioning and / or a trajectory of the second audio object in a spatial audio field.

[0050] Without limiting the scope of the claims, an advantage of the audio objects is that the metadata can be associated with the audio signals such that the audio signals can be reproduced by defining a position of the audio signals in a spatial audio field.

[0051] Receiving the audio objects from the audio device can comprise decoding the received audio objects using an audio codec. The audio codec can comprise, for example, an IVAS codec or a suitable Bluetooth audio codec.

[0052] According to example embodiments, the apparatus 200 comprises an audio codec comprising a decoder for decompressing received data such as an audio stream and / or an encoder for compressing data for transmission. The received audio data can comprise, for example, an encoded bitstream comprising binary bits of information that can be transferred from one device to another.

[0053] According to example embodiments, the audio object comprises an audio stream. The audio stream can comprise a live audio stream comprising real-time audio. The audio stream can be streamed together with other types of media streams, or the audio can be streamed as part of other types of media streams such as video streams. For example, the audio stream can comprise audio from a live performance or the like.

[0054] According to example embodiments, the apparatus 200 is configured to determine an audio audibility value related to the audio device 250.

[0055] The audio audibility value can comprise a parameter value comprising information about a relationship between the audio device 250 and the apparatus 200. For example, the parameter value can comprise contextual information such as a positioning of the audio device 250 relative to the apparatus 200. As another example, the parameter value can comprise information about characteristics of content captured by the audio device 250 in relation to characteristics of content captured by the apparatus 200.

[0056] According to example embodiments, the audio audibility value related to the audio device 250 depends on a distance between the audio device 250 and the apparatus 200. According to example embodiments, the apparatus 200 is configured to update the audio audibility value in response to receiving information about a changed distance between the audio device 250 and the apparatus 200. The apparatus 200 can receive information about the changed distance, for example, by detecting a change in distance or in response to receiving information about the changed distance from a cloud server to which the apparatus 200 and the audio device 250 are operatively connected.

[0057] According to example embodiments, the audio audibility value related to the audio device 250 comprises a distance between the audio device 250 and the apparatus 200. The distance can comprise an absolute distance or a relative distance.

[0058] The apparatus 200 can be configured to determine the distance between the apparatus 200 and the audio device 250 based on positioning information such as global positioning system (GPS) coordinates, based on a wireless connection between the apparatus 200 and the audio device 250, based on acoustic measurements such as a delay at the time of a detection event.

[0059] As another example, the apparatus 200 can be configured to determine the distance between the apparatus 200 and the audio device 250 based on information received from the cloud server. For example, if the positions of the apparatus 200 and the audio device 250 are stored on the cloud server, the cloud server can inform the apparatus 200 about the respective positions or distances between the apparatus 200 and the audio device 250.

[0060] According to an example embodiment, the audio audibility value related to the audio device 250 comprises a sound travel time between the audio device 250 and the apparatus 200.

[0061] According to an example embodiment, the audio audibility value related to the audio device 250 is adapted based on a sound pressure or a noise level. The sound pressure comprises a total sound pressure, and the noise level comprises a total noise level. According to another example embodiment, the audio audibility value related to the audio device 250 is adapted based on a correlation measure between the spatial audio information and the audio object.

[0062] According to an example embodiment, the apparatus 200 is configured to determine whether the audio audibility value satisfies at least one criterion. According to an example embodiment, determining whether the audio audibility value satisfies at least one criterion comprises comparing the audio audibility value with a value of a threshold, and determining whether the audio audibility value is equal to, below, or above the value of the threshold.

[0063] According to an example embodiment, the at least one criterion comprises a value of a threshold depending on a distance between the audio device 250 and the apparatus 200. For example, assuming that the audio audibility value comprises a distance between the apparatus 200 and the audio device 250, the value of the threshold comprises a threshold distance. As another example, assuming that the audio audibility value comprises a sound travel time, the value of the threshold comprises a threshold time.

[0064] According to an example embodiment, the value of the threshold depending on the distance between the audio device 250 and the apparatus 200 is adapted based on a sound pressure or a noise level. For example, a relatively distant sound source in a quiet environment can remain audible in a spatial audio capture using the apparatus 200, while a sound source in a noisier environment needs to be closer to the apparatus 200 to be audible.

[0065] Without limiting the scope of the claims, the advantage of adapting the value of the threshold based on a sound pressure level or a noise level is that the value of the threshold can be dynamically adapted taking into account the environment.

[0066] According to an example embodiment, determining whether the audio audibility value satisfies at least one criterion comprises determining whether the audio audibility value is above the value of the threshold.

[0067] According to another example embodiment, determining whether the audio audibility value satisfies at least one criterion comprises determining whether the audio audibility value is below the value of the threshold.

[0068] According to another example embodiment, determining whether the audio audibility value satisfies the at least one criterion comprises determining whether the audio audibility value is equal to a value of a threshold.

[0069] According to an example embodiment, the apparatus 200 is configured to activate inclusion of the audio object captured by the audio device 250 in the spatial audio information captured by the plurality of microphones in response to determining that the audio audibility value satisfies the at least one criterion.

[0070] Activating inclusion of the audio object captured by the audio device 250 in the spatial audio information captured by the plurality of microphones can comprise activating a microphone associated with the audio device 250, activating reception of an audio signal from the audio device 250, deactivating a loudspeaker associated with the audio device 250, etc.

[0071] Activating inclusion of the audio object in the spatial audio information can comprise controlling operation of the audio device 250. According to an example embodiment, the apparatus 200 is configured to switch the audio device 250 from a first mode to a second mode. The first mode can comprise, for example, a loudspeaker mode, while the second mode can comprise, for example, a microphone mode. The loudspeaker mode comprises using the audio device 250 as a loudspeaker, while the microphone mode comprises using the audio device 250 as a microphone.

[0072] According to an example embodiment, switching the audio device 250 from the first mode to the second mode comprises switching an audio output port of the audio device 250 to an audio input port of the audio device 250.

[0073] According to an example embodiment, the apparatus 200 is configured to provide modified spatial audio information in response to activating inclusion of the audio object in the spatial audio information. The modified spatial audio information can comprise a combined representation of an audio scene comprising the spatial audio information and the audio object, or a representation of an audio scene in which the spatial audio information and the audio object are separate components. For example, the modified spatial information can comprise the spatial audio information into which the audio object is downmixed. As another example, the modified spatial audio information can comprise the spatial audio information and the audio object as separate components.

[0074] Inclusion of the audio object in the spatial audio information can comprise controlling audio encoder input by the apparatus 200. For example, inclusion of the audio object in the spatial audio information can comprise including the audio object in an audio codec input format such that the same audio encoder is configured to jointly encode or package and deliver the two audio signals together.

[0075] According to an example embodiment, the apparatus 200 is configured to include the audio object in the audio encoder input. According to another example embodiment, the apparatus 200 is configured to activate the use of the audio object in the audio encoder input. According to another example embodiment, the apparatus 200 is configured to renegotiate or reinitialize the audio encoder input such that the audio object is included in the encoder input. For example, if the audio encoder input was previously negotiated to be first order ambisonics (FOA), the audio encoder input can be renegotiated to be FOA and the audio object. According to yet another example embodiment, the apparatus 200 is configured to replace the previous spatial audio information with the modified spatial audio information.

[0076] The inclusion of the audio object in the spatial audio information can be performed based on metadata associated with the audio object.

[0077] The inclusion of the audio object in the spatial audio information can be activated for a period of time. In other words, the inclusion can also be terminated. According to an example embodiment, the apparatus 200 is configured to deactivate the inclusion of the audio object captured by the audio device in the spatial audio information captured by the plurality of microphones.

[0078] According to an example embodiment, the apparatus 200 is configured to deactivate the inclusion of the audio object captured by the audio device in the spatial audio information in response to determining that the audio audibility value satisfies at least one criterion. The at least one criterion for deactivating the inclusion of the audio object can be different from the at least one criterion for activating the inclusion of the audio object.

[0079] Without limiting the scope of the claims, the advantage of different threshold values for activating and deactivating the inclusion of the audio object in the spatial audio information is that a suitable hysteresis can be provided to prevent the inclusion of the audio object in the spatial audio information from being activated and deactivated frequently.

[0080] According to an example embodiment, deactivating the inclusion of the audio object captured by the audio device 250 in the spatial audio information can include deactivating a microphone associated with the audio device 250, deactivating the reception of the audio signal from the audio device 250, activating a loudspeaker associated with the audio device 250, instructing a microphone associated with the audio device to act as a loudspeaker, or a combination thereof.

[0081] Deactivating the inclusion of the audio object in the spatial audio information can include controlling the operation of the audio device 250. According to an example embodiment, the apparatus 200 is configured to switch the audio device 250 from a second mode to a first mode. The first mode can include, for example, a loudspeaker mode, while the second mode can include, for example, a microphone mode. The loudspeaker mode includes using the audio device 250 as a loudspeaker, while the microphone mode includes using the audio device 250 as a microphone.

[0082] As described above, the apparatus 200 can comprise a user interface for enabling a user to control and / or monitor the received spatial audio information and / or the received audio objects. For example, the user interface can enable controlling and / or monitoring the volume, the position of the audio objects in the spatial audio field, the balance, etc.

[0083] According to example embodiments, the apparatus 200 is configured to provide the user interface based on the available spatial audio objects. Thus, the apparatus 200 can be configured to dynamically adapt the user interface.

[0084] According to example embodiments, the apparatus 200 is configured to provide a control element for controlling the captured spatial audio information and to adapt the user interface in response to determining that the audio audibility value fulfils the at least one criterion. For example, adapting the user interface can comprise modifying the content of the user interface by adding, removing and / or modifying one or more user interface elements. Modifying the one or more user interface elements can comprise, for example, modifying the appearance and / or operation of the one or more user interface elements. For example, the user interface can comprise a volume control for the captured spatial audio information and, in response to determining that the audio audibility value fulfils the at least one criterion, the user interface can be adapted to further comprise a volume control for the audio object.

[0085] According to example embodiments, the apparatus 200 comprises means for performing features of the claimed application, wherein the means for performing comprise at least one processor 110, at least one memory 160 including computer program code 120, the at least one memory 160 and the computer program code 120 configured to, with the at least one processor 110, cause the operation of the apparatus 200. The means for performing features of the claimed application can comprise means for receiving spatial audio information captured by a plurality of microphones, means for receiving a captured audio object from an audio device wirelessly connected to the apparatus, means for determining an audio audibility value related to the audio device, means for determining whether the audio audibility value fulfils at least one criterion, and means for activating, in response to determining that the audio audibility value fulfils the at least one criterion, the inclusion of audio captured by the audio device in the spatial audio information captured by the plurality of microphones.

[0086] The apparatus 200 can further comprise means for deactivating the inclusion of the audio object captured by the plurality of microphones. The apparatus 200 can further comprise means for switching the audio device 250 from the first mode to the second mode. The apparatus 200 can further comprise means for providing a control element for controlling the captured spatial audio information and means for adapting the user interface in response to determining that the audio audibility value fulfils the at least one criterion.

[0087] Figure 3A 、 Figure 3B and Figure 3CAn example system according to an example embodiment is shown. Figure 3A , Figure 3B and Figure 3C In one example, device 200 includes an audio codec that supports user-generated live content streaming.

[0088] exist Figure 3A In the example, a first user is having a voice or video call with a second user (not shown). For example, the first user 301 may use a device 200 for capturing spatial audio information and receive audio from the second user using an audio device 250 such as a wireless headset. The audio device 250 is wirelessly connected to the device 200 using, for example, a Bluetooth connection. The audio device 250 includes at least one speaker and at least one microphone. Figure 3A In the example, the audio received from the second user is shown by arrow 306. The first user 301 captures spatial audio information for the second user. The captured spatial audio information is shown by arrow 305. Figure 3A , Figure 3B and Figure 3C In the example, the third user 303 is the sound source of interest. For example, the third user 303 could be a person singing in a choir.

[0089] exist Figure 3A In the example, the first user 301 uses a single wireless headset. In this case, the headset can be configured to act as either a microphone or a speaker by default.

[0090] exist Figure 3B In the example, the first user 301 has already handed over the audio device 250 to the third user 303. Assuming the third user 303 is a singer in a choir, the distance between the audio device 250 and the device 200 increases as the third user 303 moves to the venue.

[0091] exist Figure 3C In the example, the distance 307 between device 200 and audio device 250 increases. Device 200 is configured to determine whether the distance 307 between device 200 and audio device 250 is above a threshold value. Device 200 is also configured to activate the inclusion of audio objects captured by audio device 250 in the spatial audio information captured by audio device 250 in response to determining that the distance 307 between device 200 and audio device 250 is above the threshold value. If audio device 250 defaults to acting as a microphone, activating the audio objects may include activating the reception of audio signals from audio device 250. If audio device 250 defaults to acting as a speaker, activating the audio objects may include switching audio device 250 from speaker mode to microphone mode.

[0092] Figure 4A , Figure 4Band Figure 4C Another example system according to example embodiments is shown. In Figure 4A 、 Figure 4B and Figure 4C In an example, the apparatus 200 comprises an audio codec supporting user generated live content streaming.

[0093] In an example, a first user is having a voice or video call with a second user (not shown). For example, the first user 301 can use the apparatus 200 for capturing spatial audio information and receive audio from the second user using a pair of audio devices 250 such as wireless earphones. The pair of audio devices 250 are wirelessly connected to the apparatus 200 using, for example, a Bluetooth connection. Figure 4A

[0094] The audio devices 250 comprise at least one loudspeaker and at least one microphone. In an example, the audio received from the second user is shown with arrow 306. The first user 301 captures spatial audio information for the second user. The captured spatial audio information is shown with arrow 305. In an example, the first user 301 is a sound source of interest. For example, the first user 301 can be a person singing in a choir. Figure 4A 、 Figure 4A 、 Figure 4B and Figure 4C In an example, a third user 303 is a sound source of interest. For example, the third user 303 can be a person singing in a choir.

[0095] In an example, the first user 301 uses a pair of wireless earphones. The pair of wireless earphones can comprise a first wireless earphone and a second wireless earphone. In this case, one earphone can be configured to act as a microphone and one earphone can be configured to act as a loudspeaker. Figure 4A In an example, the first user 301 has handed one of the audio devices 250 to the third user 303. In the following, it is assumed that the first user 301 uses the first wireless earphone and the third user 303 uses the second wireless earphone. Assuming that the third user 303 is the person singing in the choir, the distance between the audio device 250 of the third user 303 and the apparatus 200 increases as the third user 303 moves to the venue.

[0096] Figure 4B In an example, the first user 301 has handed one of the audio devices 250 to the third user 303. In the following, it is assumed that the first user 301 uses the first wireless earphone and the third user 303 uses the second wireless earphone. Assuming that the third user 303 is the person singing in the choir, the distance between the audio device 250 of the third user 303 and the apparatus 200 increases as the third user 303 moves to the venue.

[0097] In an example, the first user 301 has handed one of the audio devices 250 to the third user 303. In the following, it is assumed that the first user 301 uses the first wireless earphone and the third user 303 uses the second wireless earphone. Assuming that the third user 303 is the person singing in the choir, the distance between the audio device 250 of the third user 303 and the apparatus 200 increases as the third user 303 moves to the venue. Figure 4C ​​In the example of FIG. 3, the distance 307 between the apparatus 200 and the audio device 250 (e.g., a second wireless earphone) increases. The apparatus 200 is configured to determine whether the distance 307 between the apparatus 200 and the audio device 250 of the third user 303 is above a value of a threshold. The apparatus 200 is further configured to activate inclusion of an audio object captured by the audio device 250 in the spatial audio information captured by the apparatus 200 in response to determining that the distance 307 between the apparatus 200 and the audio device 250 of the third user 303 is above the value of the threshold. Assuming that the audio device 250 of the third user 303 is configured to act as a microphone, activating the inclusion of the audio object can include activating reception of an audio signal from the audio device 250 of the third user. On the other hand, assuming that the audio device 250 of the third user 303 is configured to act as a loudspeaker, activating the inclusion of the audio object can include transmitting an instruction to change the audio device 250 of the third user 303 from a first mode to a second mode. For example, activating the inclusion of the audio object can include transmitting an instruction to change the audio device 250 of the third user 303 from a loudspeaker mode to a microphone mode. As another example, activating the inclusion of the audio object can include transmitting an instruction to stop using a loudspeaker, which can result in activating a microphone mode.

[0098] Figure 5 A in FIG. 3A and Figure 5 B in FIG. 3B show example user interfaces according to example embodiments. More specifically, Figure 5 the example user interface in A in FIG. 3A shows a user interface for controlling captured spatial audio information, and Figure 5 the example user interface in B in FIG. 3B shows dynamically adapting Figure 5 the user interface shown in A in FIG. 3A in response to determining that the audio audibility value related to the audio device 250 satisfies at least one criterion for activating inclusion of an audio object in the spatial information.

[0099] In the example of FIG. 3, Figure 5 A in FIG. 3A and Figure 5 B in FIG. 3B, the audio device 250 comprises a pair of wireless earphones. The pair of wireless earphones can comprise a first wireless earphone and a second wireless earphone. Similar to the example of Figure 4A , Figure 4B and Figure 4C , it is assumed that the first user 301 uses the first wireless earphone and the third user 303 uses the second wireless earphone.

[0100] The apparatus 200 is configured to provide the user interfaces 501 and 510. The apparatus 200 is further configured to provide one or more control elements rendered on the user interfaces 501, 510, and a representation of the spatial audio field 502. In Figure 5 A in FIG. 3A and Figure 5In example B, it is assumed that the reference point of the spatial audio field includes the center of the spatial audio field 502 and the center of the spatial audio field corresponds to the location of the device 200.

[0101] exist Figure 5 In example A, the first user 301 utilizes spatial audio input. User interface 501 includes a control element 505 for controlling the volume of the spatial audio information. User interface 501 is also configured to present a representation of a spatial audio field 502. The representation of the spatial audio field 502 includes indications of different directions, such as front, right, back, and left relative to a reference point.

[0102] Figure 5 Example B in the figure shows a scenario where a first user 301 has handed a wireless headset (such as a second wireless headset) to a third user 303, and the audio audibility value associated with the audio device 250 satisfies at least one criterion for activating the inclusion of audio objects in spatial audio information.

[0103] exist Figure 5 In example B, at least one criterion includes a distance 307 between the wireless headset 250 of the third user 303 (the second wireless headset) and the wireless headset 250 of the first user 301 (the first wireless headset) or device 200. When the distance 307 is above a threshold value, the inclusion of the audio object in the spatial audio information is activated by device 200. Device 200 is configured to adapt to user interface 501 to be able to control the audio object.

[0104] exist Figure 5 In example B, the user interface 501 includes a control element 505 for controlling the volume of received spatial audio information and a control element 515 for controlling the volume of added audio objects. The added audio object is indicated as a far source on control element 515. The position of the audio object 504 is indicated approximately in the right-front direction within the spatial audio field 502.

[0105] Refer again Figure 5 As in example A, user interface 510 includes control elements 505 for controlling the volume of received spatial audio information and control elements 525 for controlling the volume of the voice channel. For example, a first user 301 can capture spatial audio information and simultaneously listen to audio from a second user or monitor spatial audio capture. In other words, the first user 301 utilizes two audio inputs. The representation of the spatial audio field 502 includes indications of different directions (such as front, right, back, and left relative to a reference point) and an indication of the voice channel 503's orientation generally towards the left.

[0106] exist Figure 5In the example of B in FIG. 5, the user interface 501 includes a control element 505 for controlling the volume of the received spatial audio information, a control element 525 for controlling the volume of the voice channel, and a control element 515 for controlling the volume of the added audio object. The added audio object is indicated on the user interface 510 as a far source. The position of the audio object 504 is indicated as approximately in the front right direction, and the positioning of the voice channel 503 is indicated as approximately left in the spatial audio field.

[0107] Figure 6 An example method 600 is shown that incorporates aspects of the previously disclosed embodiments. More specifically, the example method 600 shows activating the inclusion of an audio object in spatial audio information. The method can be performed by the apparatus 200, such as a mobile computing device.

[0108] The method begins by receiving 605 spatial audio information captured by a plurality of microphones. The method continues by receiving 610 an audio object captured from an audio device 250 wirelessly connected to the apparatus 200.

[0109] The method further continues by determining 615 an audio audibility value related to the audio device 250.

[0110] The method further continues by determining 620 whether the audio audibility value satisfies at least one criterion. If the audio audibility value does not satisfy the at least one criterion, the method returns to determining 620 whether the audio audibility value satisfies the at least one criterion. If the audio audibility value satisfies the at least one criterion, the method continues by activating 625 the inclusion of the audio object captured by the audio device 250 in the spatial audio information captured by the plurality of microphones.

[0111] Figure 7A and Figure 7B Examples of values of the audio audibility value and the audio audibility threshold are shown. The apparatus 200 is configured to determine the audio audibility value based on a relationship between the apparatus 200 and the audio device 250.

[0112] In the example of B in FIG. 5, the user interface 501 includes a control element 505 for controlling the volume of the received spatial audio information, a control element 525 for controlling the volume of the voice channel, and a control element 515 for controlling the volume of the added audio object. The added audio object is indicated on the user interface 510 as a far source. The position of the audio object 504 is indicated as approximately in the front right direction, and the positioning of the voice channel 503 is indicated as approximately left in the spatial audio field. Figure 7A

[0113] Figure 7B Two example embodiments of values of the audio audibility value and the audio audibility threshold are shown. In the example of A in FIG. 5, the audio audibility value is determined based on a distance between the apparatus 200 and the audio device 250. Figure 7B ​In the example of FIG. 2, the audio audibility value is determined based on the distance between the device 200 and the audio device 250, which is adapted based on the sound pressure level. Determining the audio audibility value based on the sound pressure level can include keeping the sound pressure level as a fixed value and adapting the distance or determining a value of an adaptive audio audibility threshold depending on the sound pressure level.

[0114] Without limiting the scope of the claims, an advantage of activating the inclusion of the audio object into the spatial audio information is that sound sources of interest can be combined and / or isolated in the spatial audio information. Another advantage is that a user capturing the spatial audio information can pick up sound sources of interest even if the venue is crowded or the like. Another advantage is that sound sources that can be inaudible due to distance or other factors can be included in the spatial audio information. Another advantage is that sound sources of interest can be included in the spatial audio information when necessary. Another advantage is that a regular accessory can be utilized without the need to invest in expensive and complex equipment.

[0115] Without limiting the scope, interpretation, or application of the claims appearing below in any way, a technical effect of one or more of the example embodiments disclosed herein is that high quality spatial audio capture can be provided without complex arrangements. Another technical effect is that the inclusion of audio objects can be activated automatically. A further technical effect is that computing resources and bandwidth can be saved when avoiding unnecessary inclusion of sound sources of interest in the spatial audio information.

[0116] As used in this application, the term "circuitry" can refer to one or more or all of the following: (a) hardware-only circuitry such as only analog and / or digital circuitry, and (b) combinations of hardware circuits and software, such as (as applicable): (i) combinations of analog and / or digital hardware circuit(s) with software / firmware, (ii) portions of hardware circuit(s) with software / firmware, (iii) hardware circuit(s) implemented with software / firmware and / or (iv) hardware circuits that are entirely software and / or firmware only. In at least one embodiment, the circuitry that can be a part of the circuitry can execute instructions. The term "circuitry" can also cover, for example, an implementation that includes one or both of: (i) reconfigurable circuits such as reconfigurable logic fabric (e.g., field-programmable gate arrays (FPGAs)) and (ii) circuits that include a general purpose processor (e.g., a central processing unit (CPU, or "processor")) and / or a special purpose processor (e.g., digital signal processors (DSPs), graphics processing units (GPUs), physics processing units (PPUs), and / or the like) and associated fixed function or programmable function custom circuitry and a memory subsystem (e.g., that can include one or more memories).

[0117] The definition of "circuit system" applies to all uses of the term in this application, including in any claim. As another example, as used in this application, the term "circuit system" also covers implementations of hardware circuitry or processors (or processors) alone, or a portion thereof, and their accompanying software and / or firmware. For example, if applicable to a particular claim element, the term "circuit system" also covers baseband integrated circuits or processor integrated circuits for mobile devices, or similar integrated circuits in servers, cellular network devices, or other computing or network devices.

[0118] Embodiments of the present invention can be implemented as software, hardware, application logic, or a combination of software, hardware, and application logic. The software, application logic, and / or hardware may reside on a device, a single device, or multiple devices. If desired, a portion of the software, application logic, and / or hardware may reside on a device, a portion of the software, application logic, and / or hardware may reside on a single device, and a portion of the software, application logic, and / or hardware may reside on multiple devices. In exemplary embodiments, the application logic, software, or instruction set is maintained on any of a variety of conventional computer-readable media. In the context of this document, "computer-readable media" can be any medium or component that can contain, store, transmit, propagate, or transfer instructions for use by or in connection with an instruction execution system, device, or apparatus (such as a computer). Figure 2 An example of a computer is described and depicted. A computer-readable medium may include a computer-readable storage medium, which may be any medium or component that can contain or store instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer.

[0119] If necessary, the different functions discussed in this document may be executed in different orders and / or simultaneously with each other. Furthermore, if necessary, one or more of the above functions may be optional or may be combined.

[0120] Although various aspects of the invention are set forth in the independent claims, other aspects of the invention include other combinations of features from the described embodiments and / or dependent claims with features of the independent claims, and not only the combinations expressly set forth in the claims.

[0121] It will be apparent to those skilled in the art that the concept of this invention can be implemented in various ways as technology advances. The invention and its embodiments are not limited to the examples described above, but can be varied within the scope of the claims.

Claims

1. An apparatus (200) for processing spatial audio information, comprising: Multiple first microphones were used to capture spatial audio information; A component for receiving spatial audio information captured by the plurality of first microphones; Components for wireless connection to an audio device (250), the audio device (250) including at least one speaker and at least one second microphone, wherein the audio device (250) includes a first wireless earphone and a second wireless earphone configured to function as a pair of wireless earphones; A component for receiving, when the device (200) is wirelessly connected to the audio device (250), an audio object captured by the audio device (250), wherein the audio object includes one or more audio signals captured by the at least one second microphone and associated metadata; Components for determining an audio audibility value associated with the audio device (250), wherein the audio audibility value is determined based on the distance between the device (200) and the audio device (250) or the distance between the first wireless earphone and the second wireless earphone, and the distance is used as the audio audibility value, or the audio audibility value is the sound flight time between the audio device (250) and the device (200); A component for determining whether the audio audibility value meets at least one criterion by comparing the distance with a threshold distance value or by comparing the sound flight time with a threshold time; as well as Components for activating the inclusion of the audio object captured by the audio device in the spatial audio information captured by the plurality of first microphones in response to: determining that the audio audibility value satisfies the at least one criterion by determining that the distance is above at least one threshold distance value or the sound flight time is above the threshold time.

2. The apparatus (200) according to claim 1, wherein the audio object comprises an audio stream as the audio signal.

3. The apparatus (200) according to claim 1 or 2, wherein the apparatus includes a component for deactivating the audio object captured by the audio device (250) from the spatial audio information captured by the plurality of first microphones.

4. The apparatus (200) according to claim 1 or 2, wherein the apparatus (200) includes components for switching the audio device (250) from a first mode to a second mode. The first mode includes a speaker mode, which includes using the audio device (250) as a speaker, and the second mode includes a microphone mode, which includes using the audio device (250) as a microphone. Switching the audio device (250) from the first mode to the second mode includes: The audio device (250) is switched from the speaker mode to the microphone mode based on an instruction to switch the audio device (250) from the speaker mode to the microphone mode, or based on an instruction to stop using the speaker.

5. The apparatus (200) according to claim 4, wherein switching the audio device (250) from the first mode to the second mode comprises switching the audio output port of the audio device (250) to the audio input port of the audio device (250).

6. The apparatus (200) according to claim 1 or 2, wherein the apparatus (200) includes components for providing control elements for controlling the captured spatial audio information, and components for adapting a user interface in response to determining that the audio audibility value satisfies the at least one criterion.

7. A system for processing spatial audio information, comprising at least: The apparatus according to any one of the preceding claims; as well as An audio device (250) capable of wirelessly connecting to the device, and comprising: At least one second microphone; and At least one speaker; Components for wireless connection to the device; Components for capturing audio objects, the audio objects including one or more audio signals captured by the at least one second microphone and associated metadata; Components for transmitting audio objects to the device when connected to the device.

8. A method for processing spatial audio information, comprising: The device (200) receives spatial audio information captured by a plurality of first microphones (210) of the device; The device (200) receives a captured audio object from an audio device (250) wirelessly connected to the device (200), wherein the captured audio object includes one or more audio signals and associated metadata, and wherein the captured audio object is captured by the audio device (250), and the audio device (250) includes at least one speaker and at least one second microphone for capturing audio signals, wherein the audio device (250) includes a first wireless earphone and a second wireless earphone configured to operate as a pair of wireless earphones; The device (200) determines an audio audibility value associated with the audio device (250), wherein the audio audibility value is determined based on the distance between the device (200) and the audio device (250) or the distance between the first wireless earphone and the second wireless earphone, and the distance is used as the audio audibility value, or the audio audibility value is the sound flight time between the audio device (250) and the device (200); The device (200) determines whether the audio audibility value meets at least one criterion by comparing the distance with a threshold distance value or by comparing the sound flight time with a threshold time; and The device (200) activates the inclusion of the audio object captured by the audio device into the spatial audio information captured by the plurality of first microphones in response to: determining that the audio audibility value satisfies the at least one criterion by determining that the distance is above at least one threshold distance value or the sound flight time is above the threshold time.

9. A computer-readable medium comprising a means for causing a device to perform the method according to claim 8.

Images