Audio processing in immersive audio services

Abstract

The disclosures herein broadly relate to the capture, acoustic preprocessing, encoding, decoding, and rendering of audio scenes. [Solution] More specifically, the present disclosure relates to an apparatus adapted to modify the directional characteristics of captured directional audio in response to spatial data of a microphone system capturing directional audio. The present disclosure further relates to a rendering apparatus configured to modify the directional characteristics of received directional audio in response to received spatial data.

Claims

1. A device having a microphone system including one or more microphones for capturing audio, or a device connected to said microphone system, said device: Receiving unit: The steps include receiving directional audio, which includes one or more directional source signals captured by the microphone system; A receiving unit configured to perform the steps of receiving metadata related to the microphone system, wherein the metadata includes spatial data of the microphone system, the spatial data indicates at least the spatial orientation and / or spatial position of the microphone system, and includes at least one of the yaw or azimuth angle, pitch angle, roll angle, or spatial coordinates of the microphone system; The steps include defining a rotation / translation sequence based on the aforementioned spatial data; A step of generating modified directional audio by modifying at least a portion of the directional audio by multiplying the directional audio by the rotation / translation sequence, wherein the directional characteristics of the directional audio are modified in response to the spatial data of the microphone system; A computing unit configured to perform the steps of encoding the modified directional audio into digital audio data; The system includes a transmitting unit configured to transmit the aforementioned digital audio data, Device.

2. The apparatus according to claim 1, wherein the spatial orientation of the microphone system is represented by a parameter that describes a rotational motion / orientation of one degree of freedom (DoF) in the spatial data.

3. The apparatus according to claim 1, wherein the spatial orientation of the microphone system is represented by a parameter that describes 3DoF rotational motion / orientation in the spatial data.

4. The apparatus according to claim 1, wherein the spatial data of the microphone system is represented in 6DoF.

5. The apparatus according to claim 1, wherein the received directional audio includes audio containing directional metadata.

6. The computing unit is further configured to encode at least a portion of the metadata, including the spatial data of the microphone system, into the digital audio data. The transmission unit is configured to transmit the digital audio data including the metadata. The apparatus according to claim 1.

7. The apparatus according to claim 6, wherein the receiving unit is further configured to receive a first instruction indicating to the computing unit whether to include in the digital audio data at least a portion of the metadata, including the spatial data of the microphone system, the computing unit operates accordingly.

8. The apparatus according to claim 6, wherein the receiving unit is further configured to receive a second instruction indicating to the computing unit which parameters(s) of the spatial data of the microphone system to include in the digital audio data, and the computing unit operates accordingly.

9. The apparatus according to claim 7, wherein the transmitting unit is configured to transmit the digital audio data to a further device, and instructions relating to a first command and / or a second command are received from the further device.

10. The apparatus according to claim 1, wherein the receiving unit is further configured to receive metadata including a timestamp indicating the capture time of the directional audio, and the computing unit is configured to encode the timestamp in the digital audio data.

11. The computing unit further: The modified directional audio based on the spatial data of the microphone system is downmixed using a downmix matrix; It is configured to encode the downmixed directional audio and the downmix matrix into the digital audio data. The downmixing includes adjusting the beamforming behavior of the modified directional audio based on the spatial data of the microphone system. The apparatus according to claim 1.

12. The apparatus according to claim 1, wherein the apparatus is implemented in a virtual reality (VR) gear or augmented reality (AR) gear having the microphone system and a head tracking device configured to determine the spatial data of the apparatus in 3 to 6 degrees of freedom.

13. A device for rendering an audio signal, the device being: A receiving unit configured to receive digital audio data; A decoding unit configured to decode received digital audio data into directional audio and metadata, wherein the metadata includes at least spatial data indicating the spatial orientation and / or spatial position of a microphone system; Based on the aforementioned spatial data, a rotation / translation sequence is defined; The directional characteristics of the directional audio are corrected by multiplying the directional audio by the rotation / translation sequence; Configured to render corrected directional audio Having a rendering unit, Device.

14. The apparatus according to claim 13, wherein the microphone system includes one or more microphones for capturing the directional audio, and the rendering unit modifies the directional characteristics of the directional audio to reproduce, at least partially, the audio environment of the microphone system.

15. The apparatus according to claim 13, wherein the spatial data includes parameters that describe rotational motion / orientation with one degree of freedom (DoF).

16. The apparatus according to claim 13, wherein the spatial data includes parameters that describe rotational motion / orientation in 3DoF.

17. The apparatus according to claim 13, wherein the decoded directional audio includes audio containing directional metadata.

18. The apparatus according to claim 13, further comprising a transmitting unit configured to transmit a command to a further device from which the digital audio data is received, the command indicating to the further device which parameters (one or more) the rotation data should include.

19. The apparatus according to claim 13, wherein the decoding unit is further configured to extract a timestamp indicating the capture time of the directional audio from the digital audio data.

20. The apparatus according to claim 13, wherein the spatial data includes spatial coordinates, and the rendering unit is further configured to adjust the volume of the rendered audio based on the spatial coordinates.

21. The apparatus according to claim 13, implemented in a virtual reality (VR) gear or augmented reality (AR) gear having a head tracking device configured to measure the spatial orientation and spatial position of the apparatus in 6DoF.

22. The apparatus according to claim 13, wherein the rendering unit is configured for binaural audio rendering.

23. A system having the first device according to claim 1, configured to transmit digital audio data to the second device according to claim 13, wherein the system is configured for audio and / or video conferencing.

24. The system according to claim 23, wherein the first device further comprises a video recording unit configured to encode recorded video into digital video data and transmit the digital video data to the second device, and the second device further comprises a display for displaying the decoded digital video data.

25. A system having the first device according to claim 1, configured to transmit digital audio data to a second device, wherein the second device is: A receiving unit configured to receive the aforementioned digital audio data; A decoding unit configured to decode received digital audio data into directional audio and metadata, wherein the metadata includes spatial data comprising at least one from a list of yaw or azimuth angle, pitch angle, roll angle, and spatial coordinates; It has a rendering unit for rendering audio, The rendering unit, when the second device further receives encoded video data from the first device: The spatial data is used to correct the directional characteristics of the directional audio, It is configured to render corrected directional audio; When the second device does not receive encoded video data from the first device, the rendering unit performs the following actions: The aforementioned directional audio is configured to render, system.

26. A non-temporary computer-readable medium that, when executed by one or more processors, stores instructions causing the one or more processors to perform an action, wherein the action is: The step of receiving directional audio, which includes one or more directional source signals, from a microphone system including one or more microphones; A step of receiving metadata related to the microphone system, wherein the metadata includes spatial data of the microphone system, the spatial data indicates at least the spatial orientation and / or spatial position of the microphone system, and includes at least one of the yaw or azimuth angle, pitch angle, roll angle, or spatial coordinates of the microphone system; The steps include defining a rotation / translation sequence based on the aforementioned spatial data; A step of generating modified directional audio by modifying at least a portion of the directional audio by multiplying the directional audio by the rotation / translation sequence, wherein the directional characteristics of the directional audio are modified in response to the spatial data of the microphone system; The stage of encoding the corrected directional audio into digital audio data; The step of transmitting the aforementioned digital audio data and Computer-readable media, including [specific text / data].

27. Decoding the received digital audio data into directional audio is performed by the decoding unit: The received digital audio data is decoded into downmixed audio, a downmix matrix, and the metadata. The downmix matrix is ​​used to configure the downmixed audio to be upmixed into the directional audio. The apparatus according to claim 13.

28. The apparatus according to claim 14, wherein the rendering unit modifies the directional characteristics of the directional audio by reapplying at least a portion of the acoustic scene rotation compensated in the capture device including the microphone system.

Images