Method and apparatus for external directional control of session immersive audio

Abstract

Various embodiments describe methods and apparatus for external directional control of session immersive audio. An example apparatus includes at least one processor; and at least one non-transitory memory storing instructions that, when executed with the at least one processor, cause the apparatus to perform: receive external orientation information; receiving external directional control information; selecting external directional modification; modifying a rendering orientation of the spatially rendered orientation according to the selected external orientation modification; and rendering the spatial audio according to the modified rendering orientation.

Description

Technical Field

[0001] The exemplary and non-limiting embodiments generally relate to immersive audio, and more specifically to external directional control for conversational immersive audio. Background Technology

[0002] Providing immersive audio is known. Summary of the Invention

[0003] The following description is intended to be illustrative only. It is not intended to limit the scope of the claims.

[0004] An apparatus includes: at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus to: receive external orientation information; receive external orientation control information; select an external orientation modification; modify the rendering orientation of a spatially rendered orientation according to the selected external orientation modification; and render spatial audio according to the modified rendering orientation.

[0005] The example device may further include, wherein the external orientation control information includes interactive signals for enabling, disabling, or freezing applications that provide external orientation information.

[0006] The example device may further include, wherein the device is further configured to: receive head tracking information, and wherein spatial audio is further rendered based on the head tracking information.

[0007] The example apparatus may further include external orientation control information including a head orientation flag for freezing the head orientation to the last rendered spatial audio orientation. In one embodiment, head tracking is frozen / held, and when the freeze is applied, the audio is rendered with head tracking disabled and a default or corresponding head tracking forward orientation. For example, this could be a previous head tracking orientation when the previous state was enabled or a default forward orientation when the previous state was disabled, in which case there is no change to the rendering.

[0008] The example device may further include, wherein the external orientation control information includes a head orientation flag for enabling or disabling head tracking.

[0009] The example device may further include a feature where head tracking is disabled when the head orientation flag is set to disable head tracking, and audio is rendered with head tracking disabled and the default or corresponding forward orientation.

[0010] The example device may further include a feature where head tracking is enabled when the head orientation flag is set to enable head tracking, and audio is rendered based on head orientation.

[0011] The example device may further include, wherein the device is further configured to: receive reference data for defining scene orientation; and apply scene orientation data.

[0012] The example device may further include, wherein the device is further configured to: combine external orientation data with head tracking data.

[0013] The example apparatus may further include, wherein the apparatus is further configured to: receive an external orientation activation time for indicating whether external orientation information will be applied in the current frame or in a future frame.

[0014] The example apparatus may further include, wherein the apparatus is further configured to: receive combined information affecting external orientation information, and wherein the combined information includes a combination of two or more of the following: scene orientation information provided by the sending user equipment (UE), device orientation data provided by the sending UE, or local scene orientation information provided by the receiving UE.

[0015] A method includes: receiving external orientation information; receiving external orientation control information; selecting an external orientation modification; modifying the rendering orientation of a spatially rendered orientation according to the selected external orientation modification; and rendering spatial audio according to the modified rendering orientation.

[0016] The example method may further include, whereby the external orientation control information includes interactive signals for enabling, disabling, or freezing the external orientation information. In one embodiment, the external orientation control information may be applied through end-user interaction or via application preferences. Examples of interactive signals include, but are not limited to, interactive Boolean signals.

[0017] The example method may further include receiving head tracking information, wherein spatial audio is further rendered based on the head tracking information.

[0018] The example method may further include, whereby the external orientation control information includes a head orientation flag for freezing the head orientation to the last rendered spatial audio orientation. In one embodiment, head tracking is frozen / held, and when the freeze is applied, the audio is rendered with head tracking disabled and a default or corresponding head tracking forward orientation. For example, this could be a previous head tracking orientation when the previous state was enabled or a default forward orientation when the previous state was disabled, in which case there is no change to the rendering.

[0019] The example method may further include, whereby the external orientation control information includes head orientation flags for enabling or disabling head tracking.

[0020] The example method may further include, where head tracking is disabled when the head orientation flag is set to disable head tracking, and audio is rendered with head tracking disabled and the default or corresponding forward orientation.

[0021] The example method may further include, where head tracking is enabled when the head orientation flag is set to enable head tracking, and audio is rendered based on head orientation.

[0022] The example method may further include: receiving reference data for defining scene orientation; and applying scene orientation data.

[0023] The example method could further include combining external orientation data with head tracking data.

[0024] The example method may further include receiving an external orientation activation time that indicates whether the external orientation information will be applied in the current frame or a future frame.

[0025] The example method may further include: receiving combined information that affects external orientation information, wherein the combined information includes a combination of two or more of the following: scene orientation information provided by the sending user equipment (UE), device orientation data provided by the sending UE, or local scene orientation information provided by the receiving UE.

[0026] Another device includes: a component for receiving external orientation information; a component for receiving external orientation control information; a component for selecting an external orientation modification; a component for modifying the rendering orientation of the spatially rendered orientation according to the selected external orientation modification; and a component for rendering spatial audio according to the modified rendering orientation.

[0027] The example apparatus may further include components for performing one or more methods as described in any of the preceding paragraphs. Attached Figure Description

[0028] The foregoing embodiments and other features are described in the following description in conjunction with the accompanying drawings, wherein:

[0029] Figure 1 This is a block diagram of a possible, non-limiting example system in which exemplary embodiments can be practiced;

[0030] Figure 2 This shows an example of IVAS renderer head tracking data processing.

[0031] Figure 3 An example scenario is shown.

[0032] Figure 4 Here is another example scenario.

[0033] Figure 5 An example of directional processing according to one embodiment is shown.

[0034] Figure 6 Showing according to Figure 5 This is a further extension of the proposed targeted processing described in [the document / article].

[0035] Figure 6b An example is shown. Figure 6 The alternative solution.

[0036] Figure 7 Show Figure 6 Additional descriptions of examples outside the systems described herein.

[0037] Figure 8 It is an example device that can be implemented in hardware to enable external directional control for conversational immersive audio based on the examples described herein.

[0038] Figure 9 This shows a schematic representation of a non-volatile storage medium.

[0039] Figure 10 This is an example method for implementing the example described herein, based on one embodiment. Detailed Implementation

[0040] The following abbreviations, which can be found in the specification and / or drawings, are defined as follows: 3GPP Third Generation Partnership Project 5G fifth generation 5GC 5G Core Network AMF access and mobility management functions CU Central Unit DU Distributed Unit eNB (or eNodeB) evolved Node B (e.g., LTE base station) E-UTRA evolved universal terrestrial radio access, i.e., LTE radio access technology gNB (or gNodeB) is used as a base station for 5G / NR, that is, it provides NR user plane and control plane protocol termination to the UE and is connected to the 5GC node via the NG interface. I / F interface IVAS Immersive Voice and Audio Services LTE Long Term Evolution MAC Media Access Control MME Mobility Management Entity MPEG Moving Picture Experts Group Ng or NG next generation ng-eNB or NG-eNB next-generation eNB NR New Radio N / W or NW network PDCP Packet Data Convergence Protocol PHY physical layer RAN Radio Access Network RLC Radio Link Control RRH Remote Radio Header RRC Radio Resource Control RTCPRTP Control Protocol RTP Real-Time Transport Protocol RU radio unit Rx receiver SDAP Service Data Adaptation Protocol SGW Service Gateway SMF Session Management Function Tx transmitter UE (User Equipment) (e.g., wireless equipment, typically mobile equipment) UPF User Face Functions VR Virtual Reality

[0041] Go to Figure 1 The accompanying drawing illustrates a block diagram of possible, non-limiting examples in which these examples can be practiced. A user equipment (UE) 110, a radio access network (RAN) node 170, and a network unit 190 are shown. Figure 1In the example, User Equipment (UE) 110 wirelessly communicates with Wireless Network 100. The UE is a wireless device capable of accessing Wireless Network 100. UE 110 includes one or more processors 120, one or more memories 125, and one or more transceivers 130 interconnected via one or more buses 127. Each of the one or more transceivers 130 includes a receiver Rx 132 and a transmitter Tx 133. The one or more buses 127 may be address, data, or control buses and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optic cables, or other optical communication devices. The one or more transceivers 130 are connected to one or more antennas 128. The one or more memories 125 include computer program code 123. UE 110 includes a module 140, which includes one or both of portions 140-1 and / or 140-2. Module 140 can be implemented in various ways. Module 140 may be implemented in hardware as module 140-1, for example, as part of one or more processors 120. Module 140-1 can also be implemented as an integrated circuit or through other hardware such as a programmable gate array. In another example, module 140 can be implemented as module 140-2, which is implemented as computer program code 123 and executed by one or more processors 120. For example, one or more memories 125 and computer program code 123 can be configured to perform one or more operations as described herein with the user device 110 using one or more processors 120. UE 110 communicates with RAN node 170 via wireless link 111.

[0042] In this example, RAN node 170 is a base station that provides access to wireless network 100 for wireless devices such as UE 110. RAN node 170 can be, for example, a base station for 5G (also known as New Radio (NR)). In 5G, RAN node 170 can be an NG-RAN node, which is defined as a gNB or ng-eNB. A gNB is a node that provides NR user plane and control plane protocol termination to the UE and is connected to the 5GC (such as network unit 190) via an NG interface. An ng-eNB is a node that provides E-UTRA user plane and control plane protocol termination to the UE and is connected to the 5GC via an NG interface. An NG-RAN node can include multiple gNBs, and can also include a central unit (CU) (gNB-CU) 196 and a distributed unit (DU) (gNB-DU), of which DU 195 is shown. Note that a DU can include or be coupled to and control a radio unit (RU). The gNB-CU is a logical node that hosts the RRC, SDAP, and PDCP protocols of the gNB or the en-gNB that control the operation of one or more gNB-DUs. The gNB-CU terminates the F1 interface connected to the gNB-DU. The F1 interface is shown as reference numeral 198, although reference numeral 198 also shows links between remote units of RAN node 170 and centralized units of RAN node 170, such as the link between gNB-CU 196 and gNB-DU 195. The gNB-DU is a logical node that hosts the RLC, MAC, and PHY layers of the gNB or en-gNB, and its operation is partially controlled by the gNB-CU. One gNB-CU supports one or more cells. A cell is supported by only one gNB-DU. The gNB-DU terminates the F1 interface 198 connected to the gNB-CU. Note that DU 195 is considered to include transceiver 160, for example as part of an RU, but some examples of this could have transceiver 160 as part of a separate RU, for example, under the control of and connected to DU 195. RAN node 170 could also be an eNB (evolved Node B) base station for LTE (Long Term Evolution), or any other suitable base station or node.

[0043] RAN node 170 includes one or more processors 152, one or more memories 155, one or more network interfaces (N / WI / F) 161, and one or more transceivers 160 interconnected via one or more buses 157. Each of the one or more transceivers 160 includes a receiver Rx 162 and a transmitter Tx 163. The one or more transceivers 160 are connected to one or more antennas 158. The one or more memories 155 include computer program code 153. CU 196 may include processor 152, memory 155, and network interface 161. Note that DU 195 may also contain its own memory and processor, and / or other hardware, but these are not shown.

[0044] RAN node 170 includes module 150, which includes one or both of portions 150-1 and / or 150-2. Module 150 can be implemented in various ways. Module 150 can be implemented in hardware as module 150-1, for example, as part of one or more processors 152. Module 150-1 can also be implemented as an integrated circuit or through other hardware such as a programmable gate array. In another example, module 150 can be implemented as module 150-2, which is implemented as computer program code 153 and executed by one or more processors 152. For example, one or more memories 155 and computer program code 153 are configured, together with one or more processors 152, to enable RAN node 170 to perform one or more operations as described herein. Note that the functionality of module 150 can be distributed, for example, distributed between DU 195 and CU 196, or implemented only in DU 195.

[0045] One or more network interfaces 161 communicate on the network, for example, via links 176 and 131. Two or more gNBs 170 can communicate using, for example, link 176. Link 176 can be wired or wireless, or both, and can, for example, implement an Xn interface for 5G, an X2 interface for LTE, or other suitable interfaces for other standards.

[0046] One or more buses 157 may be address, data, or control buses and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, optical fiber or other optical communication equipment, wireless channels, etc. For example, one or more transceivers 160 may be implemented as a Remote Radio Header (RRH) 195 for LTE or a Distributed Unit (DU) 195 for a gNB implementation of 5G, wherein other units of the RAN node 170 may be physically located differently from the RRH / DU, and one or more buses 157 may be partially implemented as, for example, optical fiber or other suitable network connections to connect other units of the RAN node 170 (e.g., Central Unit (CU), gNB-CU) to the RRH / DU 195. Reference numeral 198 also indicates those suitable network links.

[0047] Note that the description in this document indicates that a "cell" performs functions; however, it should be clear that the equipment forming the cell will perform these functions. A cell constitutes part of a base station. That is, each base station can have multiple cells. For example, for a single carrier frequency and associated bandwidth, there can be three cells, each covering one-third of a 360-degree area, so that the coverage area of ​​a single base station is approximately elliptical or circular. Furthermore, each cell can correspond to a single carrier, and a base station can use multiple carriers. Therefore, if there are three 120-degree cells per carrier and two carriers, the base station has a total of six cells.

[0048] Wireless network 100 may include one or more network units 190, which may include core network functions and provide connectivity to another network (e.g., a telephone network and / or a data communication network (e.g., the Internet)) via one or more links 181. Such core network functions for 5G may include Access and Mobility Management Functions (AMF) and / or User Plane Functions (UPF) and / or Session Management Functions (SMF). Such core network functions for LTE may include MME (Mobility Management Entity) / SGW (Serving Gateway) functions. These are merely exemplary functions that can be supported by network unit 190, and it should be noted that both 5G and LTE functions may be supported. RAN node 170 is coupled to network unit 190 via link 131. Link 131 may, for example, be implemented as an NG interface for 5G, or an S1 interface for LTE, or other suitable interfaces for other standards. Network unit 190 includes one or more processors 175, one or more memories 171, and one or more network interfaces (N / WI / F) 180 interconnected via one or more buses 185. One or more memories 171 include computer program code 173. The one or more memories 171 and computer program code 173 are configured, together with one or more processors 175, to cause the network unit 190 to perform one or more operations.

[0049] Wireless network 100 can implement network virtualization, which is the process of combining hardware and software network resources and network functions into a single software-based management entity, i.e., a virtual network. Network virtualization involves platform virtualization, often combined with resource virtualization. Network virtualization is classified as external virtualization, which combines many networks or network segments into virtual units, or internal virtualization, which provides network-like functionality to software containers on a single system. Note that at certain levels, hardware such as processors 152 or 175 and memories 155 and 171 are still used to implement the virtualized entities created by network virtualization, and these virtualized entities also produce technical effects.

[0050] Computer-readable storage devices 125, 155, and 171 can be of any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as semiconductor-based storage devices, flash memory, magnetic storage devices and systems, optical storage devices and systems, fixed memory, and removable memory. Computer-readable storage devices 125, 155, and 171 can be components for performing storage functions. Processors 120, 152, and 175 can be of any type suitable for the local technical environment and, as non-limiting examples, can include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture. Processors 120, 152, and 175 can be components for performing functions such as controlling UE 110, RAN node 170, and other functions described herein.

[0051] Typically, various embodiments of user equipment 110 may include, but are not limited to, cellular phones such as smartphones, tablet computers, personal digital assistants (PDAs) with wireless communication capabilities, portable computers with wireless communication capabilities, image capture devices such as digital cameras with wireless communication capabilities, gaming devices with wireless communication capabilities, music storage and playback devices with wireless communication capabilities, internet devices that allow wireless internet access and browsing, tablet computers with wireless communication capabilities, and portable units or terminals incorporating combinations of these functions.

[0052] One or more of modules 140-1, 140-2, 150-1, and 150-2 can be configured to implement external directional control for conversational immersive audio. Computer program code 173 can also be configured to implement external directional control for conversational immersive audio.

[0053] 3GPP IVAS

[0054] The Immersive Voice and Audio Services (IVAS) codec is an extension of the 3GPP Enhanced Voice Services (EVS) codec and is designed for new immersive voice and audio services based on 4G / 5G. These immersive services include, for example, immersive voice and audio for virtual reality (VR). The versatile audio codec is expected to handle the encoding, decoding, and rendering of voice, music, and general audio. It is expected to support various input formats, such as channel-based and scene-based inputs. It is also expected to enable session services with low latency and support high fault tolerance under various transport conditions. The IVAS codec is expected to provide an internal renderer and at least components that work in conjunction with a suitable external renderer. For example, an external renderer interface may exist that is designated as part of the IVAS codec.

[0055] The IVAS codec also includes features such as directional processing at the receiving UE. Directional processing is important for rendering spatial audio as intended. For example, applying head tracking to binaural rendering is one type of spatial audio directional processing. Various embodiments have been proposed for methods and apparatuses to implement directional processing in immersive audio conversations. This is relevant to immersive audio conversation scenarios as well as immersive audiovisual conversation scenarios.

[0056] Various embodiments specify how to combine different orientation data (including head tracking data and modifications thereof) in a practical immersive audio renderer (e.g., an IVAS internal renderer) and propose additional control methods for interactively selecting whether and how one or more orientations should be applied to the rendering. This interactive selection can be based on user or application preferences. For example, consider a use case where audio scene orientation in the receiving UE's rendering is switched based on the camera selection at the sending UE, where the user's current operating mode determines the desired audio behavior.

[0057] IVAS Open Collaborative Software Baseline

[0058] The current IVAS codec baseline renderer version provides means / components for orientation processing by acquiring reference data to define scene orientation in specific use cases where, for example, device position 'r' can be used as the front of the scene. Such reference data can typically be, for example, orientation data or a vector between two points. In alternative operating modes, the head tracking unit in the IVAS codec baseline renderer treats the user's average orientation as a natural reference to the slowly evolving forward direction. For example, for a user riding a bus, the movement of vehicles turning at an intersection is slowly filtered out so that they do not significantly affect the rendered scene orientation. Various embodiments consider orientation signals / data separate from head tracking data. For example, the reference orientation based on the receiver UE's position involves head tracking data rather than the scene data of interest itself. These also need to be considered in separate processing steps so that the reference data or averaging operations present in the IVAS codec baseline renderer do not incorrectly modify these data.

[0059] Figure 2An example of head tracking data processing in the IVAS renderer is shown. In this example, there may be, for example, three modes 202, where the first mode corresponds to providing reference data 204 indicating the forward orientation for rendering. For example, this could be the position of a mobile device relative to the user's face, headphones, or equivalent. The second mode could correspond to averaging the reference, for example, a mechanism to remove slowly evolving directional changes from the head tracking data. The third mode could be a default mode or a normal mode, which does not apply any reference data or average the head tracking or orientation data 206. The head tracker processing 208 includes a rotator that can apply a final rotation of the scene for rendering binaural audio 210 via headphones. The final orientation data 212 can also be provided from the renderer for external use as needed.

[0060] Figure 3 An example scenario is illustrated. In this example scenario, the sending UE 302 is making a video call with the receiving UE 304. The sending UE 302 is using its rear camera 306 to record video. In this example, it is assumed that the audio capture 303 is positioned forward away from the user (e.g., from the back of the sending UE 302 looking outwards). The sending UE 302 captures an audio source 308 in a forward-right direction, and the sender's own voice is captured by the on-device microphone. Subsequently, the sending UE 302 transmits the corresponding video and audio to the receiving UE 304. Everything works as expected. This is because the captured audio is oriented in a specific direction with reference to "forward," and the rear camera captures the visual scene where the audio source is visible in the correct direction. There is no incompatibility between the capture and transmission of audio and video content.

[0061] Figure 4 Another example scenario is shown. In this example scenario, the sender switches to using the front-facing camera 402 to capture video (e.g., the face of the sender user 403). However, the sender UE 404 itself remains in contact with... Figure 3 The scene described is in the same location and orientation, only the active camera is switched. The front direction of audio capture 405 remains the same (from the back of the transmitting UE 404 outwards). Therefore, when the audio should actually come from the rear left direction, the receiver is in the same position and orientation as described. Figure 3 The phantom source is heard from the same direction as the scene. There is a mismatch between the received video and audio.

[0062] This inconsistency between audio and visual representations needs to be addressed in a way that is effective in all scenarios. The lack of mechanisms to properly handle such changes can complicate the consumption of conversational audio, as well as spatial audio in audiovisual scenarios, for end users. This could adversely affect the usability of the IVAS standard and thus negatively impact its widespread adoption.

[0063] In one example, to correct the direction of the audio source, the receiver can be instructed to point to the active camera, allowing the receiver to then perform the correct rotation of the audio. Alternatively, the sender can readjust the capture direction based on which camera is active.

[0064] Various embodiments relate to a method for external orientation processing, wherein external orientation control information and external orientation information are provided to enable user intent-related applications of the external orientation information, and to achieve spatial audio rendering orientation aligned with the user intent. An example method may include the following operations: Receive externally directed information; Receive external directional control information; Select external targeted modification; Modify the rendering orientation of the spatially rendered orientation based on the selected external orientation modification; and Render spatial audio based on the modified render orientation.

[0065] In one embodiment, external orientation modification can be selected by the user. In another embodiment, external orientation modification can be automatically selected, for example, by the UE or application preferences.

[0066] In one embodiment, the selection of external orientation modification is performed based on the received external orientation control information.

[0067] In one embodiment, external targeting control information may include interactive signals for enabling, disabling, or freezing applications that provide external targeting information. In one embodiment, external targeting control information may be applied through end-user interaction or via application preferences.

[0068] In another embodiment, the external orientation control information includes information for enabling the head orientation to be frozen as the final rendered spatial audio orientation.

[0069] In another embodiment, controlling external orientation information includes information for enabling / disabling head tracking to stop head tracking and rendering audio under disabled head tracking and default or corresponding forward orientation.

[0070] In one embodiment, enabling can mean applying, for example, an external orientation or interpolating based on it when there is no future activation or interpolation. For example, in a simple case, the external orientation is a 45-degree left yaw, and the new value is a 55-degree left turn --> the final orientation applied based on this input is a 55-degree left yaw rotation relative to the format's default orientation. In one embodiment, contributions from head tracking data can still modify this for other purposes in rendering.

[0071] In one embodiment, disabling can mean not applying, for example, an external orientation. For instance, if the external orientation is 45 degrees left yaw, and the new value is 55 degrees left yaw, then the final orientation applied based on this input is a 0-degree rotation relative to the format's default orientation. In one embodiment, contributions from head tracking data can still modify this for other purposes in rendering.

[0072] In one embodiment, freezing can mean applying the existing external orientation and keeping it unchanged until otherwise indicated (via enabling / disabling). For example, if the external orientation is 45 degrees left yaw and the new value is 55 degrees left yaw, the final orientation applied based on this input is 45 degrees left yaw. In one embodiment, contributions from head tracking data can still modify this for other purposes in rendering.

[0073] Figure 5 An example of orientation processing according to one embodiment is shown. In this example, in addition to head tracking or orientation data 506, at least one external orientation 502 is provided to the renderer 504. The head tracking or orientation data 506 can be as follows: Figure 2 It is processed similarly as described in the text. In one embodiment, a head orientation flag is added that indicates at least enabling, disabling, and / or freezing of head tracking 508. When head tracking is enabled, head tracking processing 510 is therefore performed as described in the text. Figure 2 The process is executed as described above. When head tracking is disabled, the default forward direction is maintained in front of the user (as long as head tracking data is considered), and head rotation does not affect rendering. In some embodiments, reference data 512 input may be applied. On the other hand, freezing means setting the current head tracking direction to the currently maintained forward direction without any further head tracking data affecting rendering. Therefore, when the user is looking to the right and freezing is applied, the right-hand side direction remains in front of the user regardless of subsequent changes in head orientation.

[0074] In one embodiment, the rotator function 514 is now separated from the head tracker processing 510. This is because the external orientation data 502 needs to be combined with the output of the head tracker processing 510 by the orientation data combiner 516. The orientation data combiner 516 may further accept the external orientation flag 509 for head tracking data as described above as input, for example, a command to enable, disable, or freeze the application of external orientation information. The function is similar, except that it now controls the application of external orientation 502. External orientation 502 can change significantly with any update, and therefore in some cases, it may be desirable to smooth out this change. For this purpose, an interpolation function 518 is provided. In various embodiments, the interpolation function 518 can be controlled more precisely. However, in this embodiment, the interpolation function 518 is described only in terms of enabling and disabling. For example, when the external orientation changes by 180 degrees (yaw) according to the camera view signaling described above, it may generally be desirable to apply this change fairly instantly. Therefore, in this use case, the orientation interpolation flag can be set to disabled.

[0075] Figure 6 It shows according to Figure 5 This is a further extension of the proposed orientation processing described in the example. Additional input describing the external orientation activation time 602 is added. Activation time 602 indicates that the currently provided external orientation information 502 will be applied at the current frame (activation time 0) or at a future frame (e.g., activation time 50 corresponding to 50 frames). For example, in any pre-planned section of professionally created "pre-made" content or live content, there may be an intention to change the orientation at a specific point in time. For example, consider an immersive audio transmission with a panel discussion in front of a studio audience. The panel may initially appear in front of the listener, spanning an arc from left to right. At some point, there may be a Q&A section of the discussion where the scene is instructed to rotate so that the panel is on the user's left and the audience on the user's right. Such a change in orientation can be planned and indicated at a future time (e.g., a future frame). Alternatively, it may be desirable to make this change smooth. In this case, a new orientation can be sent in each frame, or interpolation can be enabled. Therefore, renderer 504 calculates the target orientation at each frame based on the current orientation, external orientation 502, and external orientation activation time 602. Orientation interpolation is therefore particularly useful when the future orientation is known. Otherwise, interpolation can provide a smooth orientation change, but this smooth orientation may lag, which may negatively impact the user's (listener's) perception in some cases. When the external orientation interpolation flag 518 is set to disabled and has a non-zero orientation activation time, this corresponds to a wait command. For example, the renderer waits X frames until the orientation changes.

[0076] Figure 6b An embodiment is shown. Figure 6 An alternative is provided. In this embodiment, in addition to the external orientation flag 509, external orientation 502, and external orientation activation time 602, the head orientation flag 508 is also provided as an input to the orientation data combiner 516. Furthermore, in this embodiment, the orientation interpolation flag 518 is provided as an optional input to the orientation data combiner 516. Therefore, in this embodiment, all operations not inherent to the head tracker position and / or operations related to user interaction can be processed in the orientation data combiner 516.

[0077] Figure 7 It shows Figure 6 Additional descriptions of examples outside the system described herein. This example describes an example source that may at least affect external orientation data 502. For example, the sending UE may send scene orientation data 702 and device orientation data 704; and the receiving UE may provide local scene orientation data 706. The sending of scene orientation data 702, device orientation data 704, and local scene orientation data 706, as well as any other suitable orientation data, may be externally combined by an external orientation data combiner 708 located outside the renderer 504.

[0078] Figure 8 This is an example device 800, which can be implemented in hardware to enable external directional control for conversational immersive audio based on the examples described herein. Device 800 includes at least one processor 802 and at least one non-transitory memory 804 including computer program code 805, wherein the at least one memory 804 and the computer program code 805 are configured, together with the at least one processor 802, to enable device 800 to perform external directional control for conversational immersive audio based on the examples described herein.

[0079] The device 800 optionally includes a display 808 that can be used to display content during rendering. The device 800 optionally includes one or more network (NW) interfaces (I / F) 810. The one or more NW I / F 810s can be wired and / or wireless and communicate via the Internet / (one or more) other networks through any communication technology. The one or more NW I / F 810s can include one or more transmitters and one or more receivers. The one or more N / W I / F 810s can include standard, well-known components such as amplifiers, filters, frequency converters, (de)modulators, and (one or more) encoder / decoder circuits, as well as one or more antennas.

[0080] Examples of device 800 may be remote, virtual, or cloud devices. Device 800 may be an encoder or decoder, or both. At least one memory 804 may be implemented using any suitable data storage technology, such as semiconductor-based storage devices, flash memory, magnetic storage devices and systems, optical storage devices and systems, fixed memory, and removable memory. At least one memory 804 may include a database for storing data. Device 800 does not need to include every feature mentioned, or may include other features. Device 800 may correspond to Figure 1 Another embodiment of the apparatus shown (including UE110, RAN node 170 or (one or more) network units 190) or this embodiment.

[0081] Another example of device 800 is an IVAS internal renderer. Yet another example of this device includes external renderers, for example, renderers that may not be specified as part of the IVAS standard and can be used in place of one or more IVAS standard renderers. This can allow for manufacturer differentiation, etc.

[0082] Figure 9 A schematic representation of a non-volatile storage medium 900a (e.g., a computer / optical disc (CD) or digital multifunction disc (DVD)) and 900b (e.g., a Universal Serial Bus (USB) memory stick) is shown, which, when executed by a processor, allows the processor to perform one or more steps of the methods described herein.

[0083] Figure 10 This is an example method 1000 that implements the example described herein according to one embodiment. At 1002, method 1000 includes receiving external orientation information. At 1004, method 1000 includes receiving external orientation control information. At 1006, method 1000 includes selecting an external orientation modification. In one embodiment, the external orientation modification is selected based on the received external orientation control information. At 1008, method 1000 includes modifying the rendering orientation of the spatially rendered orientation according to the selected external orientation modification. At 1010, method 1000 includes rendering spatial audio according to the modified rendering orientation.

[0084] In one embodiment, the external orientation control information includes a flag for freezing the head orientation to the last rendered spatial audio orientation. In another embodiment, the external orientation control information includes a flag for enabling or disabling head tracking. When the flag is set to disable head tracking, head tracking is disabled, and audio is rendered with head tracking disabled and the default or corresponding front orientation. When the flag is set to enable head tracking, head tracking is enabled, and audio is rendered based on the head orientation.

[0085] Method 1000 may further include receiving head tracking information and rendering spatial audio further based on the head tracking information.

[0086] Method 1000 can be performed using the apparatus described herein (e.g., apparatus 800, etc.).

[0087] As mentioned above, Figure 10 The flowcharts include apparatus (e.g., 800), methods, and computer program products according to certain example embodiments. It will be understood that each block of the flowchart, and combinations of blocks in the flowchart, can be implemented by various means, such as hardware, firmware, processors, circuitry, and / or other devices associated with the execution of software including one or more computer program instructions. For example, one or more of the processes described above can be embodied by computer program instructions. In this regard, computer program instructions embodying the processes described above can be stored in the memory (e.g., 125 or 804) of an apparatus employing embodiments of the invention and executed by the processing circuitry (e.g., 120 or 802) of that apparatus. As will be understood, any such computer program instructions can be loaded onto a computer or other programmable device (e.g., hardware) to produce a machine such that the resulting computer or other programmable device performs the functions specified in the flowchart blocks. These computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable device to operate in a particular manner such that the instructions stored in the computer-readable storage medium produce an article of art whose execution performs the functions specified in the flowchart blocks. These computer program instructions may also be loaded onto a computer or other programmable device to cause a series of operations to be performed on the computer or other programmable device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable device, provide operations for implementing the functions specified in the flowchart box.

[0088] Therefore, a computer program product is defined in such a case, wherein computer program instructions, such as computer-readable program code portions, are stored by at least one non-transitory computer-readable storage medium, and the computer program instructions, such as computer-readable program code portions, are configured to, at execution time, such as in combination with... Figure 10 One or more flowcharts perform the functions described above. In other embodiments, computer program instructions, such as computer-readable program code portions, do not need to be stored or otherwise embodied by a non-transitory computer-readable storage medium, but may instead be embodied by a temporary medium having computer program instructions, such as computer-readable program code portions, which are still configured to perform the functions described above when executed.

[0089] Therefore, the boxes in a flowchart support combinations of means / components for performing a specified function, as well as combinations of operations for performing the specified function. It will also be understood that one or more boxes in a flowchart, and combinations of boxes in a flowchart, can be implemented by a dedicated hardware-based computer system or a combination of dedicated hardware and computer instructions to perform the specified function.

[0090] In some embodiments, specific operations in the above-described operations may be modified or further amplified. Furthermore, in some embodiments, additional optional operations may be included. Modifications, additions, or amplifications of the above-described operations can be performed in any order and in any combination.

[0091] In the preceding text, some example implementations have been described using bitstream syntax. However, it should be understood that the corresponding structures and / or computer programs may reside at the encoder for generating bitstreams and / or reside at the decoder for decoding bitstreams.

[0092] In the foregoing description of example embodiments with reference to the encoder, it is important to understand that the resulting bitstream and decoder have corresponding elements. Similarly, in the description of example embodiments with reference to the decoder, it is important to understand that the encoder has a structure and / or computer program for generating the bitstream that will be decoded by the decoder.

[0093] Benefiting from the teachings set forth in the foregoing description and related drawings, those skilled in the art will conceive of many modifications and other embodiments of the invention set forth herein. Therefore, it should be understood that the invention is not limited to the specific embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims. Furthermore, while the foregoing description and related drawings describe exemplary embodiments in the context of specific example combinations of elements and / or functions, it should be understood that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, combinations of elements and / or functions different from those explicitly described above are also contemplated as being set forth in some of the appended claims. Therefore, this specification is intended to include all such alternatives, modifications, and variations falling within the scope of the appended claims. Although specific terms are used herein, they are used only in a general and descriptive sense and not for limiting purposes.

[0094] It should be understood that the foregoing description is merely illustrative. Various alternatives and modifications can be devised by those skilled in the art. For example, the features described in the various dependent claims can be combined with each other in any suitable combination (one or more). Furthermore, features from the different embodiments described above can be selectively combined to form new embodiments. Therefore, this specification is intended to include all such alternatives, modifications, and variations that fall within the scope of the appended claims.

[0095] References to "computer," "processor," etc., should be understood to encompass not only computers with different architectures (such as single / multiprocessor architectures and sequential (von Neumann) / parallel architectures) but also special-purpose circuits (such as field-programmable gate arrays (FPGAs), special-purpose circuits (ASICs), signal processing devices, and other processing circuits). References to computer programs, instructions, code, etc., should be understood to encompass software or firmware used in programmable processors, such as programmable content of hardware devices, such as instructions for processors, or configuration settings for fixed-function devices, gate arrays, or programmable logic devices.

[0096] As used herein, the term “non-transitory” refers to a limitation on the medium itself (i.e., tangible, non-signal), rather than a limitation on the persistence of data storage (e.g., RAM versus ROM).

[0097] As used in this application, the term "circuit" may refer to one or more or all of the following: (a) Pure hardware circuit implementation (e.g., implementation in pure analog and / or digital circuits), and (b) A combination of hardware circuitry and software, such as (if applicable): (i) A combination of analog and / or digital hardware circuitry with software / firmware, and (ii) Any part of a hardware processor (including a digital signal processor) with software, software, and memory, which work together to enable a device such as a mobile phone or server to perform various functions, and (iii) Hardware circuitry and / or processors (such as microprocessors or parts thereof) that require software (e.g. firmware) to operate, but may be absent when no software is required to operate.

[0098] This definition of "circuit" applies to all uses of the term in this application (including in any claim). As another example, as used in this application, the term "circuit" also covers only hardware circuitry or processors (or processors) or portions thereof and their accompanying software and / or firmware implementations. The term "circuit" also covers, for example (and if applicable to a particular claim element), baseband integrated circuits or processor integrated circuits used in mobile devices or similar integrated circuits in servers, cellular network devices or other computing or network devices.

[0099] It should be understood that the foregoing description is merely illustrative. Various alternatives and modifications can be devised by those skilled in the art. For example, the features described in the various dependent claims can be combined with each other in any suitable combination (one or more). Furthermore, features from the different embodiments described above can be selectively combined to form new embodiments. Therefore, this specification is intended to include all such alternatives, modifications, and variations that fall within the scope of the appended claims.

Claims

1. An apparatus comprising: At least one processor; as well as At least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the device to perform: Receive externally directed information; Receive external directional control information; Select external targeted modification; Modify the rendering orientation of the spatially rendered orientation based on the selected external orientation modification; as well as Render spatial audio based on the modified render orientation.

2. The apparatus according to claim 1, wherein, The external orientation control information includes interactive signals for enabling, disabling, or freezing the application of the external orientation information.

3. The apparatus according to claim 1, wherein, The device further enables the following: receiving head tracking information, wherein the spatial audio is further rendered based on the head tracking information.

4. The apparatus according to claim 1, wherein, The external orientation control information includes a head orientation flag used to freeze the head orientation as the final rendered spatial audio orientation.

5. The apparatus according to claim 1 or 4, wherein, The external orientation control information includes head orientation flags for enabling or disabling head tracking.

6. The apparatus according to claim 4, wherein, When the head orientation flag is set to disable head tracking, head tracking is disabled, and audio is rendered with head tracking disabled and the default forward orientation.

7. The apparatus according to claim 4, wherein, When the head orientation flag is set to enable head tracking, head tracking is enabled, and audio is rendered based on head orientation.

8. The apparatus according to claim 1, wherein, Further, the device: Receive reference data used to define scene orientation; and Apply the scenario-oriented data.

9. The apparatus according to any one of the preceding claims, wherein, The device further combines the external orientation data with the head tracking data.

10. The apparatus according to any one of the preceding claims, wherein, The device further enables the following: receiving an external orientation activation time for indicating whether the external orientation information will be applied in the current frame or a future frame.

11. The apparatus according to any one of the preceding claims, wherein, The device further enables the following: receiving combined information affecting the external orientation information, wherein the combined information includes a combination of two or more of the following: scene orientation information provided by the sending user equipment (UE), device orientation data provided by the sending UE, or local scene orientation information provided by the receiving UE.

12. A method comprising: Receive externally directed information; Receive external directional control information; Select external targeted modification; Modify the rendering orientation of the spatially rendered orientation based on the selected external orientation modification; as well as Render spatial audio based on the modified render orientation.

13. The method according to claim 12, wherein, The external orientation control information includes interactive signals for enabling, disabling, or freezing the application of the external orientation information.

14. The method of claim 12, further comprising: Head tracking information is received, and the spatial audio is further rendered based on the head tracking information.

15. The method according to claim 12, wherein, The external orientation control information includes a head orientation flag used to freeze the head orientation as the final rendered spatial audio orientation.

16. The method according to claim 12 or 15, wherein, The external orientation control information includes head orientation flags for enabling or disabling head tracking.

17. The method according to claim 16, wherein, When the head orientation flag is set to disable head tracking, head tracking is disabled, and audio is rendered with head tracking disabled and the default or corresponding forward orientation.

18. The method according to claim 16, wherein, When the head orientation flag is set to enable head tracking, head tracking is enabled, and audio is rendered based on head orientation.

19. The method of claim 12, comprising: Receive reference data used to define scene orientation; as well as Apply the scenario-oriented data.

20. The method according to any one of the preceding claims, further comprising: Combine the external orientation data with the head tracking data.

21. The method according to any one of the preceding claims, further comprising: Receive the external orientation activation time, which indicates whether the external orientation information will be applied in the current frame or a future frame.

22. The method according to any one of the preceding claims, further comprising: Receive combined information affecting the external orientation information, wherein the combined information includes a combination of two or more of the following: scene orientation information provided by the sending user equipment (UE), device orientation data provided by the sending UE, or local scene orientation information provided by the receiving UE.

23. An apparatus comprising: Components used to receive external orientation information; Components used to receive external directional control information; Used to select components for externally oriented modifications; This component is used to modify the rendering orientation of the spatially rendered orientation based on the selected external orientation. as well as A component used to render spatial audio based on a modified render orientation.

24. The apparatus according to claim 23, wherein, The apparatus further includes components for performing one or more methods according to claims 12 to 22.

Images