Method and apparatus for external direction control for conversational immersive audio

Abstract

Various embodiments describe methods and apparatus for external direction control for conversational immersive audio. An exemplary apparatus includes at least one processor and at least one non-temporary memory for storing instructions, which, when executed by at least one processor, cause the apparatus to: receive external direction information, receive external direction control information, select an external direction modification, modify the rendering direction of the spatially rendered direction according to the selected external direction modification, and render the spatial audio according to the modified rendering direction.

Description

Technical Field

[0001] Exemplary and non - limiting embodiments generally relate to immersive audio, and more particularly to external orientation control for conversational immersive audio.

Background Art

[0002] It is known to provide immersive audio.

Summary of the Invention

[0003] The following summary is merely intended to be an example. It is not intended to limit the scope of the claims.

[0004] The apparatus comprises at least one processor and at least one non - transient memory storing instructions, which when executed by the at least one processor, cause the apparatus to: receive external orientation information, receive external orientation control information, select an external orientation modification, modify a rendering orientation of a spatially rendered direction according to the selected external orientation modification, and render spatial audio according to the modified rendering orientation.

[0005] Exemplary apparatus may further include that the external orientation control information includes an interactive interaction signal for enabling, disabling, or freezing the application of the external orientation information.

[0006] Exemplary apparatus may further include that the apparatus is further caused to receive head - tracking information and the spatial audio is further rendered based on the head - tracking information.

[0007] An exemplary device may further include external directional control information containing a head direction flag for freezing the head direction in the last rendered spatial audio direction. In one embodiment, head tracking is frozen / maintained, and the audio is rendered using the default or corresponding head-tracked forward direction at the time the disabled head tracking and freeze operation is applied. For example, this may be either the previously head-tracked direction when the previous state was enabled, or the default forward direction when the previous state was disabled, in which case there is no change to the rendering.

[0008] The exemplary device may further include external direction control information that includes a head direction flag to enable or disable head tracking.

[0009] The exemplary device may further include, when the head direction flag is set to disable head tracking, head tracking is disabled and audio is rendered using disabled head tracking and the default or corresponding forward direction.

[0010] The exemplary device may further include the fact that when the head orientation flag is set to enable head tracking, head tracking is enabled and audio is rendered based on the head orientation.

[0011] The exemplary apparatus may further include receiving reference data for defining scene orientation and applying scene orientation data.

[0012] The exemplary apparatus may further include the ability to combine external orientation data with head tracking data.

[0013] The exemplary device may further include receiving an external directional activation time to indicate that external directional information is applied to the current frame or a future frame.

[0014] The exemplary device may further include the device receiving combined information that affects external directional information, wherein the combined information includes two or more combinations of scene directional information provided by a sender user device (UE), device directional data provided by the sender UE, or local scene directional information provided by a receiver UE.

[0015] The method includes receiving external direction information, receiving external direction control information, selecting an external direction correction, correcting the rendering direction of the spatially rendered direction according to the selected external direction correction, and rendering the spatial audio according to the corrected rendering direction.

[0016] An exemplary method may further include the external direction control information including interactive interaction signals to enable, disable, or freeze the application of the external direction information. In one embodiment, the external direction control information may be applied by end-user interaction or via application preferences. Examples of interactive interaction signals include, but are not limited to, interactive Boolean interaction signals.

[0017] The exemplary method may further include receiving head tracking information, and the spatial audio is further rendered based on the head tracking information.

[0018] An exemplary method may further include external directional control information including a head direction flag for freezing the head direction in the last rendered spatial audio direction. In one embodiment, head tracking is frozen / maintained, and the audio is rendered using the default or corresponding head-tracked forward direction at the time the disabled head tracking and freeze operation is applied. For example, this may be either the previously head-tracked direction when the previous state was enabled, or the default forward direction when the previous state was disabled, in which case there is no change to the rendering.

[0019] The exemplary method may further include the external direction control information including a head direction flag to enable or disable head tracking.

[0020] An exemplary method may further include, when the head direction flag is set to disable head tracking, head tracking is disabled and audio is rendered using disabled head tracking and the default or corresponding forward direction.

[0021] An exemplary method may further include, when the head orientation flag is set to enable head tracking, head tracking is enabled and audio is rendered based on head orientation.

[0022] An exemplary method may further include receiving reference data to define the scene orientation and applying the scene orientation data.

[0023] The exemplary method may further include combining external orientation data with head tracking data.

[0024] The exemplary method may further include receiving an external direction activation time for indicating that external direction information is applied to a current frame or a future frame.

[0025] The exemplary method may further include receiving combined information that affects the external direction information, the combined information including a combination of two or more of scene direction information provided by a transmitting user equipment (UE), device direction data provided by the transmitting UE, or local scene direction information provided by a receiving UE.

[0026] Another apparatus includes means for receiving external direction information, means for receiving external direction control information, means for selecting an external direction modification, means for modifying a rendering direction of a spatially rendered direction according to the selected external direction modification, and means for rendering spatial audio according to the modified rendering direction.

[0027] The exemplary apparatus may further include means for the apparatus to perform one or more of the methods described in any of the above sections.

[0028] The above embodiments and other features are described in the following description considered in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0029] [Figure 1] FIG. is a block diagram of one conceivable and non-limiting exemplary system in which exemplary embodiments may be implemented. [Figure 2] FIG. shows an example of IVAS renderer head tracking data handling. [Figure 3] FIG. shows an exemplary scenario. [Figure 4] FIG. shows another exemplary scenario. [Figure 5] FIG. shows an example of direction handling according to one embodiment. [Figure 6] Figure 5 shows a further extension of the proposed directional handling, as explained in the diagram. [Figure 6b] This figure shows an alternative to Figure 6 according to one embodiment. [Figure 7] Figure 6 shows a further example illustrating a system beyond the one described. [Figure 8] This is a diagram of an exemplary device, which may be implemented in hardware, capable of performing external direction control for conversational immersive audio, based on the examples described herein. [Figure 9] This figure shows a schematic representation of a non-volatile memory medium. [Figure 10] This is a diagram illustrating an exemplary method for carrying out an example described herein, according to one embodiment. [Modes for carrying out the invention]

[0030] The following abbreviations, which may be found in this specification and / or in the drawings, are defined below: 3GPP Third Generation Partnership Project 5G (fifth generation) 5GC 5G core network AMF (Access and Mobility Management Function) CU (Central Unit) DU (Distributed Unit) eNB (or eNode B) Evolved Node B (e.g., LTE base station) E-UTRA stands for evolved universal terrestrial radio access, or LTE radio access technology. gNB (or gNode B) is a base station for 5G / NR, i.e., a node that provides NR user plane and control plane protocol termination toward the UE and is connected to the 5GC via the NG interface. I / F Interface IVAS (Immersive Voice and Audio Service) LTE Long Term Evolution MAC Medium Access Control MME (Mobility Management Entity) MPEG Moving Picture Experts Group ng or NG new generation ng-eNB or NG-eNB: New generation eNB NR New Radio N / W or NW Network PDCP (Packet Data Convergence Protocol) PHY (Physical Layer) RAN (Radio Access Network) RLC radio link control RRH Remote Radio Head RRC (Radio Resource Control) RTCP (Retard Transition Control Protocol) RTP (Real-time Transport Protocol) RU (Radio Unit) Rx receiver SDAP (Service Data Adaptation Protocol) SGW (Serving Gateway) SMF session management function Tx transmitter UE (User Equipment) (e.g., wireless, typically mobile devices) UPF User Plane Function VR (Virtual Reality)

[0031] Referring to Figure 1, this figure shows a block diagram of one possible and non-limiting example in which the example may be implemented. A user device (UE) 110, a radio access network (RAN) node 170, and a network element 190 are shown. In the example of Figure 1, the user device (UE) 110 is in wireless communication with a wireless network 100. The UE is a wireless device that can access the wireless network 100. The UE 110 includes one or more processors 120, one or more memories 125, and one or more transceivers 130 interconnected through one or more buses 127. Each of the one or more transceivers 130 includes a receiver Rx132 and a transmitter Tx133. The one or more buses 127 may be an address, data, or control bus and may include any interconnection mechanism such as a series of lines on a motherboard or integrated circuit, fiber optics, or other optical communication equipment, and so on. One or more transceivers 130 are connected to one or more antennas 128. One or more memories 125 contain computer program code 123. The UE 110 includes a module 140 which contains one or both of components 140-1 and / or 140-2, which may be executed in multiple ways. Module 140 may be executed in hardware as module 140-1, such that it is executed as part of one or more processors 120. Module 140-1 may be executed as an integrated circuit or through other hardware such as a programmable gate array. In another example, module 140 may be executed as computer program code 123 and as module 140-2, which is executed by one or more processors 120. For example, one or more memories 125 and computer program code 123 may be configured by one or more processors 120 to cause the user device 110 to perform one or more of the operations described herein. The UE 110 communicates with the RAN node 170 via a wireless link 111.

[0032] In this example, RAN node 170 is a base station that provides access to the wireless network 100 by wireless devices such as UE 110. RAN node 170 may also be a base station for 5G, also known as New Radio (NR). In 5G, RAN node 170 may be an NG-RAN node, defined as a gNB or ng-eNB. A gNB is a node that provides NR user plane and control plane protocol termination to the UE and is connected to the 5GC (e.g., network element 190) via an NG interface. An ng-eNB is a node that provides E-UTRA user plane and control plane protocol termination to the UE and is connected to the 5GC via an NG interface. An NG-RAN node may include multiple gNBs, which may similarly include a central unit (CU) (gNB-CU) 196 and distributed units (DUs) (gNB-DU) whose DU 195 is indicated. Note that a DU may include or be coupled to and control a radio unit (RU). A gNB-CU is a logical node that hosts the RRC, SDAP, and PDCP protocols of a gNB, or the RRC and PDCP protocols of an en-gNB, and controls the operation of one or more gNB-DUs. A gNB-CU terminates an F1 interface connected to a gNB-DU. The F1 interface is shown as reference 198, but reference 198 also indicates links between remote elements of RAN node 170 and centralized elements of RAN node 170, such as between gNB-CU196 and gNB-DU195. A gNB-DU is a logical node that hosts the RLC, MAC, and PHY layers of a gNB or en-gNB, and its operation is partially controlled by a gNB-CU. A single gNB-CU supports one or more cells. A single cell is supported by only one gNB-DU. A gNB-DU terminates an F1 interface 198 connected to a gNB-CU.DU195 is thought to include transceiver 160 as part of a RU, for example, but it should be noted that some examples of this may have transceiver 160 as part of a separate RU, for example, under the control of and connected to DU195. RAN node 170 may also be an eNB (Evolutionary Node B) base station for LTE (Long-Term Evolution), or any other suitable base station or node.

[0033] RAN node 170 includes one or more processors 152, one or more memories 155, one or more network interfaces (N / WI / F) 161, and one or more transceivers 160 interconnected through one or more buses 157. Each of the one or more transceivers 160 includes a receiver Rx162 and a transmitter Tx163. One or more transceivers 160 are connected to one or more antennas 158. One or more memories 155 contain computer program code 153. CU 196 may include a processor 152, memory 155, and network interface 161. DU 195 may also include its own one / more memories and processors, and / or other hardware, but these are not shown.

[0034] RAN node 170 includes module 150 comprising one or both of components 150-1 and / or 150-2, which may be executed in multiple ways. Module 150 may be executed in hardware as module 150-1, such as being executed as part of one or more processors 152. Module 150-1 may also be executed as an integrated circuit or through other hardware such as a programmable gate array. In another example, module 150 may be executed as computer program code 153 and as module 150-2, which is executed by one or more processors 152. For example, one or more memories 155 and computer program code 153 are configured by one or more processors 152 to cause RAN node 170 to perform one or more of the operations described herein. Note that the functionality of module 150 may be distributed, such as being distributed between DU 195 and CU 196, or it may be executed independently within DU 195.

[0035] One or more network interfaces 161 communicate over the network, for example, via links 176 and 131. Two or more gNBs 170 may communicate using link 176, for example. Link 176 may be wired, wireless, or both, and may implement, for example, an Xn interface for 5G, an X2 interface for LTE, or other suitable interface for other standards.

[0036] One or more buses 157 may be address, data, or control buses and may include any interconnection mechanisms such as a series of lines on a motherboard or integrated circuit, fiber optics, or other optical communication equipment, wireless channels, and the like. For example, one or more transceivers 160 may be running as distributed units (DUs) 195 for a remote radio head (RRH) 195 for LTE or a gNB implementation for 5G, with other elements of the RAN node 170 possibly located in a different location from the RRH / DU, and one or more buses 157 may be running in part, for example, as a fiber optic cable or other suitable network connection to connect other elements of the RAN node 170 (e.g., a central unit (CU), gNB-CU) to the RRH / DU 195. Reference 198 also indicates those suitable network links.

[0037] It should be noted that the description herein indicates that a "cell" performs a function, but it should be clear that the equipment forming the cell will perform the function. A cell constitutes part of a base station. That is, there can be multiple cells for one base station. For example, there may be three cells for a single carrier frequency and associated bandwidth, and each cell covers one-third of a 360° area such that the coverage area of ​​a single base station covers an approximately oval or circular shape. Furthermore, each cell can correspond to a single carrier, and a base station may use multiple carriers. Thus, if there are three 120° cells for one carrier and two carriers exist, the base station has a total of six cells.

[0038] The wireless network 100 may include one or more network elements 190, each of which may include core network functions and provide connectivity to further networks such as telephone networks and / or data communication networks (e.g., the Internet) via one or more links 181. Such core network functions for 5G may include access and mobility management functions (AMF) and / or user plane functions (UPF) and / or session management functions (SMF). Such core network functions for LTE may include MME (Mobility Management Entity) / SGW (Serving Gateway) functions. These are merely illustrative functions that may be supported by the network elements 190, and it should be noted that both 5G and LTE functions may be supported. The RAN node 170 is coupled to the network elements 190 via link 131. Link 131 may be, for example, an NG interface for 5G or an S1 interface for LTE or other suitable interface for other standards. The network element 190 includes one or more processors 175, one or more memories 171, and one or more network interfaces (N / WI / F) 180 interconnected through one or more buses 185. One or more memories 171 contain computer program code 173. One or more memories 171 and the computer program code 173 are configured to cause one or more operations to be performed by one or more processors 175 on the network element 190.

[0039] The wireless network 100 may perform network virtualization, which is the process of combining hardware and software network resources and network functions into a single software-based management entity, a virtual network. Network virtualization is often combined with platform virtualization, which is associated with resource virtualization. Network virtualization can be classified as either external, combining many networks or parts of a network into a virtual unit, or internal, providing network-like functions to a software container on a single system. It should be noted that the virtualized entities resulting from network virtualization are also performed at some level using hardware such as processors 152 or 175 and memory 155 and 171, and similarly, such virtualized entities produce technical effects.

[0040] Computer-readable memories 125, 155, and 171 may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. Computer-readable memories 125, 155, and 171 may also be means for performing storage functions. Processors 120, 152, and 175 may be of any type suitable for the local technical environment and may include, in non-limiting examples, one or more general-purpose computers, dedicated computers, microprocessors, digital signal processors (DSPs), and processors based on multicore processor architectures. Processors 120, 152, and 175 may be means for performing functions such as controlling the UE 110, the RAN node 170, and other functions described herein.

[0041] In general, various embodiments of the user device 110 may include, but are not limited to, a cellular phone such as a smartphone, a tablet, a personal digital assistant (PDA) with wireless communication capabilities, a portable computer with wireless communication capabilities, an image capture device such as a digital camera with wireless communication capabilities, a gaming device with wireless communication capabilities, a music storage and playback appliance with wireless communication capabilities, an internet appliance that enables wireless internet access and browsing, a tablet with wireless communication capabilities, and a portable unit or terminal incorporating a combination of such functions.

[0042] One or more of modules 140-1, 140-2, 150-1, and 150-2 may be configured to perform external direction control for conversational immersive audio. Computer program code 173 may also be configured to perform external direction control for conversational immersive audio.

[0043] 3GPP IVAS

[0044] The Immersive Voice and Audio Services (IVAS) codec is an extension of the 3GPP Enhanced Voice Services (EVS) codec and is intended for new immersive voice and audio services over 4G / 5G. Such immersive services include, for example, immersive voice and audio for virtual reality (VR). The multipurpose audio codec is expected to handle encoding, decoding, and rendering of speech, music, and general-purpose audio. The multipurpose audio codec is expected to support various input formats, such as channel-based and scene-based inputs. The multipurpose audio codec is also expected to operate with low latency to enable conversational services, as well as support high error robustness under various transmission conditions. The IVAS codec is expected to provide an internal renderer and, at least, a means of working with a suitable external renderer. For example, an external renderer interface may exist that is posited as part of the IVAS codec.

[0045] The IVAS codec also includes features such as direction handling at the receiving UE. Direction handling is crucial for presenting spatial audio as intended. For example, applying head tracking to binaural rendering is one type of spatial audio direction handling. Various embodiments propose methods and apparatus for enabling direction handling in conversational immersive audio calls. This is important for both audio conversational immersive audio scenarios and immersive audiovisual conversation scenarios.

[0046] Various embodiments specify how different orientation data (including head tracking data and modifications applied thereto) are processed in combination with a practical immersive audio renderer (e.g., an IVAS internal renderer), and propose further control methods for interactively selecting whether and how one or more orientations should be applied for rendering. This interactive selection can be based on user or application preferences. For example, consider a use case where audio scene orientation is switched in a receiving UE rendering based on camera selection in a transmitting UE, where the user's current operating mode influences the desired audio behavior.

[0047] IVAS Publication Collaboration Software Baseline

[0048] The current IVAS codec baseline renderer version provides means for orientation handling by incorporating reference data to define the scene orientation, for example, in certain use cases where the device position "r" may be used as the front of the scene. Such reference data may generally be, for example, orientation data or a vector between two points. In an alternative mode of operation, the head tracking unit in the IVAS codec baseline renderer considers the user's average orientation as a slowly evolving natural reference for the forward direction. For example, if the user is on a bus, the movement of the vehicle turning at an intersection is slowly filtered out so that the movement does not significantly affect the rendered scene orientation. Various embodiments consider orientation signals / data that are separate from the head tracking data. For example, a reference orientation based on the receiving UE position is related to the head tracking data, rather than the scene data of interest itself. These also need to be considered in a separate processing step so that the reference data or averaging operation present in the IVAS codec baseline renderer does not inaccurately modify this data.

[0049] Figure 2 shows an example of IVAS renderer head tracking data handling. In this example, for example, there may be three modes 202, the first mode corresponding to providing reference data 204 that indicates the forward direction for rendering. For example, this may be the position of the mobile device relative to the user's face, headphones, or equivalent. The second mode may correspond to averaging the reference, for example, a mechanism to remove slowly progressing directional changes from the head tracking data. The third mode may be a default mode or normal mode in which no reference data or averaging is applied to the head tracking or directional data 206. The head tracker processing 208 includes a rotator which may apply a final rotation of the scene for rendering binaural audio 210 across the headphones. Final directional data 212 may also be provided from the renderer for external use as needed.

[0050] Figure 3 illustrates an exemplary scenario. In this exemplary scenario, the transmitting UE302 has a video call with the receiving UE304. The transmitting UE302 is recording video using its rear camera 306. In this example, it is assumed that the forward direction for audio capture 303 is far from the user (e.g., outward from the back of the transmitting UE302). The transmitting UE302 captures the audio source 308 in the right-front direction, while its own voice is captured by the in-device microphone. The transmitting UE302 then sends the corresponding video and audio to the receiving UE304. Everything works as expected. This is because the captured audio is oriented with the reference "front" in a specific direction, and the rear camera captures the visual scene with the audio source appearing in the correct direction. There are no inconsistencies between audio and video content capture and transport.

[0051] Figure 4 shows another exemplary scenario. In this exemplary scenario, the transmitter switches to using the front camera 402 to capture video (e.g., the face of the transmitter user 403). However, the transmitter UE 404 itself remains in the same position and orientation as in the scenario described in Figure 3, and only the active camera is switched. The forward direction for audio capture 405 remains the same (outward from the back of the transmitter UE 404). Therefore, the receiver hears a phantom (virtual) sound source in the same direction as in the scenario of Figure 3, when the audio would actually be coming from the left rear. A mismatch exists between the received video and the received audio.

[0052] This inconsistency in audio and visual representation needs to be addressed to ensure it works well in all scenarios. The lack of a mechanism for properly handling such variations can complicate the consumption of spatial audio in conversational and audiovisual scenarios for end users. This may negatively impact the usefulness of the IVAS standard and, consequently, its widespread adoption.

[0053] In one example, to correct the direction of the audio source, the active camera may be instructed by the receiver, which may then perform the correct rotation for the audio. Alternatively, the transmitter may readjust the front of the capture direction based on which camera is active.

[0054] Various embodiments relate to methods for external direction handling, in which external direction control information and external direction information are provided to enable user intent-dependent applications of the external direction information and to achieve spatial audio rendering directions consistent with user intent. Exemplary methods are as follows: Receiving external information, Receiving external direction control information, Choosing external direction correction, Modify the rendering direction of the spatially rendered direction according to the selected external direction correction, This may include rendering spatial audio according to the modified rendering direction.

[0055] In one embodiment, the external direction correction may be selected by the user. In another embodiment, the external direction correction may be automatically selected, for example, by the UE or application preference.

[0056] In one embodiment, selecting an external direction correction is performed based on received external direction control information.

[0057] In one embodiment, the external direction control information may include interactive interaction signals to enable, disable, or freeze the application of the external direction information. In one embodiment, the external direction control information may be applied by end-user interaction or via application preferences.

[0058] In another embodiment, the external direction control information includes information to enable freezing the head direction in the last rendered spatial audio direction.

[0059] In another embodiment, the control external direction information includes information to enable / disable head tracking in order to stop head tracking and render audio using disabled head tracking and the default or corresponding forward direction.

[0060] In one embodiment, enabling may mean applying the external direction as is (for example, when no future launch or interpolation exists) or interpolating based on it. For example, in a simple case, the external direction is 45° left yaw and the new value is 55° left way → what is applied to the final direction based on this input is a 55° left yaw rotation relative to the format's default direction. In one embodiment, contributions from head tracking data may also modify this into something else for rendering.

[0061] In one embodiment, disabling may mean, for example, not applying the external direction. For example, if the external direction is 45° left yaw, and the new value is 55° left yaw, then what is applied to the final direction based on this input is a 0° rotation relative to the format's default direction. In one embodiment, a contribution from head tracking data may also modify this to something else for rendering.

[0062] In one embodiment, "freeze" may mean applying an existing external direction and keeping that direction there until otherwise instructed (by enabling / disabling). For example, if the external direction is 45° left yaw and the new value is 55° left yaw, then the final direction applied based on this input is 45° left yaw. In one embodiment, a contribution from head tracking data may also modify this into something else for rendering.

[0063] Figure 5 shows an example of direction handling according to one embodiment. In this example, at least one external direction 502 is provided to the renderer 504 in addition to head tracking or direction data 506. The head tracking or direction data 506 may be processed similarly as described in Figure 2. In one embodiment, head tracking is accompanied by a head direction flag indicating at least enable, disable, and / or freeze 508. When head tracking is enabled, head tracking processing 510 is therefore performed as described in Figure 2. When head tracking is disabled, the default forward is maintained in front of the user (as long as head tracking data is taken into account), and head turns do not affect rendering. In some embodiments, a reference data 512 input may be applied. Freezing, on the other hand, means setting the current head-tracked direction as the currently maintained forward without any further head tracking data affecting rendering. So, if the user looks to the right and the freeze operation is applied, the right-hand direction is maintained in front of the user, regardless of subsequent head direction changes.

[0064] In one embodiment, the rotator function 514 is hereby decoupled from the head tracker processing 510. This is because the external direction data 502 needs to be combined with the output of the head tracker processing 510 by the direction data combiner 516. The direction data combiner 516 may further accept the external direction flag 509 described above for the head tracking data as an input, e.g., a command, to enable, disable, or freeze the application of the external direction information. The function is similar, and now only, the direction data combiner 516 controls the application of the external direction 502. The external direction 502 may change significantly in any update, and therefore, in some cases, it may be desirable to smooth this change. For this reason, an interpolation function 518 is provided. The interpolation function 518 may be controlled more precisely in various embodiments. However, in this embodiment, the interpolation function 518 is described only in terms of enabling and disabling. For example, if the external direction changes by 180° (yaw) according to the camera view signaling described above, it may generally be desirable to apply the change immediately. Therefore, the direction interpolation flag may be set to be disabled in such use cases.

[0065] Figure 6 shows a further extension of the proposed orientation handling described in Figure 5. In this example, an additional input describing the external orientation trigger time 602 is added. The trigger time 602 indicates that the currently provided external orientation information 502 is applied to the current frame (trigger time 0) or a future frame (e.g., trigger time 50 corresponds to frame 50). For example, in professionally created "canned" content, or in any pre-planned portion of real-time content, there may be an intention to change orientation at a specific point in time. For example, consider immersive audio transmission of a panel discussion in front of a studio audience. The panel may initially appear before the listeners, spanning an arc from left to right. After a while, there may be a Q&A portion of the discussion, at which point the scene is instructed to rotate so that the panel is on the left of the user and the audience is on the right of the user. This change of orientation may be planned and instructed in future time (e.g., a future frame). On the other hand, it may be desirable to smooth this change. In this case, the new direction may be transmitted within each frame, or rather, interpolation may be enabled. Therefore, the renderer 504 calculates the target direction in each frame based on the current direction, the external direction 502, and the external direction activation time 602. Thus, direction interpolation is particularly useful when the future direction is known. Otherwise, interpolation may provide a smooth direction change, but such a smooth direction lags behind, which in some cases may be perceived poorly by the user (listener). When the external direction interpolation flag 518 is set to disabled with a non-zero direction activation time, this corresponds to a wait command. For example, the renderer waits for X frames until the direction changes.

[0066] Figure 6b shows an alternative to Figure 6 according to one embodiment. In this embodiment, in addition to the external direction flag 509, external direction 502, and external direction activation time 602, the head direction flag 508 is also provided as an input to the direction data combiner 516. Furthermore, in this embodiment, the direction interpolation flag 518 is provided as an optional input to the direction data combiner 516. Thus, all operations and / or operations related to user interactivity that are not inherently part of the head tracker position may be processed in the direction data combiner 516 in several embodiments.

[0067] Figure 7 provides a further exemplary description beyond the system described in Figure 6. This example describes an exemplary sound source that may affect at least the external directional data 502. For example, the transmitting UE may transmit scene directional data 702 and device directional data 704; and the receiving UE may provide local scene directional data 706. The transmitted scene directional data 702, device directional data 704, and local scene directional data 706, as well as any other suitable directional data, may be combined externally by an external directional data combiner 708 located outside the renderer 504.

[0068] Figure 8 shows an exemplary device 800, which may be hardware-operated, that can be made to perform external direction control for conversational immersive audio, based on the examples described herein. The device 800 comprises at least one processor 802 and at least one non-temporary memory 804 containing computer program code 805, the at least one memory 804 and the computer program code 805 being configured, based on the examples described herein, to cause the at least one processor 802 to cause the device 800 to perform external direction control for conversational immersive audio.

[0069] The device 800 optionally includes a display 808 which may be used to display content during rendering. The device 800 optionally includes one or more network interfaces (I / F) 810. The NW I / F 810 is wired and / or wireless and may communicate over the Internet / other networks by any communication technique. The NW I / F 810 may comprise one or more transmitters and one or more receivers. The NW I / F 810 may comprise well-known standard components such as amplifiers, filters, frequency converters, modulators (demodulators), and encoder / decoder circuits, as well as one or more antennas.

[0070] An example of the device 800 may be a remote, virtual, or cloud device. The device 800 may be a coder or a decoder or both a coder and a decoder. At least one memory 804 may be implemented using any suitable data storage technology, e.g., semiconductor-based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. At least one memory 804 may include a database for storing data. The device 800 does not have to include each of the features described, or may include other features as well. The device 800 corresponds to or may correspond to another embodiment of the device shown in Figure 1, including UE 110, RAN node 170, or network element 190.

[0071] Another example of the device 800 may be an IVAS internal renderer. Yet another example of the device includes an external renderer, for example, a renderer that does not have to be specified as part of the IVAS standard and may be used in place of an IVAS standard renderer. This may allow for manufacturer differentiation of capabilities and similar.

[0072] Figure 9 shows schematic representations of non-volatile memory media 900a (e.g., a computer / compact disc (CD) or digital versatile disc (DVD)) and 900b (e.g., a universal serial bus (USB) memory stick) that store instructions and / or parameters 902 that, when executed by the processor, enable the processor to perform one or more steps of the method described herein.

[0073] Figure 10 shows an exemplary method 1000 that performs an example described herein according to one embodiment. In 1002, method 1000 includes receiving external direction information. In 1004, method 1000 includes receiving external direction control information. In 1006, method 1000 includes selecting an external direction modification. In one embodiment, the external direction modification is selected based on the received external direction control information. In 1008, method 1000 includes modifying the rendering direction of the spatially rendered direction according to the selected external direction modification. In 1010, method 1000 includes rendering spatial audio according to the modified rendering direction.

[0074] In one embodiment, the external orientation control information includes a flag for freezing the head orientation in the direction of the last rendered spatial audio. In another embodiment, the external orientation control information includes a flag for enabling or disabling head tracking. When the flag is set to disable head tracking, head tracking is disabled and the audio is rendered using disabled head tracking and the default or corresponding forward orientation. When the flag is set to enable head tracking, head tracking is enabled and the audio is rendered based on the head orientation.

[0075] Method 1000 may further include receiving head tracking information and rendering spatial audio based on the head tracking information.

[0076] Method 1000 may be carried out by the apparatus described herein, for example, apparatus 800 and similar.

[0077] As described above, Figure 10 includes flowcharts of a device (e.g., 800), a method, and a computer program product according to a particular exemplary embodiment. It will be understood that each block of the flowchart and combinations of blocks within the flowchart may be executed by various means, e.g., hardware, firmware, processors, circuitry, and / or other devices related to the execution of software, including one or more computer program instructions. For example, one or more of the procedures described above may be embodied by computer program instructions. In this regard, computer program instructions that embodied the procedures described above may be stored in the device's memory (e.g., 125 or 804) and executed by the device's processing circuitry (e.g., 120 or 802), using one embodiment of the present invention. As recognized, any such computer program instructions may be loaded into a computer or other programmable device (e.g., hardware) to produce a machine such that the resulting computer or other programmable device performs the functions specified in the flowchart blocks. These computer program instructions may also be stored in computer-readable memory, and the computer program instructions may instruct a computer or other programmable device to function in a particular way such that the instructions stored in computer-readable memory produce a product whose execution performs a function specified in a flowchart block. The computer program instructions may also be loaded into a computer or other programmable device so that a series of operations are performed on the computer or other programmable device, generating a process of computer execution such that the instructions executed on the computer or other programmable device provide the operations to perform a function specified in a flowchart block.

[0078] Therefore, a computer program product is defined in an instance in which computer program instructions, such as computer-readable program code portions, are stored in at least one non-temporary computer-readable storage medium, and in such instance, the computer program instructions, such as computer-readable program code portions, are configured to perform the functions described above when executed, such as in conjunction with the flowchart in Figure 10. In other embodiments, computer program instructions, such as computer-readable program code portions, do not need to be stored in a non-temporary computer-readable storage medium or otherwise embodied, but may instead be embodied in a temporary medium, in which case the computer program instructions, such as computer-readable program code portions, are still configured to perform the functions described above when executed.

[0079] Therefore, the blocks in a flowchart support a specified function and a combination of means for performing a combination of actions to perform that specified function. It will also be understood that one or more blocks in a flowchart and combinations of blocks within a flowchart may be executed by a dedicated hardware-based computer system or a combination of dedicated hardware and computer instructions to perform the specified function.

[0080] In some embodiments, certain operations of the above operations may be modified or further amplified. Furthermore, in some embodiments, additional optional operations may be included. Modifications, additions, or augments to the above operations may be carried out in any order and in any combination.

[0081] In the above, several exemplary embodiments were described with the help of bitstream syntax. However, it should be understood that corresponding structures and / or computer programs may exist in an encoder for generating a bitstream and / or in a decoder for decoding a bitstream.

[0082] In the above, if an exemplary embodiment is described with reference to an encoder, it should be understood that the resulting bitstream and decoder have corresponding elements in them. Similarly, if an exemplary embodiment is described with reference to a decoder, it should be understood that the encoder has a structure and / or computer program for generating a bitstream that is decoded by the decoder.

[0083] Many modifications and other embodiments of the inventions described herein will be conjured upon those skilled in the art to which these inventions relate, and who benefit from the teachings presented in the above description and the accompanying drawings. It is understood, therefore, that the invention is not limited to the specific embodiments disclosed, and that modifications and other embodiments are intended to fall within the scope of the appended claims. Furthermore, while the above description and the accompanying drawings illustrate exemplary embodiments in the context of specific exemplary combinations of elements and / or functions, it should be recognized that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, combinations of elements and / or functions different from those expressly described above are also intended to be described in part of the appended claims. Therefore, the description is intended to encompass all such alternatives, modifications, and variations that fall within the scope of the appended claims. Certain terms are used herein, but not for limitation, but merely in a general and descriptive sense.

[0084] It should be understood that the above description is merely illustrative. Various alternatives and modifications may be devised by those skilled in the art. For example, the features listed in various dependent claims may be combined with one another in any suitable combination. Furthermore, features from the different embodiments described above may be selectively combined to form new embodiments. Accordingly, the description is intended to encompass all such alternatives, modifications, and variations that fall within the scope of the attached claims.

[0085] References to "computer," "processor," etc., should be understood to include not only computers with various architectures such as single / multi-processor architectures and sequential (von Neumann) / parallel architectures, but also dedicated circuits such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), signal processing devices, and other processing circuit sections. References to computer programs, instructions, code, etc., should be understood to include programmable content for hardware devices, such as instructions for processors, or software or firmware for programmable processors, such as configuration settings for fixed-function devices, gate arrays, or programmable logic devices, and similar devices.

[0086] The term "non-transitory," as used herein, refers to limitations on the medium itself (i.e., tangible, non-signal) in contrast to limitations on data storage persistence (e.g., RAM vs. ROM).

[0087] As used in this application, the term “circuitry” may refer to one, more, or all of the following: (a) Hardware-only circuit implementation configurations (such as simply the analog and / or digital circuit sections), and (b) combinations of hardware circuits and software, for example (if applicable): (i) A combination of analog hardware circuitry and / or digital hardware circuitry having software / firmware, (ii) A hardware processor with software (including a digital signal processor), software, and any part of memory) that works together to perform various functions in a device such as a mobile phone or server, (iii) Hardware circuits and / or processors, such as a microprocessor or a part of a microprocessor, that require software (e.g., firmware) for operation, however, the software may be absent when it is not required for operation.

[0088] This provision of "circuit section" applies to all use of the term in this application, including any claims. As a further example, as used in this application, the term "circuit section" also covers simply hardware circuits or processors (or more processors) or parts of hardware circuits or processors, as well as the execution of their (or their) accompanying software and / or firmware. The term "circuit section" also covers, for example, baseband integrated circuits or processor integrated circuits for mobile devices or similar integrated circuits in servers, cellular network devices, or other computing or network devices, where applicable to a particular claim element.

[0089] It should be understood that the above description is merely illustrative. Various alternatives and modifications may be devised by those skilled in the art. For example, the features cited in various dependent claims may be combined with one another in any suitable combination. Furthermore, features from the different embodiments described above may be selectively combined to form new embodiments. Accordingly, the description is intended to encompass all such alternatives, modifications, and variations that fall within the scope of the attached claims.

Claims

1. It is a device, At least one processor, The device comprises at least one non-temporary memory for storing instructions, and when an instruction is executed by the at least one processor, the device: Receiving external information, Receiving external direction control information, Selecting external direction correction, In accordance with the selected external direction correction, the rendering direction of the spatially rendered direction is corrected, and Render spatial audio according to the modified rendering direction. A device that enables the execution of an action.

2. The apparatus according to claim 1, wherein the external direction control information includes interactive interaction signals for enabling, disabling, or freezing the application of the external direction information.

3. The apparatus according to claim 1, further configured to receive head tracking information, wherein the spatial audio is further rendered based on the head tracking information.

4. The apparatus according to claim 1, wherein the external direction control information includes a head direction flag for freezing the head direction in the last rendered spatial audio direction.

5. The apparatus according to claim 1 or 4, wherein the external direction control information includes a head direction flag for enabling or disabling head tracking.

6. The apparatus according to claim 4, wherein when the head direction flag is set to disable the head tracking, the head tracking is disabled and the audio is rendered using the disabled head tracking and the corresponding forward direction default.

7. The apparatus according to claim 4, wherein when the head direction flag is set to enable head tracking, head tracking is enabled and audio is rendered based on the head direction.

8. moreover, Reference data is received to define the scene orientation. The apparatus according to claim 1, to which the aforementioned scene direction data can be applied.

9. Furthermore, the apparatus according to any one of claims 1 to 8, which can combine the external direction data and head tracking data.

10. Furthermore, the apparatus according to any one of claims 1 to 9, which can receive an external direction activation time to indicate that the external direction information is applied to the current frame or a future frame.

11. Furthermore, the apparatus according to any one of claims 1 to 10, wherein it is made to receive combined information that affects the external direction information, and the combined information includes two or more combinations of scene direction information provided by a transmitting user device (UE), device direction data provided by the transmitting UE, or local scene direction information provided by a receiving UE.

12. Receiving external information, Receiving external direction control information, Choosing external direction correction, The rendering direction of the spatially rendered direction is modified according to the selected external direction modification, Rendering spatial audio according to the aforementioned modified rendering direction and Methods that include...

13. The method according to claim 12, wherein the external direction control information includes interactive interaction signals to enable, disable, or freeze the application of the external direction information.

14. The method according to claim 12, further comprising receiving head tracking information, wherein the spatial audio is further rendered based on the head tracking information.

15. The method according to claim 12, wherein the external direction control information includes a head direction flag for freezing the head direction in the last rendered spatial audio direction.

16. The method according to claim 12 or 15, wherein the external direction control information includes a head direction flag for enabling or disabling head tracking.

17. The method according to claim 16, wherein when the head direction flag is set to disable the head tracking, the head tracking is disabled and the audio is rendered using the disabled head tracking and the default or corresponding forward direction.

18. The method according to claim 16, wherein when the head direction flag is set to enable head tracking, head tracking is enabled and audio is rendered based on head direction.

19. Receiving reference data to define the scene orientation, Applying the aforementioned scene orientation data The method according to claim 12, including the method described in claim 12.

20. The method according to any one of claims 12 to 19, further comprising combining the external direction data with head tracking data.

21. The method according to any one of claims 12 to 20, further comprising receiving an external direction activation time to indicate that the external direction information is applied to the current frame or to a future frame.

22. The method according to any one of claims 12 to 21, further comprising receiving combined information that affects the external directional information, wherein the combined information includes two or more combinations of scene directional information provided by a transmitting user equipment (UE), device directional data provided by the transmitting UE, or local scene directional information provided by a receiving UE.

23. Means for receiving external information, Means for receiving external direction control information, Means for selecting external direction correction, Means for modifying the rendering direction of the spatially rendered direction according to the selected external direction modification, Means for rendering spatial audio according to the modified rendering direction, A device including a device.

24. The apparatus according to claim 23, further comprising means for carrying out one or more methods according to claims 12 to 22.

Images