Rescaling audio sources based on movement in an extended reality system

By using dynamic rescaling technology, extended reality systems solve the problem of user movement and environmental changes during sound field reproduction, enhancing the creativity and immersion of audio elements and adapting to changes in different playback positions.

CN123077128APending Publication Date: 2026-11-10QUALCOMM INC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202580027442.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2025-04-17
Filing Date
2025-04-18
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing extended reality systems cannot effectively handle the user's dynamic movement and environmental changes during sound field reproduction, resulting in reduced immersion or limited ways of perceiving audio elements.

Method used

Extended reality systems use dynamic rescaling technology to adjust the position and rendering of audio elements in real time based on the user's movement and scene changes in the virtual or physical environment, in order to adapt to changes in different playback positions.

Benefits of technology

It enhances the creative reproduction and immersion of the sound field, better adapts to a wide range of virtual environment changes, and improves the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123077128A_ABST
    Figure CN123077128A_ABST
Patent Text Reader

Abstract

Generally, techniques are described that enable a device to rescale an audio source in an extended reality system. The device can include a memory configured to store an audio element that is resident within a virtual environment that represents a source location. The device also includes processing circuitry communicatively coupled to the memory and configured to determine that a listener has done either or both of: moved between virtual rooms within the virtual environment; and moved between physical rooms within a playback location. The processing circuitry is also configured to, in response to determining that the listener has done either or both of: moved between virtual rooms within the virtual environment; and moved between physical rooms within the playback location, rescale the audio element to obtain a rescaled audio element, and reproduce the rescaled audio element to obtain an output audio signal.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application claims priority to U.S. Patent Application No. 19 / 182,324, filed April 17, 2025; U.S. Provisional Application No. 63 / 636,611, filed April 19, 2024; and U.S. Provisional Application No. 63 / 636,621, filed April 19, 2024, the entire contents of each of which are incorporated herein by reference. U.S. Patent Application No. 19 / 182,324, filed April 17, 2025, claims the benefit of U.S. Provisional Application Nos. 63 / 636,611 and 63 / 636,621, filed April 19, 2024. Technical Field

[0002] This disclosure relates to the processing of audio data. Background Technology

[0003] Computer-mediated reality systems are being developed to allow computing devices to enhance, add to, remove from, reduce, or generally modify the existing reality experienced by a user. Computer-mediated reality systems (which may also be referred to as “extended reality systems” or “XR systems”) can include, for example, virtual reality (VR) systems, augmented reality (AR) systems, and mixed reality (MR) systems. The perceptual success of computer-mediated reality systems is often related to their ability to provide realistic, immersive experiences in both visual and audio aspects, where the visual and audio experiences are aligned in a way that the user expects.

[0004] Although the human visual system is more sensitive than the human auditory system (e.g., in perceiving and locating various objects within a scene), ensuring a full auditory experience is becoming an increasingly important factor in ensuring a realistic and immersive experience, especially as visual experiences improve to allow for better localization of visual objects that enable users to better identify the source of audio content. Summary of the Invention

[0005] This disclosure relates in its entirety to techniques for scaling audio sources in extended reality systems. Various aspects of the technique enable extended reality systems to scale source locations to fit playback locations, rather than requiring users to operate the extended reality system only in locations that correspond one-to-one with the source locations where the captured extended reality scene is situated and / or to which the generated extended reality scene is targeted. Therefore, if the source locations include microphones spaced 10 meters (10M), the extended reality system can scale the 10M interval resolution to fit the scaling of the playback location using a scaling factor determined based on the source dimension defining the size of the source locations and the playback dimension defining the size of the playback locations. An audio renderer can be initialized to statically perform this rescaling to fit playback locations of different sizes, which improves the reproduction of the sound field.

[0006] However, while initializing the audio renderer in this manner improves sound field reproduction by taking into account the initial differences between the source and playback locations, various updates during sound field reproduction, such as listener positioning (whether in a virtual environment and / or in a physical location), changes in the virtual scene, and changes in the playback location environment (e.g., opening a door, drawing curtains, etc. at the playback location), can affect the automatic rescaling of audio element 15 in a way that reduces immersion or creativity. In other words, statically initializing the audio renderer to account for differences between the source and playback locations may not account for dynamic changes occurring during sound field reproduction, which could reduce the immersive effect of automatic rescaling operations (or otherwise limit how audio rescaling is perceived).

[0007] To address these changes, extended reality systems can dynamically perform automatic rescaling during sound field reproduction. As an example, an extended reality system can determine that a listener has moved between two or more audio elements, which may reside in different or overlapping areas within a virtual environment representing the source location. In response to determining that the listener has moved between two or more audio elements, the extended reality system can dynamically rescale (e.g., during sound field reproduction) two or more audio elements based on the corresponding areas to obtain at least one rescaled audio element. Rescaling between audio elements allows for more creative sound field reproduction while also adapting to a wide range of virtual environments, including those generated via computing devices, such as in computer game applications.

[0008] Furthermore, extended reality systems can dynamically rescale during sound field reproduction based on various listener movements (either in the virtual environment or the physical playback location), changes in the virtual scene, and changes in the playback location environment (e.g., opening / closing a door, drawing / drawing curtains, etc.). Instead of using static rendering initialized by the initial renderer, extended reality systems can alternatively initialize the audio renderer and then continuously adapt the automatic rescaling during sound field reproduction to potentially account for the various changes listed above. In this respect, extended reality systems can perform dynamic automatic rescaling, which better reproduces the content creator's intent, better considers listener preferences, and / or further promotes a more immersive experience.

[0009] In one example, these techniques relate to a device configured to scale audio between a source location and a playback location, the device comprising: a memory configured to store metadata specifying two or more audio elements, the metadata identifying a corresponding area in which each of the two or more audio elements resides within a virtual environment representing the source location; and processing circuitry communicatively coupled to the memory and configured to: determine that a listener has moved between the two or more audio elements; in response to determining that the listener has moved between the two or more audio elements and based on the corresponding area, rescale the two or more audio elements to obtain at least one rescaled audio element; and reproduce the at least one rescaled audio element to obtain an output audio signal.

[0010] In another example, these techniques relate to a method for scaling audio between a source location and a playback location, the method comprising: storing metadata specified for two or more audio elements by processing circuitry, the metadata identifying a corresponding area in which each of the two or more audio elements resides within a virtual environment representing the source location; determining, by the processing circuitry, that a listener has moved between the two or more audio elements; rescaling, by the processing circuitry, based on the corresponding area, the two or more audio elements in response to determining that the listener has moved between the two or more audio elements to obtain at least one rescaled audio element; and reproducing, by the processing circuitry, the at least one rescaled audio element to obtain an output audio signal.

[0011] In another example, these techniques relate to a non-transitory computer-readable storage medium storing instructions that, when executed, cause processing circuitry to: store metadata specifying two or more audio elements, the metadata identifying a corresponding area in a virtual environment representing the source location where each of the two or more audio elements resides; determine that a listener has moved between the two or more audio elements; in response to determining that the listener has moved between the two or more audio elements and based on the corresponding area, rescale the two or more audio elements to obtain at least one rescaled audio element; and reproduce the at least one rescaled audio element to obtain an output audio signal.

[0012] In another example, these techniques relate to a device configured to encode an audio bitstream, the device comprising: a memory configured to store two or more audio elements; and processing circuitry coupled to the memory and configured to: encode the two or more audio elements to obtain an audio bitstream; specify syntax elements in the audio bitstream indicating how to perform rescaling when a listener moves between the two or more audio elements; and output the audio bitstream.

[0013] In another example, these techniques relate to a method for encoding an audio bitstream, the method comprising: encoding two or more audio elements to obtain an audio bitstream; specifying syntax elements in the audio bitstream that indicate how to perform rescaling when a listener moves between the two or more audio elements; and outputting the audio bitstream.

[0014] In another example, these techniques relate to a non-transitory computer-readable storage medium storing instructions that, when executed, cause processing circuitry to: encode two or more audio elements to obtain an audio bitstream; specify syntax elements in the audio bitstream that indicate how to perform rescaling when a listener moves between the two or more audio elements; and output the audio bitstream.

[0015] In another example, these techniques relate to a device configured to scale audio between a source location and a playback location, the device comprising: a memory configured to store audio elements residing within a virtual environment representing the source location; and processing circuitry communicatively coupled to the memory and configured to: determine that a listener has moved between virtual rooms within the virtual environment; and between physical rooms within the playback location; in response to determining that a listener has moved between the virtual rooms within the virtual environment; and between the physical rooms within the playback location, rescale the audio elements to obtain rescaled audio elements; and reproduce the rescaled audio elements to obtain an output audio signal.

[0016] In another example, these techniques relate to a method for scaling audio between a source location and a playback location, the method comprising: storing an audio element residing within a virtual environment representing the source location; determining that a listener has moved between virtual rooms within the virtual environment and between physical rooms within the playback location; in response to determining that the listener has moved between the virtual rooms within the virtual environment and between the physical rooms within the playback location, rescaling the audio element to obtain a rescaled audio element; and reproducing the rescaled audio element to obtain an output audio signal.

[0017] In another example, these techniques relate to a non-transitory computer-readable storage medium storing instructions that, when executed, cause processing circuitry to: store an audio element residing within a virtual environment representing the source location; determine that a listener has moved between virtual rooms within the virtual environment and between physical rooms within the playback location; rescale the audio element to obtain a rescaled audio element in response to determining that the listener has moved between the virtual rooms within the virtual environment and between the physical rooms within the playback location; and reproduce the rescaled audio element to obtain an output audio signal.

[0018] Details of one or more examples of this disclosure are set forth in the accompanying drawings and the following description. Further features, objects, and advantages of various aspects of the technology will be apparent from the description, the drawings, and the claims. Attached Figure Description

[0019] Figure 1A and Figure 1B This is a diagram illustrating various aspects of the techniques described in this disclosure.

[0020] Figure 2 This is a block diagram illustrating various aspects of implementing rescaling techniques to promote increased immersion in an example physical space while consuming expanded real-world scenes.

[0021] Figure 3A and Figure 3B This is a block diagram illustrating various aspects of implementing rescaling techniques to promote increased immersion in physical spaces while expanding the real-world scene for consumers.

[0022] Figure 4A and Figure 4B This is a flowchart illustrating exemplary operation of the extended reality system shown in the example of Figure 1 when performing various aspects of the rescaling techniques described in this disclosure.

[0023] Figure 5A and Figure 5B This is an example illustration of an XR device.

[0024] Figure 6 Examples of wireless communication systems supporting audio streaming are illustrated according to various aspects of this disclosure.

[0025] Figures 7A to 7C This is an example Figure 1A and Figure 1B The example shown is a diagram illustrating various aspects of the extended reality system's operation when performing tolerance-modified rescaling techniques.

[0026] Figures 8A to 8C This is an example Figure 1A and Figure 1B Additional illustrations show sample operations of the extended reality system in performing various aspects of tolerance-modified rescaling techniques, as shown in the example.

[0027] Figure 9A and Figure 9B This is an example Figure 1A and Figure 1B The example shown is an additional illustration of the extended reality system's operation in various aspects of performing tolerance-modified rescaling techniques.

[0028] Figure 10 This is an example Figure 1A and Figure 1B The example shown is another illustration of how the extended reality system performs various aspects of the rescaling technique for tolerance modification.

[0029] Figures 11A to 11C This is a diagram illustrating the syntax table for various aspects of rescaling techniques used to implement tolerance modifications.

[0030] Figure 12 This is a block diagram illustrating various aspects of implementing offset-based rescaling techniques to promote increased immersion in an example physical space while consuming expanded reality scenes.

[0031] Figure 13 This is a block diagram illustrating an example audio scene in which offset-based rescaling can be performed according to various aspects of the technology.

[0032] Figure 14 This is a block diagram of the renderer initializer of Figure 1, which is configured to implement offset-based scaling techniques in more detail.

[0033] Figure 15 Examples of providing through Figures 1A to 14The example shown is a flowchart illustrating sample operations of the renderer initializer when performing various aspects of the offset-based scaling techniques described in this disclosure.

[0034] Figure 16 This is a block diagram illustrating some examples of components of a device capable of performing offset scaling for interval-based audio source group processing, according to some examples of this disclosure.

[0035] Figure 17 This is an example Figure 1A and Figure 1B A flowchart illustrating example operations of the scene manager when performing various aspects of the dynamic rescaling techniques described in this disclosure.

[0036] Figure 18A and Figure 18B This is a diagram illustrating examples of automatic rescaling in response to a listener's movement between audio elements, based on various aspects of dynamic rescaling technology.

[0037] Figure 19 This is an illustration of examples of automatic rescaling when a listener moves between physical rooms, illustrating various aspects of the dynamic rescaling techniques described in this disclosure.

[0038] Figure 20 This is an illustration of an example of automatic rescaling performed when a listener moves between virtual rooms, according to various aspects of the dynamic rescaling techniques described in this disclosure.

[0039] Figure 21 This is a flowchart illustrating example operations of the playback system shown in Figure 1 when performing various aspects of audio source rescaling techniques.

[0040] Figure 22 This is a flowchart illustrating example operations of the sound field representation generator shown in Figure 1 when performing various aspects of audio source rescaling techniques.

[0041] Figure 23 This is a flowchart illustrating another example operation of the playback system shown in Figure 1 when performing various aspects of audio source rescaling techniques. Detailed Implementation

[0042] There are several different ways to represent a sound field. Example formats include channel-based audio formats, object-based audio formats, and scene-based audio formats. Channel-based audio formats refer to 5.1 surround sound, 7.1 surround sound, 22.2 surround sound, or any other channel-based format that positions audio channels to specific locations around the listener in order to reproduce the sound field.

[0043] Object-based audio formats can refer to a format in which audio objects (typically encoded using Pulse Code Modulation (PCM) and referred to as PCM audio objects) are specified to represent a sound field. Such audio objects may include metadata identifying the position of the audio object relative to a listener or other reference points in the sound field, allowing the audio object to be rendered to one or more speaker channels for playback in an attempt to reproduce the sound field. The techniques described in this disclosure are applicable to any of the formats described above, including scene-based audio formats, channel-based audio formats, object-based audio formats, or any combination thereof.

[0044] Scene-based audio formats can include a set of hierarchical elements that define the sound field in three dimensions. An example of such a set is the set of spherical harmonic coefficients (SHCs). The following expression illustrates a description or representation of the sound field using SHCs:

[0045]

[0046] This expression indicates that, in time t At any point in the sound field Pressure at the location Available by SHC Uniquely indicated. Here, , c It is the speed of sound (approximately 343 m / s). It is a reference point (or observation point). Is the order of n The spherical Bessel function, and Is the order of n The order of the sub-sub is m The spherical harmonic basis functions (which can also be called spherical basis functions). It can be recognized that the terms in square brackets are the frequency domain representation of the signal (i.e., It can be approximated by various time-frequency transforms, such as the Discrete Fourier Transform (DFT), Discrete Cosine Transform (DCT), or wavelet transform. Other examples of hierarchical sets include wavelet transform coefficient sets and other sets of coefficients for multi-resolution basis functions.

[0047] SHC These can be physically acquired (e.g., recorded) through various microphone array configurations, or alternatively, they can be derived from channel-based or object-based descriptions of the sound field. SHC (which can also be referred to as surround sound coefficients) represents scene-based audio, where SHC can be input into an audio encoder to obtain encoded SHC that can facilitate more efficient transmission or storage. For example, a (1+4) approach can be used. 2 The fourth-order representation of (25, and therefore fourth-order) coefficients.

[0048] As noted above, SHC can be derived from microphone recordings using a microphone array. Various examples of how SHC can be physically obtained from a microphone array are described in the following document: Poletti, M., “Three-Dimensional Surround Sound Systems Based on Spherical Harmonics,” Journal of the Audio Engineering Society, Vol. 53, No. 11, November 2005, pp. 1004-1025.

[0049] The following equation illustrates how SHC can be derived from an object-based description. Coefficients used for the sound field corresponding to a single audio object. It can be expressed as:

[0050]

[0051] Where i is , It is a spherical Hankel function of order n (of the second kind), and It refers to the object's location. Knowing the object's source energy as a function of frequency. (For example, using time-frequency analysis techniques, such as performing a Fast Fourier Transform on a Pulse Code Modulated (PCM) stream, it is possible to convert each PCM object and its corresponding location to an SHC.) Furthermore, it can be shown (since the above is a linear and orthogonal decomposition) for each object The coefficients are added together. In this way, multiple PCM objects can be generated from... The coefficients are represented (e.g., as the sum of coefficient vectors for a single object). These coefficients can contain information about the sound field (pressure as a function of 3D coordinates), and the above indicates the position at the viewpoint. The transformation of the representation from a single object to the entire sound field.

[0052] Computer-mediated reality systems (also known as “extended reality systems” or “XR systems”) are being developed to take advantage of the many potential benefits offered by atmos coefficients. For example, atmos coefficients can represent a sound field in three dimensions, potentially enabling accurate three-dimensional (3D) localization of sound sources within the sound field. Therefore, XR devices can render atmos coefficients as speaker feeds, which accurately reproduce the sound field when played back through one or more speakers.

[0053] Using Atmos coefficients in XR enables the development of multiple use cases that rely on the more immersive sound fields provided by Atmos coefficients, particularly for computer gaming applications and real-time visual streaming applications. In these highly dynamic use cases that depend on low-latency reproduction of the sound field, XR devices may prefer Atmos coefficients (compared to other representations that are more difficult to manipulate or involve complex rendering). The following section discusses... Figure 1A and Figure 1B More information is provided for these use cases.

[0054] Although VR devices are described in this disclosure, various aspects of the technology can be implemented in the context of other devices such as mobile devices. In this case, a mobile device (such as a so-called smartphone) can present the displayed world via a screen that can be mounted on the user's head or viewed as is typically done when using a mobile device. Therefore, any information on the screen can be part of the mobile device. The mobile device is capable of providing tracking information, thereby allowing both a VR experience (when mounted on the head) and a normal experience of viewing the displayed world, where the normal experience can still allow the user to view the displayed world as a simplified version of the VR experience (e.g., raising the device and rotating or panning it to view different parts of the displayed world).

[0055] This disclosure provides for scaling audio sources in extended reality systems. Various aspects of the technology enable extended reality systems to scale source locations to fit playback locations, rather than requiring users to operate the extended reality system only in locations that correspond one-to-one with the source locations where the captured extended reality scene is located and / or where the generated extended reality scene is targeted (or in other words, in space). Therefore, if the source locations include microphones spaced 10 meters (10M), the extended reality system can scale the 10M interval resolution to fit the scaling of the playback location using a scaling factor determined based on the source dimension defining the size of the source locations and the playback dimension defining the size of the playback locations. Using the scaling provided by the various aspects of the technology described in this disclosure, the extended reality system can improve the reproduction of the sound field to modify the location of the audio source to fit the size of the playback space.

[0056] Figure 1A and Figure 1B These are diagrams illustrating systems capable of performing various aspects of the techniques described in this disclosure. For example... Figure 1AAs illustrated in the example, system 10 includes a source device 12 and a content consumer device 14. While described in the context of source device 12 and content consumer device 14, these techniques can be implemented in any context in which any hierarchical representation of a sound field is encoded to form a bitstream representing audio data. Furthermore, source device 12 can represent any form of computing device capable of generating hierarchical representations of sound fields, and is generally described herein in the context of a VR content creator device. Similarly, content consumer device 14 can represent any form of computing device capable of implementing the audio stream interpolation techniques and audio playback described in this disclosure, and is generally described herein in the context of a VR client device.

[0057] Source device 12 can be operated by an entertainment company or other entity capable of generating multi-channel audio content for consumption by operators of content consumer devices (such as content consumer device 14). In many VR scenarios, source device 12 combines visual content to generate audio content. Source device 12 includes content capture device 300 and content sound field representation generator 302.

[0058] The content capture device 300 can be configured to interface with or otherwise communicate with one or more microphones 5A-5N (“microphone 5”). Microphone 5 may represent Eigenmike. ® Or other types of 3D audio microphones, capable of capturing a sound field and representing it as corresponding scene-based audio data 11A-11N (which may also be referred to as immersive sound coefficients 11A-11N or "immersive sound coefficients 11"). In the context of scene-based audio data 11 (which is another way of referring to "immersive sound coefficients 11"), each microphone in microphone 5 can represent a cluster of microphones arranged within a single housing according to a set geometry that facilitates the generation of immersive sound coefficients 11. Therefore, the term microphone can refer to a cluster of microphones (which is actually a geometrically arranged transducer) or a single microphone (which may be referred to as a point microphone or point transducer).

[0059] The Atmos coefficient 11 can represent an example of an audio stream. Therefore, the Atmos coefficient 11 can also be referred to as audio stream 11. Although the Atmos coefficient 11 has been described primarily, these techniques can be performed on other types of audio streams, including Pulse Code Modulation (PCM) audio streams, channel-based audio streams, object-based audio streams, and so on.

[0060] In some examples, the content capture device 300 may include an integrated microphone integrated into the housing of the content capture device 300. The content capture device 300 may dock with the microphone 5 wirelessly or via a wired connection. The content capture device 300 may process the immersive sound coefficient 11 after it has been input wirelessly and / or via a wired input process via some type of removable storage device, or as an alternative to or in combination with the above-described manner, after generating or otherwise creating the immersive sound coefficient 11 (from stored sound samples, such as those common in gaming applications), rather than capturing audio data via or combining it with audio data captured via the microphone 5. Therefore, various combinations of the content capture device 300 and the microphone 5 are possible.

[0061] The content capture device 300 can also be configured to interface with or otherwise communicate with the sound field representation generator 302. The sound field representation generator 302 can include any type of hardware device capable of interfaced with the content capture device 300. The sound field representation generator 302 can use the immersive sound coefficients 11 provided by the content capture device 300 to generate various representations of the same sound field represented by the immersive sound coefficients 11.

[0062] For example, in order to generate different representations of the sound field using atmos coefficients (which are again an example of an audio stream), the sound field representation generator 302 can use a decoding scheme for the atmos representation of the sound field, which is called Mixed-Order Atmos (MOA), as discussed in more detail in U.S. Application Serial No. 15 / 672,058 entitled “MIXED-ORDER AMBISONICS (MOA) AUDIO DATA FO COMPUTER-MEDIATEDREALITY SYSTEMS”, filed August 8, 2017 and published January 3, 2019 as U.S. Patent No. 20190007781.

[0063] To generate a specific MOA representation of a sound field, the sound field representation generator 302 can generate a partial subset of the complete set of atmos coefficients (wherein the term "subset" is not used in a strictly mathematical sense to include zero or more (if not all) of the complete set, but may refer to one or more, but not all, of the complete set). For example, each MOA representation generated by the sound field representation generator 302 may provide accuracy for some regions of the sound field, but lower accuracy for others. In one example, the MOA representation of the sound field may include eight (8) uncompressed atmos coefficients, while a third-order atmos representation of the same sound field may include sixteen (16) uncompressed atmos coefficients. Thus, each MOA representation of the sound field generated as a partial subset of the atmos coefficients may be less storage-intensive and less bandwidth-intensive (if and when transmitted as part of bitstream 27 via the illustrated transmission channel) (compared to the corresponding third-order atmos representation of the same sound field generated from the atmos coefficients).

[0064] Although the MOA representation has been described, the techniques disclosed herein can also be performed for a first-order Atmos (FOA) representation, wherein all Atmos coefficients associated with the first-order and zero-order spherical basis functions are used to represent the sound field. In other words, the sound field representation generator 302 can represent the sound field using all Atmos coefficients of a given order N, resulting in a total of (N+1) Atmos coefficients. 2 Instead of using a subset of non-zero immersive coefficients to represent the sound field.

[0065] In this respect, atmos audio data (which refers to atmos coefficients in MOA representation or full-order representation (such as the first-order representation mentioned above) can include atmos coefficients associated with spherical basis functions of order one or less (which may be referred to as "first-order atmos audio data"), atmos coefficients associated with spherical basis functions of mixed order and sub-order (which may be referred to as "MOA representation" discussed above), or atmos coefficients associated with spherical basis functions of order greater than one (which are referred to as "full-order representation" above).

[0066] In some examples, the content capture device 300 may be configured to communicate wirelessly with the sound field representation generator 302. In some examples, the content capture device 300 may communicate with the sound field representation generator 302 via one or both of a wireless or wired connection. Through the connection between the content capture device 300 and the sound field representation generator 302, the content capture device 300 can provide content in various formats, which, for the purposes of discussion, are described herein as part of the immersive sound coefficient 11.

[0067] In some examples, the content capture device 300 may utilize various aspects of the sound field representation generator 302 (in terms of the hardware or software capabilities of the sound field representation generator 302). For example, the sound field representation generator 302 may include dedicated hardware configured to perform psychoacoustic audio coding (or dedicated software that, when executed, causes one or more processors to perform psychoacoustic audio coding) (such as the Unified Speech and Audio Decoder denoted as "USAC" by the Moving Picture Experts Group (MPEG), the MPEG-H 3D Audio Decoding Standard, the MPEG-I Immersive Audio Standard, or proprietary standards such as AptX). ™ This includes various versions of AptX, such as Enhanced AptX (E-AptX), AptX Live, AptX Stereo, and AptX Highdefinition (AptX-HD), Advanced Audio Codec (AAC), Audio Codec 3 (AC-3), Apple Lossless Audio Codec (ALAC), MPEG-4 Audio Lossless Streaming (ALS), Enhanced AC-3, Free Lossless Audio Codec (FLAC), Monkey Audio, MPEG-1 Audio Layer II (MP2), MPEG-1 Audio Layer III (MP3), Opus, and Windows Media Audio (WMA).

[0068] The content capture device 300 may not include dedicated hardware or software for a psychoacoustic audio encoder, but instead provide the audio aspects of the content 301 in a non-psychoacoustic audio decoding form. The sound field representation generator 302 can assist in capturing the content 301 by performing psychoacoustic audio encoding on at least part of the audio aspects of the content 301.

[0069] The sound field representation generator 302 can also assist in content capture and transmission by generating one or more bitstreams 21, at least in part, based on audio content generated from the atlas coefficients 11 (e.g., MOA representation, third-order atlas representation, and / or first-order atlas representation). The bitstreams 21 can represent compressed versions of the atlas coefficients 11 (and / or a subset thereof of the MOA representation used to form the sound field) and any other different types of content 301 (such as compressed versions of spherical visual data, image data, or text data).

[0070] The sound field representation generator 302 can generate a bitstream 21 for transmission, for example, across a transmission channel (which may be a wired or wireless channel), a data storage device, etc. Bitstream 21 may represent an encoded version of the Atmos coefficients 11 (and / or a subset thereof of the MOA representation used to form the sound field), and may include a main bitstream and another side bitstream, which may be referred to as side channel information. In some cases, the bitstream 21 representing a compressed version of the Atmos coefficients 11 may conform to a bitstream generated according to the MPEG-H 3D audio decoding standard and / or the MPEG-I standard for "decoded representation of immersive media".

[0071] Content consumer device 14 can be operated by an individual and can represent a VR client device. Although described as a VR client device, content consumer device 14 can represent other types of devices, such as augmented reality (AR) client devices, mixed reality (MR) client devices (or other types of head-mounted displays or extended reality (XR) devices), standard computers, head-mounted devices, headsets, or any other device capable of tracking the head movements and / or general translational movements of the individual operating content consumer device 14. Figure 1A As shown in the example, the content consumer device 14 includes an audio playback system 16A, which can refer to any form of audio playback system capable of rendering atmos coefficients (whether in the form of first-order, second-order, and / or third-order atmos representations and / or MOA representations) for playback as multi-channel audio content.

[0072] Content consumer device 14 can retrieve bit stream 21 directly from source device 12. In some examples, content consumer device 14 can interface with a network (including a fifth-generation (5G) cellular network) to retrieve bit stream 21 or otherwise cause source device 12 to send bit stream 21 to content consumer device 14.

[0073] Despite Figure 1A The source device 12 is shown sending the bitstream 21 directly to the content consumer device 14, but it may also output the bitstream 21 to an intermediate device located between the source device 12 and the content consumer device 14. The intermediate device may store the bitstream 21 for later delivery to the content consumer device 14, which may request it. The intermediate device may include a file server, web server, desktop computer, laptop computer, tablet computer, mobile phone, smartphone, or any other device capable of storing the bitstream 21 for later retrieval by an audio decoder. The intermediate device may reside in a content delivery network capable of streaming the bitstream 21 (and possibly in conjunction with sending a corresponding visual data bitstream) to subscribers (such as the content consumer device 14) that request the bitstream 21.

[0074] Alternatively, source device 12 may store bitstream 21 to a storage medium, such as an optical disc, digital visual disc, high-definition visual disc, or other storage medium, most of which are computer-readable and therefore may be referred to as a computer-readable storage medium or a non-transitory computer-readable storage medium. In this context, a transmission channel may refer to the channel through which the content stored on the medium is transmitted (and may include retail stores and other store-based delivery mechanisms). In any case, the technology disclosed herein should therefore not be limited to this point. Figure 1A Examples.

[0075] As noted above, the content consumer device 14 includes an audio playback system 16. The audio playback system 16 can represent any system capable of playing back multi-channel audio data. The audio playback system 16A can include multiple different audio renderers 22. Each renderer 22 can provide different forms of audio rendering, wherein the different forms of rendering can include one or more of various methods of performing vector-based amplitude shifting (VBAP) and / or one or more of various methods of performing sound field synthesis. As used herein, “A and / or B” means “A or B” or both “A and B”.

[0076] The audio playback system 16A may also include an audio decoding device 24. The audio decoding device 24 may represent a device configured to decode the bitstream 21 to output reconstructed atmos coefficients 11A'-11N' (which can form a complete first-order, second-order, and / or third-order atmos representation or a subset thereof of the same sound field MOA representation, such as the dominant audio signal, ambient atmos coefficients, and vector-based signals described in the MPEG-H 3D audio decoding standard and / or the MPEG-I immersive audio standard).

[0077] Therefore, the immersive sound coefficients 11A'-11N' (“immersive sound coefficients 11”) can be similar to a complete set or a partial subset of the immersive sound coefficients 11, but can differ due to lossy operations (e.g., quantization) and / or transmission via a transmission channel. The audio playback system 16 can obtain immersive sound audio elements 15 from different streams of immersive sound coefficients 11' after decoding the bitstream 21 to obtain the immersive sound coefficients 11', and render the immersive sound audio elements 15 to the output speaker feed 25. The speaker feed 25 can drive one or more speakers (for illustrative purposes, in...). Figure 1A (Not shown in the example). The panoramic sound representation of the sound field can be normalized in a variety of ways, including N3D, SN3D, FuMa, N2D, or SN2D.

[0078] To select an appropriate renderer, or in some cases generate an appropriate renderer, the audio playback system 16A may obtain loudspeaker information 13 indicating the number of loudspeakers and / or the spatial geometry of the loudspeakers. In some cases, the audio playback system 16A may use a reference microphone to obtain the loudspeaker information 13 and output a signal to activate (or in other words, drive) the loudspeakers so as to dynamically determine the loudspeaker information 13 via the reference microphone. In other cases, or in conjunction with the dynamic determination of the loudspeaker information 13, the audio playback system 16A may prompt the user to dock with the audio playback system 16A and input the loudspeaker information 13.

[0079] The audio playback system 16A can select an audio renderer from the audio renderer 22 based on the loudspeaker information 13. In some cases, when none of the audio renderers in the audio renderer 22 falls within a threshold similarity metric (in terms of loudspeaker geometry) specified in the loudspeaker information 13, the audio playback system 16A can generate an audio renderer from the audio renderer 22 based on the loudspeaker information 13. In some cases, the audio playback system 16A can generate an audio renderer from the audio renderer 22 based on the loudspeaker information 13 without first attempting to select an existing audio renderer from the audio renderer 22.

[0080] When speaker feed 25 is output to headphones, audio playback system 16A can utilize one of the renderers in renderer 22, which uses a Head-Related Transfer Function (HRTF) or other functions capable of rendering to the left and right speaker feeds 25 to provide binaural rendering for headphone speaker playback. The term "speaker" or "transducer" can generally refer to any speaker, including loudspeakers, headphone speakers, etc. One or more speakers can then play back the rendered speaker feed 25.

[0081] Although described as rendering speaker feed 25 from Atmos audio element 15, the reference to rendering speaker feed 25 can refer to other types of rendering, such as rendering directly incorporated into decoding Atmos audio element 15 from bitstream 21. Examples of alternative rendering can be found in Appendix G of the MPEG-H 3D audio decoding standard, where rendering occurs during the formation of the dominant signal and background signal prior to the synthesis of the sound field. Therefore, the reference to rendering Atmos audio element 15 should be understood to refer to the rendering of either the actual Atmos audio element 15 or a decomposition or representation of Atmos audio element 15 (such as the aforementioned dominant audio signal, ambient Atmos coefficients, and / or vector-based signals (which may also be referred to as V-vectors)).

[0082] As described above, content consumer device 14 can refer to a VR device in which a human wearable display is mounted in front of the eyes of a user operating the VR device. Figure 5A and Figure 5B This is an illustration showing examples of VR devices 400A and 400B. Figure 5A In the example, VR device 400A is coupled to or otherwise includes headphones 404, which can reproduce the sound field represented by Atmos audio element 15 (another way of referring to Atmos coefficients) through playback of speaker feed 25. Speaker feed 25 can represent analog or digital signals that cause the diaphragm within the transducer of headphones 404 to vibrate at various frequencies. This process is commonly referred to as driving headphones 404.

[0083] Visual, audio, and other sensory data can play a significant role in VR experiences. To participate in a VR experience, user 402 can wear VR device 400A (which may also be referred to as VR headset 400A) or other wearable electronic devices. The VR client device (such as VR headset 400A) can track the head movements of user 402 and adapt the visual data displayed via VR device 400A to take head movements into account, thereby providing an immersive experience in which user 402 can experience the visual world displayed in visual data in three visual dimensions.

[0084] While VR (and other forms of AR and / or MR, which can often be referred to as computer-mediated reality devices) allows users 402 to visually reside in a virtual world, VR headsets 400A typically lack the ability to audibly place the user in the virtual world. In other words, a VR system (which may include a computer responsible for rendering visual and audio data) requires a different approach. Figure 5A (The computer is not shown in the example) and the VR headset 400A may not be able to support full 3D immersion in an audible manner.

[0085] Figure 5BThis is an illustration of an example of a wearable device 400B that can operate according to various aspects of the techniques described in this disclosure. In various examples, wearable device 400B can represent a VR headset (such as the VR headset 400A described above), an AR headset, a MR headset, or any other type of XR headset. Augmented Reality “AR” can refer to computer-rendered images or data overlaid on the real world in which the user actually resides. Mixed Reality “MR” can refer to computer-rendered images or data locked to a specific location in the real world, or can refer to a variant of VR in which a combination of computer-rendered 3D elements and filmed real elements is combined to create an immersive experience that simulates the user’s physical presence in the environment. Extended Reality “XR” can be a general term for VR, AR, and MR. More information on the terminology of XR can be found in Jason Peterson’s document entitled “Virtual Reality, Augmented Reality, and Mixed Reality Definitions” dated July 7, 2017.

[0086] Wearable device 400B can refer to other types of devices, such as watches (including so-called "smartwatches"), glasses (including so-called "smart glasses"), headphones (including so-called "wireless headphones" and "smart headphones"), smart clothing, smart jewelry, etc. Whether referring to VR devices, watches, glasses, and / or headphones, wearable device 400B can communicate with computing devices that support wearable device 400B via wired or wireless connections.

[0087] In some cases, the computing device supporting the wearable device 400B can be integrated within the wearable device 400B, and therefore, the wearable device 400B can be considered the same device as the computing device supporting the wearable device 400B. In other cases, the wearable device 400B can communicate with a separate computing device that can support the wearable device 400B. In this regard, the term "support" should not be understood as requiring a separate dedicated device, but rather as meaning that one or more processors configured to perform various aspects of the technologies described in this disclosure can be integrated within the wearable device 400B, or integrated within a computing device separate from the wearable device 400B.

[0088] For example, when wearable device 400B represents an example of VR device 400B, a separate dedicated computing device (such as a personal computer including one or more processors) can render audio and visual content, and wearable device 400B can determine translational head movements according to various aspects of the techniques described in this disclosure, wherein the dedicated computing device can render audio content (as a speaker feed) based on the translational head movements. As another example, when wearable device 400B represents smart glasses, wearable device 400B can include one or more processors that both determine translational head movements (by docking within one or more sensors of wearable device 400B) and render speaker feeds based on the determined translational head movements.

[0089] As shown in the figure, wearable device 400B includes one or more directional speakers and one or more tracking and / or recording cameras. Additionally, wearable device 400B includes one or more inertial, haptic, and / or health sensors, one or more eye-tracking cameras, one or more high-sensitivity audio microphones, and optical / projection hardware. The optical / projection hardware of wearable device 400B may include durable translucent display technology and hardware.

[0090] The wearable device 400B also includes connectivity hardware, which may represent one or more network interfaces supporting multi-mode connectivity (such as 4G communication, 5G communication, Bluetooth, etc.). The wearable device 400B also includes one or more ambient light sensors and a bone conduction transducer. In some cases, the wearable device 400B may also include one or more passive and / or active cameras with fisheye and / or telephoto lenses. Although in Figure 5B Not shown, but the wearable device 400B may also include one or more light-emitting diode (LED) lights. In some examples, the LED lights may be referred to as “super bright” LED lights. In some implementations, the wearable device 400B may also include one or more rear cameras. It will be understood that the wearable device 400B can exhibit a variety of different shape and form factors.

[0091] Furthermore, tracking and recording cameras, along with other sensors, can facilitate the determination of translation distance. Although in Figure 5B The example is not shown, but the wearable device 400B may include other types of sensors for detecting translational distance.

[0092] Although for specific examples of wearable devices (such as those mentioned above) Figure 5B The example discussed is the VR device 400B and in Figure 1A and Figure 1B Other devices illustrated in the examples have been described, but those skilled in the art will understand that, with Figures 1A to 5BThe related descriptions can be applied to other examples of wearable devices. For example, other wearable devices (such as smart glasses) may include sensors that obtain translational head movement. As another example, other wearable devices (such as smartwatches) may include sensors that obtain translational movement. Therefore, the techniques described in this disclosure should not be limited to a particular type of wearable device, but any wearable device can be configured to perform the techniques described in this disclosure.

[0093] exist Figure 1A In the example, source device 12 also includes camera 200. Camera 200 can be configured to capture visual data and provide the captured raw visual data to content capture device 300. Content capture device 300 can provide the visual data to another component of source device 12 for further processing into viewport-segmented portions.

[0094] Content consumer device 14 also includes wearable device 800. It will be understood that, in various implementations, wearable device 800 may be included within content consumer device 14 or externally coupled to that content consumer device. As stated above regarding... Figure 5A and Figure 5B The wearable device 800 under discussion includes display hardware and speaker hardware for outputting (e.g., associated with various viewports) visual data and for rendering audio data.

[0095] In any case, the audio aspects of VR have been categorized into three separate categories of immersion. The first category provides the lowest level of immersion and is known as three degrees of freedom (3DOF). 3DOF refers to audio rendering that takes into account head movement in three degrees of freedom (yaw, pitch, and roll), thus allowing the user to freely look around in any direction. However, 3DOF cannot account for translational head movements where the head is not centered on the optical and acoustic center of the sound field.

[0096] In addition to the limited spatial translation caused by head movement away from the optical and acoustic centers within the sound field, the second category (known as 3DOF plus (3DOF+)) offers three degrees of freedom (yaw, pitch, and roll). 3DOF+ can provide support for perceptual effects such as motion parallax, which can enhance immersion.

[0097] The third category (referred to as six degrees of freedom (6DOF)) renders audio data in a manner that considers three degrees of freedom of head movement (yaw, pitch, and roll), as well as the user's translation in space (x, y, and z translations). Spatial translation can be sensed by sensors tracking the user's position in the physical world or via an input controller.

[0098] 3DOF rendering is the current state of technology for audio in VR. As a result, VR is less immersive in terms of audio than in terms of visual immersion, which can potentially reduce the overall immersion experienced by the user and introduce positioning errors, such as when the auditory playback does not match or precisely relate to the visual scene.

[0099] While 3DOF rendering is the current state, more immersive audio rendering, such as 3DOF+ and 6DOF rendering, can lead to greater complexity in terms of processor cycles, memory consumption, and bandwidth. Furthermore, rendering for 6DOF may require additional granularity in terms of pose (which can refer to positioning and / or orientation), resulting in even greater complexity and also complicating the asynchronous capture of audio and visual data in certain XR scenarios.

[0100] For example, consider an XR scenario involving the capture of live audio data (e.g., XR meetings, visual meetings, visual chat, metaverse, XR games, live action events such as concerts, sporting events, discussions, conferences, etc.), where avatars (example visual objects) use microphones to speak to each other to capture live audio and convert such live audio into audio objects (which can also be called audio elements, since audio objects are not necessarily defined in an object format).

[0101] In some cases, XR scene capture can occur in large physical spaces, such as concert halls (e.g., for sports, concerts, discussions, etc.) or stadiums (e.g., for discussions) or conference halls, which can be larger than normal playback spaces, such as living rooms, entertainment rooms, TV rooms, or bedrooms. That is, the person experiencing the XR scene may not have a physical space at the same scale as the capture location.

[0102] In VR scenarios, users can teleport to different locations within the capture environment. The ability to overlay digital content onto real-world (or in other words, physical) space (e.g., in AR) coupled with the inability to teleport to different locations (which can refer to virtual locations, or in other words, locations within an XR scenario) can limit how the playback space can recreate an immersive experience for such XR scenarios. Specifically, the audio experience may be compromised due to the scaling difference between the content capture space and the playback space, potentially preventing the user from fully engaging with the XR scenario.

[0103] In some cases, the audio playback system 16 can scale the audio source in the extended reality system 10. Instead of requiring the user to operate the extended reality system 10 only in locations that correspond one-to-one with the source locations where the captured extended reality scene is located and / or where the generated extended reality scene is targeted, the extended reality system 10 can scale the 10M interval resolution to accommodate the scaling of the playback location, provided that the source locations include microphones spaced 10 meters (10M).

[0104] In operation, the sound field representation generator 302 can specify syntax elements in the audio bitstream 21 that indicate the rescaling factor used for audio elements. The rescaling factor indicates how the position of the audio element will be rescaled relative to other audio elements. The sound field representation generator 302 can also specify syntax elements that indicate that automatic rescaling will be performed for the duration during which the audio element is present for playback. The sound field representation generator 302 can then output the audio bitstream 21.

[0105] The audio playback system 16A can receive the audio bitstream 21 and extract syntax elements that indicate one or more source dimensions associated with the source space for the XR scene. In some cases, this syntax element can be specified in a side channel of metadata or in other extensions to the audio bitstream 21. In any case, the audio playback system 16A can parse the source dimensions from the audio bitstream 21 (e.g., as syntax elements) or otherwise obtain the source dimensions associated with the source space for the XR scene.

[0106] The audio playback system 16A can also obtain a playback dimension (one or more playback dimensions) associated with the physical space in which the playback of the audio bitstream 21 will occur. That is, the audio playback system 16A can interface with a user operating the audio playback system 16A to identify playback dimensions (e.g., through an interactive user interface in which the user moves around the room to identify playback dimensions) (and / or via a remote camera installed to capture playback of an XR scene, via a head-mounted camera for capturing playback / interaction with an XR scene, etc.).

[0107] The audio playback system 16A can then modify the position associated with the audio element based on the playback dimension and the source dimension to obtain the modified position of the audio element. At this point, the audio decoding device 24 can parse the audio element from the audio bitstream 21, which is represented in the example of Figure 1 as a panoramic audio element 15 (which may represent one or more audio elements). Although described relative to the panoramic audio element 15, various other audio data formats can be rescaled according to various aspects of the technology, including object-based formats, channel-based formats, etc.

[0108] In performing audio rescaling, the audio playback system 16A can determine whether the difference between the source dimension and the playback dimension exceeds a threshold difference (e.g., the difference between the source dimension and the playback dimension is greater than 1%, 5%, 10%, 20%, etc.). If the difference between the source dimension and the playback dimension exceeds different thresholds (e.g., defined as 1%, 5%, 10%, 20%, etc.), the audio playback system 16A can then determine a rescaling factor. The audio playback system 16A can determine the rescaling factor as a function of the playback dimension divided by the source dimension.

[0109] In the case of linear rescaling, the audio playback system 16A can determine the rescaling factor as the playback dimension divided by the source dimension. For example, assuming the source dimension is 20M and the playback dimension is 6M, the audio playback system 16A can calculate the rescaling factor as 6M / 20M, which equals 0.3 or 30% as the rescaling factor. The audio playback system 16A can apply the audio factor (e.g., 30%) when the scene manager 23 is invoked.

[0110] The audio playback system 16A can invoke the scene manager 23, passing a rescaling factor to it. The scene manager 23 can apply the rescaling factor while processing audio elements extracted from the audio bitstream 21. The scene manager 23 can modify the metadata defining the position of the audio element in the XR scene, rescaling the metadata to obtain the modified position of the audio object. The scene manager 23 can then pass the modified position to the audio renderer 22, which can render the audio element to the speaker feed 25 based on the modified position. The audio renderer 22 can then output the speaker feed 25 for playback by the audio playback system 16A (which may include one or more loudspeakers configured to reproduce the sound field based on the speaker feed 25).

[0111] The audio decoding device 24 can parse the aforementioned syntax element indicating a rescaling factor and automatic rescaling, the automatic rescaling indicating whether automatic rescaling should be performed. When the rescaling factor is specified as described above, the audio playback system 16A can directly obtain the rescaling factor from the audio bitstream 21 (meaning that, in this case, the rescaling factor is not calculated as a function of the playback dimension divided by the source dimension). The audio playback system 16A can then apply the rescaling factor as described above to obtain the location of the modified audio element.

[0112] When processing a syntax element that indicates automatic rescaling, audio playback system 16A may suppress rescaling of audio elements associated with the auto-rescaling syntax element that indicates automatic rescaling will not be performed. Otherwise, audio playback system 16A may use various aspects of the techniques described in this disclosure to process audio elements associated with an audio rescaling syntax element that indicates automatic rescaling will be performed for the purpose of performing rescaling. The auto-rescaling syntax element may configure audio playback system 16A to continuously apply a rescaling factor to the associated audio elements as long as the associated audio elements are present in the XR scene.

[0113] Using scaling (which refers to another way of rescaling) provided by various aspects of the techniques described in this disclosure, the audio playback system 16A (which refers to another way of XR playback system, playback system, etc.) can improve the reproduction of the sound field to modify the position of the audio source (which refers to another way of audio elements resolved from audio bitstream 21) to fit the size of the playback space. In implementing such scaling, the audio playback system 16A can improve the immersive experience for the user when consuming an XR scene, provided that the XR scene more closely matches the playback space. The user can then safely experience the entire XR scene within the limits of the allowed playback space. In this respect, these techniques can improve the operation of the audio playback system 16A itself.

[0114] However, even when zooming is used, there are cases where the playback position, or in other words, the real-world space is irregular (e.g., sloping walls, vaulted ceilings, domed ceilings, sloping ceilings, etc.) or the representation of the real-world space is incomplete (e.g., a scan or mapping of the real-world space contains elements that prevent a complete scan or mapping of the real-world space, such as furniture, lighting fixtures, etc.).

[0115] Therefore, the sound field representation generator 302 can specify another syntax element (e.g., a second syntax element) in the audio bitstream 21, which indicates a tolerance 50 for applying the rescaling factor. The content creator can define the tolerance 50, or an analysis of the source location (e.g., possibly using a trained artificial intelligence model, such as a neural network, statistical analysis, etc.) can produce the tolerance 50. The tolerance 50 can define the percentage of the source location that is retained outside the playback location (which may also be referred to as the "playback space" or "real-world space").

[0116] A tolerance of 50 can be defined as a percentage in three dimensions (e.g., width-x-axis, height-y-axis, and depth-z-axis). In other words, a tolerance of 50 can be defined as a width tolerance, a height tolerance, and a depth tolerance. Alternatively, a tolerance of 50 can be defined as the minimum and maximum values ​​(min / max) of each of the three dimensions (e.g., width-x-axis, height-y-axis, and depth-z-axis). That is, a tolerance of 50 can be defined as the maximum and minimum values ​​of each of the width tolerance, height tolerance, and depth tolerance. Using the minimum / maximum values, the sound field generator device 302 enables the definition of tolerances for irregular or incomplete representations of real-world space.

[0117] Audio decoding device 24 can obtain a syntax element indicating tolerance 50 and a rescaling factor. Tolerance 50 (which can be used to refer to one or more syntax elements representing the decoding of tolerance 50) can modify the application of the rescaling factor to the decoded immersive audio element 15 (or other forms of audio data). Furthermore, various aspects of the technology enable scene manager 23 to modify the scaling in three dimensions to accommodate irregular real-world spaces.

[0118] Scene manager 23 obtains a playback dimension associated with the physical space (or, in other words, real-world space, which may also be referred to as the playback location) in which playback of audio bitstream 21 will occur. As described in more detail in this disclosure, scene manager 23 also obtains a source dimension associated with a source space (which may also be referred to as the source location) used to extend the realistic scene. Scene manager 23 also obtains a tolerance 50 from audio decoding device 24, which in this example resolves the tolerance 50 from audio bitstream 21.

[0119] Scene Manager 23 can modify the position of an audio element based on the audio playback dimension, the source dimension, and a tolerance of 50 to obtain the modified position of the audio element (e.g., represented by Atmos audio element 15). Scene Manager 23 can first determine a rescaling factor based on the playback dimension and the source dimension. Scene Manager 23 can then adjust the Atmos audio element 15 based on the rescaling factor within the tolerance of 50. That is, as an example, Scene Manager 23 can modify the rescaling factor based on the tolerance of 50 to obtain the modified rescaling factor. Scene Manager 23 can then apply the modified rescaling factor to the position of the audio element (i.e., represented by Atmos audio element 15 in this example) to obtain the modified position of the audio element.

[0120] By incorporating a tolerance of 50 and scaling in three dimensions, various aspects of the technology enable the extended reality system 10 to provide a more immersive experience that can take into account the irregularities or incomplete representations of real-world space. In implementing such scaling, the extended reality system 10 can improve the immersive experience for users when consuming extended reality scenes, provided that the extended reality scene more closely matches the playback space. Users can then safely experience the entire extended reality scene within the boundaries of the permissible playback space. In this respect, these technologies can improve the operation of the extended reality system itself, such as regarding... Figures 7A to 11C As described.

[0121] As noted above, given that the source metadata (which, for the corresponding audio element, can be a way of generally referring to the Atmos audio element 15, channel-based audio data, and / or object-based audio data) can be defined at a non-ideal origin within the physical constraints of the playback location where the listener is located, the audio renderer 22 may not accurately reproduce the sound field. The content creator of audio element 15 (another way of referring to Atmos audio data 15) can specify an origin for reproduction in the virtual environment. This origin can represent the origin of the Atmos audio element 15 used to reproduce the virtual environment, which matches the source location (e.g., a concert venue, stadium, outdoor location, etc.).

[0122] The source location may have a dimension much larger (or alternatively, a much smaller) than the playback location identified by the audio playback system 16A. Maintaining the origin defined by the content creator may affect the rendering of the audio renderer 22 at the playback location, which may also be smaller (or potentially larger) than the source location. Various aspects of the technology enable the extended reality system to scale the source location to fit the playback location, rather than requiring the user (another way of referring to the listener) to operate the extended reality system only in locations that allow a one-to-one correspondence with the intervals of the source locations where the captured extended reality scene is located and / or to which the generated extended reality scene is targeted. Thus, if the source location includes microphones 5 spaced 10 meters (10M), the extended reality system can use a rescaling factor determined based on the source dimension defining the size of the source location and the playback dimension defining the size of the playback location to scale (which is effectively rescaling given that the scaling was initially defined by the content creator) the 10M interval resolution to fit the scaling of the playback location.

[0123] However, while rescaling can be performed, content creators can define the origin of the extended reality scene (via metadata associated with the immersive audio element 15 to be reproduced). The audio playback system 16A can use the origin to perform rescaling (which may also be referred to as "rescaling translation"), which could lead to incorrect audio reproduction given that the origin may not be correctly defined for real-world space (another way of referring to the playback location). The origin could be a global origin for both the source and playback locations, potentially leading to incorrect rescaling that fails to properly position the audio source within the playback location (when the playback location is scaled differently from the source location).

[0124] The audio playback system 16A can calculate, determine, or otherwise obtain offset 27, rather than relying solely on the global origin. This offset adjusts the origin to potentially rescale the extended real-world audio scene more accurately for the playback location. The audio playback system 16A can calculate offset 27 based on the audio element reference origin (another way of referring to the global origin), where offset 27 realigns the reference origin of the immersive audio element 15 with identified anchor points determined by the audio playback system 16A at the playback location, such as the listener's position obtained by the audio playback system 16A (which may also be referred to as listener positioning), and / or the virtual world origin obtained by the audio playback system 16A. The audio playback system 16A can utilize the origin to obtain the adjusted global origin of the immersive audio element 15 and perform rescaling relative to the adjusted global origin to potentially improve the reproduction of the audio element and create a more immersive user experience.

[0125] As noted above, the audio playback system 16A includes a scene manager 23 (which may be implemented or configured in the processing circuitry) configured to implement a renderer initializer 26 that provides additional metadata, including an offset 27 and a scaling factor. The renderer initializer 26 performs an audio renderer initialization phase that may include obtaining an offset 27 for the playback position based on a source origin (another way of referring to a "global origin" or "origin") defined by the content creator for reproduction in a virtual environment representing the source location. The renderer initializer 26 may output the offset 27 (possibly along with a rescaling factor) as additional metadata to the scene manager 23.

[0126] Additionally, renderer initializer 26 may receive from bitstream 21 syntax elements or other indications configuring how offset 27 will be obtained, where the indication may specify the expected origin from which the audio element will be positioned within the playback position when the audio element is rescaled. In some cases, the listener (or in other words, the user) and / or playback system 16A may override this expected origin. In some cases, bitstream 21 may not explicitly contain an indication specifying the expected origin, and a default expected origin may be selected.

[0127] Therefore, the sound field representation generator 302 can specify the expected origin in bitstream 21, where the expected origin is provided by the content creator (manually or possibly by analyzing the underlying sound field represented by audio stream 11). In some cases, the sound field representation generator 302 can implicitly specify the expected origin (e.g., the expected origin is a default setting and does not need to be explicitly indicated by a signal in bitstream 21, but can be implicitly specified). The expected origin can specify one of the following: anchor positioning within a virtual environment, listener positioning within a virtual environment, virtual origin of a virtual environment reproduced at a playback location, or playback origin for a playback location.

[0128] The renderer initializer 26 can determine the offset 27 in one or more different ways (possibly using a specified or default expected origin). For example, the renderer initializer 26 can obtain the anchor position within the virtual environment. That is, the renderer initializer 26 can determine the anchor position as a virtual object within the virtual environment (e.g., a virtual monitor, a virtual graphical user interface, a virtual "window," etc.). The renderer initializer 26 can then obtain the offset 27 relative to the playback position based on the source origin and the anchor position (in this case, an example of a reference origin).

[0129] As another example, renderer initializer 26 can be configured to obtain the playback origin relative to the playback position. Renderer initializer 26 can then obtain the offset 27 relative to the playback position based on the source origin and the playback origin. Relative to Figure 12The first three examples are illustrated in more detail below, and these examples will be discussed in more detail below.

[0130] However, while initializing the audio renderer 22 in this manner improves sound field reproduction by taking into account the initial differences between the source and playback locations, various updates during sound field reproduction, such as listener positioning (whether in a virtual environment and / or in a physical location), changes in the virtual scene, and changes in the playback location environment (e.g., opening a door, drawing curtains, etc. at the playback location), can affect the automatic rescaling of the immersive audio elements 15 in a way that reduces immersion or creativity. In other words, while the renderer initializer 26 can be configured or otherwise initialized to take into account the differences between the source and playback locations, these dynamic changes occurring during sound field reproduction may reduce the immersive effect of the automatic rescaling operation (or otherwise limit how audio rescaling is perceived).

[0131] According to various aspects of the technology described in this disclosure, scene manager 23 can dynamically perform automatic rescaling during sound field reproduction. As an example, scene manager 23 can determine that a listener has moved between two or more panoramic sound elements 15, which may reside in different or overlapping areas within a virtual environment representing the source location. In response to determining that a listener has moved between two or more panoramic sound elements 15 and based on the corresponding area, scene manager 23 can dynamically rescale (e.g., during sound field reproduction) two or more panoramic sound elements 15 to obtain at least one rescaled audio element. Rescaling between audio elements allows for more creative sound field reproduction while also adapting to a wide range of virtual environments (e.g., including virtual environments generated via computing devices, such as in the case of computer game applications).

[0132] In operation, the scene manager 23 can store metadata specified for audio elements 15 (this metadata can be parsed from bitstream 21 or otherwise specified by the listener, playback system 16A, scene manager 23, etc.). The metadata identifies corresponding regions, where each of the immersive audio elements 15 resides within a virtual environment representing the source location. Regions can be defined for one or more immersive audio elements 15 (meaning regions can be defined for a group of immersive audio elements 15, including multiple immersive audio elements 15 within a single group).

[0133] In some cases, the sound field representation generator 302 may specify a syntax element in the audio bitstream 21 that indicates how rescaling should be performed when a listener moves between two or more immersive audio elements 15. The syntax element may indicate how to perform rescaling relative to one of the two or more immersive audio elements 15 that the listener moves into. In some cases, the syntax element may also indicate how to perform rescaling relative to two or more audio elements that the listener moves out of. The listener can override the automatic rescaling specified by the syntax element via listener / user settings.

[0134] Scene Manager 23 can interface with tracking device 306, playback system 16A (e.g., including inputs to playback system 16A, such as a game controller), wearable device 800 (e.g., head-mounted display), etc., to determine if a listener (who may be wearing wearable device 800) has moved between two or more of the Atmos audio elements 15. That is, when two or more of the Atmos audio elements 15 are located in different areas within the virtual environment, Scene Manager 23 can dynamically rescale the Atmos audio elements 15 in different ways to potentially allow for a more creative reproduction of the sound field and / or increase audio immersion (especially associated with video immersion created by the reproduction of the virtual environment). When two or more of the Atmos audio elements 15 are in overlapping areas, Scene Manager 23 can dynamically rescale the Atmos audio elements 15 to the same extent (this can again be overridden via listener / user settings to allow the aforementioned different dynamic rescaling).

[0135] Scene manager 23 can dynamically rescale the immersive audio elements 15 to fit their positions and / or dimensions, as described throughout this disclosure. Scene manager 23 can also dynamically change or otherwise update the rescaling factor used when initializing audio renderer 22 to account for listener movement between two or more immersive audio elements 15. Audio renderer 22 can then reproduce at least one rescaled audio element to obtain output speaker feed 25. Output speaker feed 25 may also be referred to as megaphone feed 25, which is used to drive megaphones, headphone speakers, and / or any other transducers (where driving refers to activating a transducer to reproduce the sound field).

[0136] Additionally, the Scene Manager 23 can dynamically rescale during sound field reproduction based on various listener movements (either in the virtual environment or the physical playback location), changes in the virtual scene, changes in the playback location environment (e.g., opening / closing a door, drawing / drawing curtains, etc. at the playback location), etc. Instead of using static rendering initialized by the initial renderer, the Scene Manager 23 alternatively initializes the audio renderer 22 (via renderer initializer 26, as discussed in more detail throughout this disclosure) and can then continuously adapt to automatic rescaling during sound field reproduction to potentially account for the various changes listed above. In this respect, the Scene Manager 23 can perform dynamic automatic rescaling, which better reproduces the content creator's intent, better considers listener preferences, and / or further promotes a more immersive experience. The following is relative to... Figure 17 Describe an example procedure for performing this dynamic automatic rescaling.

[0137] Figure 17 This is an example Figure 1A and Figure 1B The flowchart illustrates example operations of the scene manager when performing various aspects of the dynamic rescaling technique described in this disclosure. Scene manager 23 may invoke renderer initializer 26 to perform scene initialization (1700), determine whether automatic rescaling is enabled (e.g., via syntax elements, via listener / user settings, etc.) (1702), and perform an automatic rescaling initialization process (1704). If automatic rescaling is not enabled, scene manager 23 may not initialize renderer 22 to perform automatic rescaling. Audio renderer 22 may then perform scene rendering (1706).

[0138] For example, scene manager 23 can parse bitstreams as follows:

[0139]

[0140] In this case, the various syntax elements referenced in the table above can be described as follows:

[0141] The isAudioSourceRescaleVariable flag indicates whether the value is provided via the renderer interface.

[0142] The `audioSourceRescaleVariableIndex` is an integer that is the index of the update value provided via the renderer interface.

[0143] newAudioSourceRescaleValue is the new rescaling factor value.

[0144] Scene manager 23 can then determine the dynamic conditions that changed during the reproduction of the sound field. In one example, scene manager 23 can determine whether the source localization has changed (e.g., due to initial rescaling) (1708), which allows for an effective determination of whether the rescaled audio element is smaller than the initial immersive audio element 15. When smaller than the initial immersive audio element 15, scene manager 23 can determine whether a rescaling Doppler setting has been enabled (e.g., true) (1710), which can be specified as a syntax element in bitstream 21, defined by listener / user settings, etc. Scene manager 23 can then, in response to determining that the rescaled audio element is smaller than the audio element (and possibly in the case where rescaling Doppler is enabled), reduce the speed associated with the audio element to reduce the Doppler effect (1712). In some cases, scene manager 23 can, in response to determining that the rescaled audio element is smaller than the audio element, increase the speed associated with the audio element to maintain the Doppler effect. In some examples, scene manager 23 may increase the speed associated with the audio element to maintain the Doppler effect in response to determining that the rescaled audio element is not smaller than the immersive audio element 15.

[0145] In other words, `audioSourceRescale` is a set of three floating-point values ​​{1 1 1}, meaning the audio element positioning will not change. These values ​​are multiplication factors, currently set once during encoding. During renderer 22 initialization, a tolerance value for scaling the multiplication values ​​is set once. In this case, scene manager 23 determines how to handle the Doppler effect. Scene manager 23 can perform the following operations:

[0146] Doppler can be disabled after rescaling;

[0147] If the scene is scaled down (made smaller), the Doppler effect of objects will be reduced. If the listener moves, the effect will be exaggerated, but this is only likely to be changed if the content creator wants a hyper-realistic feel;

[0148] Alternatively, as an example, Scene Manager 23 can determine how to preserve the original Doppler effect after rescaling by doing the following:

[0149] Regardless of whether the zoom level is resized, the listener's positioning and movement remain the same;

[0150] The default settings are likely designed to match what the user sees. When transmitting to the Doppler effect stage, is it unnecessary to undo and rescale the source localization?

[0151] Example 1: A car drives past the listener. In this case, after rescaling, the car's speed relative to the listener will be slower, thus weakening the Doppler effect compared to the original scene.

[0152] Example Scenario 2: A jet flies over the listener. In this case, after rescaling, the jet may move closer to the listener's head, thus enhancing the Doppler effect.

[0153] Example 3: The listener walks toward or away from the sound source. In this case, the listener is now moving faster than in the original scene (for a smaller, rescaled scene), so the Doppler is amplified and matches what the listener sees.

[0154] Additionally, Scene Manager 23 can determine whether the listener has moved between the Atmos audio elements 15 (1714). Scene Manager 23 can rescale based on a corresponding region defined for each Atmos audio element 15 in response to determining that the listener has moved between the Atmos audio elements 15. The corresponding region can be defined via metadata associated with the Atmos audio element 15, where the corresponding region indicates the position of each Atmos audio element 15 within a virtual environment representing the source location.

[0155] If scene manager 23 determines that the corresponding areas are the same (or in other words, different areas, e.g., non-overlapping) (1716), scene manager 23 may, in response to determining that the listener has moved between the immersive audio elements 15 and determining that the corresponding areas are different, rescale the immersive audio elements 15 to obtain at least one rescaled audio element. Scene manager 23 may perform rescaling relative to two or more immersive audio elements 15 in which the listener has moved or from which the listener has moved. In these cases, scene manager 23 may individually rescale two or more immersive audio elements 15 in different ways to obtain at least one rescaled audio element.

[0156] However, if the scene manager 23 determines that the corresponding areas are not different (e.g., overlapping), the scene manager 23 may, in response to determining that the listener has moved between two or more panoramic audio elements in the panoramic audio element 15 and determining that the corresponding areas overlap, rescale the panoramic audio element 15 to obtain the corresponding two or more scaled audio elements. The scene manager 23 may rescale two or more panoramic audio elements in the panoramic audio element 15 by the same amount. That is, the scene manager 23 may rescale two or more panoramic audio elements in the panoramic audio element 15 by the same amount and avoid rescaling two or more panoramic audio elements in the panoramic audio element 15 individually. At this point, the scene manager 23 may perform automatic rescaling relative to the panoramic audio element 15 in which the listener has moved or from which the listener has moved (1718).

[0157] Figure 18A and Figure 18B This is a diagram illustrating examples of automatic rescaling in response to a listener's movement between audio elements, based on various aspects of dynamic rescaling technology. Figure 18A In the example, audio element 1802A is associated with region 1800A, while audio element 1802B is associated with region 1800B. Scene manager 23 can rescale audio element 1802A relative to region 1800A to obtain rescaled audio element 1804A, or rescale an audio element relative to region 1800B to obtain rescaled audio element 1804B. Scene manager 23 can rescale audio element 1802B relative to region 1800A to obtain rescaled audio element 1806A, or rescale audio element 1802B relative to region 1800A to obtain rescaled audio element 1806B.

[0158] exist Figure 18B In the example, audio element 1802A overlaps with audio element 1802B, sharing the same area that spans regions 1800A and 1800B. Scene manager 23 can identify audio elements 1802A and 1802B that overlap in terms of region, and does not automatically rescale audio elements 1802A and 1802B separately (or, in other words, they share the same area, so there is no difference in handling rescaling between audio elements 1802A and 1802B).

[0159] Return to reference Figure 17Scene Manager 23 can determine whether the rescaling factor has changed during the reproduction of the sound field (1720). Scene Manager 23 can adapt renderer 22 to reinitialize renderer 22 to take the updated rescaling factor into account. Scene Manager 23 can also determine whether the listener has moved between virtual rooms (1722), whether the physical playback position has changed (1724), and / or whether the listener has moved between physical rooms (1726). Scene Manager 23 can then perform automatic rescaling based on the movement / change / dynamic rescaling factor (1728). Figure 19 This is an illustration of examples of automatic rescaling when a listener moves between physical rooms, illustrating various aspects of the dynamic rescaling techniques described in this disclosure.

[0160] exist Figure 19 In the example, the listener can move between physical locations 1900A (e.g., living room 1900A) and 1900B (e.g., side room 1900B), where different physical locations 1900A and 1900B can be associated with different audio elements 1902A (e.g., content A 1902A) and 1902B (e.g., content B 1902B). In this example, the listener starts from living room 1900A, and scene manager 23 configures rescaling for living room 1900A (which can also be referred to as living room area 1900A) during renderer initialization. The listener can move to side room 1900B, where scene manager 23 rescales with the dimensions of the new room (which is another way of referring to side room 1900B). In this case, the source is shifted. Furthermore, multiple regions can be defined in the AudioScene of the bitstream, or multiple LSDFs can be loaded (one LSDF for each of rooms 1900A and 1900B).

[0161] In some cases, Scene Manager 23 performs rescaling using the dimensions of the new room during the initialization of new content (e.g., Content B 1902B). In one case, the listener stops listening to Stage A when moving to Side Room 1900B to listen to Content B 1902B. In some examples, when the listener wants to continue listening to audio from the initial Content A 1902A, Scene Manager 23 may retain the rescaling as initially initialized, or rescale relative to Side Room 1900B (and thus obtain rescaled Content A (rescaled relative to the side room) 1904).

[0162] Figure 20 This is a diagram illustrating examples of automatic rescaling performed when a listener moves between virtual rooms, according to various aspects of the dynamic rescaling techniques described in this disclosure. Figure 20As shown in the example, listener 2000 moves from virtual room 2002A to virtual room 2002C via virtual room 2002B (e.g., corridor 2002B).

[0163] exist Figure 20 In the example, a listener might start in virtual room 2002A (e.g., office 2002A) and move via corridor 2002B to virtual room 2002C (e.g., lobby 2002C). The relatively small office 2002A might not require rescaling, but as the listener moves along corridor 2002B, scene manager 23 can rescale to allow the listener to proceed to lobby 2002C (where rescaling can be adjusted to the size of the minimap, which can be assumed to be as follows). Figure 20 (As illustrated). Scene Manager 23 can perform coordinate processing to ensure continuity between rooms 2002A-2002C (which may involve reorientation and / or translation). In some cases, if the listener walks next to a wall, the room the listener may be transitioning to may be inaccessible.

[0164] Return to reference Figure 17 For example, Scene Manager 23 can handle changes to physical locations that alter or otherwise change the dimensions of the physical location (e.g., opening / closing curtains, opening / closing a door). Scene Manager 23 can interface with Wearable Device 800 to perform mesh rescanning, where Scene Manager 23 can present an API or other dynamic interface through which a rescaling factor is passed to Renderer 22 when scene events occur. In this example, Scene Manager 23 can perform geometry manipulation (e.g., meshes, voxels, basic geometry, etc.).

[0165] Additionally, the scene manager 23 can dynamically change the rescaling factor as noted above. These changes can be applied during scene initialization (which can occur during the reproduction of the sound field), where, for example, a listener can begin experiencing the scene via a zoom-out effect. In some cases, an adaptation process exists whenever a transition occurs, which may include timed rescaling translations (1730), which can be applied to various aspects of the automatic rescaling techniques described above.

[0166] Figure 12 This is a block diagram illustrating various aspects of implementing offset-based rescaling techniques to promote increased immersion in an example physical space while consuming expanded realistic scenes. Figure 12 In the example, source space 5500A (which may also be referred to as source location 5500A) and physical playback space (PPS) 5500B (which may also be referred to as playback location 5500B) are shown.

[0167] Source space 5500A represents the concert venue (in this example), and stage 5502 extends from the concert venue via a loudspeaker (for illustrative purposes, in...). Figure 12 (Not shown in the example) emits an audio sound field 5504. The source space 5500A also includes microphones 5506A and 5506B (“microphone 5506”) that capture audio data representing the audio sound field 5504. Content capture device 300 (in...) Figure 1A (As shown in the example) can capture audio data representing an audio sound field 5504, which is in various audio formats (such as scene-based formats (which can be defined via HOA audio formats), object-based formats, channel-based formats, etc.).

[0168] The sound field representation generator 302 can generate an audio bitstream 21 of specified audio elements (one or more of the aforementioned audio formats) based on audio data. The sound field representation generator 302 can specify the aforementioned syntax elements in the audio bitstream 21, such as a rescaling syntax element that specifies the rescaling factor relative to (typically denoted as "wrt") the positioning (or in other words, location) of the audio element, and an auto-rescaling syntax element that specifies whether (e.g., a boolean value) the rescaling factor is generated based on an environment dimension (e.g., the dimension of physical location 5500B, which may also be referred to as the playback position).

[0169] exist Figure 12 In the example, source space 5500A is a square with dimensions of 20M in width and 20M in length. Although not shown, source space 5500A may also include a height dimension (e.g., 20M). Sound field representation generator 302 may specify one or more syntax elements in audio bitstream 21 that specify one or more of the width, space, and / or height of source space 5500A. Sound field representation generator 302 may use any form of syntax element (including reference syntax elements indicating that one or more dimensions are the same as the other dimensions of source space 5500A) to specify one or more syntax elements specifying one or more of the width, space, and / or height of source space 5500A.

[0170] Microphone 5506 can capture audio data representing the audio sound field 5504 in a manner that preserves how the audio sound field 5504 arrives at each individual microphone 5506, including its orientation, angle of incidence, etc. Microphone 5506 can represent the ability to capture audio data in 360 degrees to fully represent the audio sound field 5504. ®Or other 3D microphones. In this respect, microphone 5506 can refer to an example of microphone 5 shown in the examples of Figures 33A and 33B. Microphone 5506 can therefore generate audio data reflecting specific aspects of the audio sound field 5504, including the angle of incidence 5508A (for microphone 5506A, but there is also an angle of incidence 5508B that is not shown for microphone 5506B for illustrative purposes).

[0171] In this way, the sound field generator device 302 can generate an audio bitstream 21 to represent the audio elements captured by the microphone 5506 in 3D, allowing the reproduction of the audio sound field 5504 to occur in a 6DOF system that provides a highly immersive experience. However, the PPS 5500B has a significantly different dimension, as it is 6M wide and 6M long (and possibly 6M high or some other height).

[0172] In this example, the microphones in source space 5500A are located approximately 10 meters apart, while PPS 5500B includes loudspeakers 5510A and 5510B (“loudspeaker 5510”) spaced approximately 3 meters apart. Audio playback system 16A can receive audio bitstream 21 and parse (e.g., by invoking audio decoding device 24) audio elements representing audio sound field 5504 from audio bitstream 21. Audio playback system 16A can also (e.g., by invoking audio decoding device 24) parse the various syntax elements indicated above for relative rescaling and automatic rescaling of audio elements.

[0173] When initializing the audio playback system 16A for playback in the PPS 5500B, a user of the content consumer device 14 can input the dimensions of the PPS 5500B (e.g., one or more of height, length, and width) via, for example, a user interface or by allowing the user to move around the PPS 5500B. In some examples, the content consumer device 14 can automatically determine the dimensions of the PPS 5500B (e.g., using a camera, infrared sensing device, radar, lidar, ultrasound, etc.).

[0174] In any case, the audio playback system 16A can obtain both the dimensions of the source space 500A and the dimensions of the PPS 5500B. The audio playback system 16A can then process syntax elements for relative and automatic rescaling to identify how and which audio elements specified in the audio bitstream 21 will be rescaled. Assuming that in this example the audio sound field 5504 will be rescaled as indicated by the syntax elements, the audio playback system 16A can determine the rescaling factor 5512 based on the dimensions of the source space 5500A and the dimensions of the PPS 5500B.

[0175] As noted above, the audio playback system 16A can also determine an offset relative to the source origin 5550 (which can also be referred to as the virtual environment origin of a virtual environment not captured in the physical source location / space). Bitstream 21 can define the source dimension of the virtual environment (e.g., the source dimension of the source space 5500A) relative to the source origin 5550, where the source origin can be specified in bitstream 21 or implicitly determined due to its static characteristics (e.g., ...). Figure 12 (The upper left corner of the source space 5500A in the example). The audio playback system 16A may then be determined based on one of the anchor position 5552 and the playback space origin 5554, as well as the source origin 5550, which can be obtained in the manner described herein (e.g., automatically by the audio playback system 16A).

[0176] exist Figure 12 In the example, the audio playback system 16A can determine the rescaling factor as the width of PPS 5500B (6M) divided by the width of the source space 5500A (20M), resulting in a rescaling factor of 0.3, or in other words, 30%. The audio playback system 16A can then modify the position of the audio elements representing the audio sound field 5504 captured by the microphone 5506A to obtain a modified position of the audio sound field 5504 captured by the microphone 5506A that is 30% closer to the reference position (e.g., the reference position of the center (0,0,0) of the XR scene (or in other words, the XR world)) compared to the position of the microphone 506A relative to the center of the XR scene / world.

[0177] In other words, the audio playback system 16A can represent the position of audio elements that indicate the audio sound field 5504 captured by the microphone 5506A (as described above relative to...). Figure 1A After applying the offset as described in the example, the position of the audio elements representing the modified audio sound field 5504 captured by microphone 5506A is multiplied by a rescaling factor to obtain the position of the modified audio elements. Given that the approximate dimension of source space 5500A is uniform compared to the dimension of PPS 5500B (meaning that the width and length of source space 5500A are proportionally linearly equivalent to the dimension of PPS 5500B), audio playback system 16A can apply a rescaling factor to modify the position of the audio elements representing the audio sound field 5504 captured by microphone 5506A (possibly as described above relative to the dimensions of PPS 5500B). Figure 1A (After the offset is applied as described) without changing the angle of incidence 5508B during reproduction / playback by loudspeaker 5510.

[0178] Using scaling provided by various aspects of the techniques described in this disclosure (which refers to another way of rescaling), the audio playback system 16A (which refers to another way of XR playback system, playback system, etc.) can improve the reproduction of the sound field to modify the position of the audio source (which refers to another way of audio elements resolved from audio bitstream 21) to fit the size of the PPS 5500B. In implementing such scaling, the audio playback system 16A can improve the immersive experience for the user when consuming an XR scene, provided that the XR scene more closely matches the playback space. The user can then safely experience the entire XR scene within the limits of the allowed playback space. In this respect, these techniques can improve the operation of the audio playback system 16A itself.

[0179] Return to Figure 1A As an example, and as another example, renderer initializer 26 can obtain the listener location within the virtual environment in the manner described above. Renderer initializer 26 can then obtain an offset 27 for the playback position based on the source origin and the listener location. The following is relative to... Figure 12 The examples provide more information about listener positioning.

[0180] Figure 13 This is a block diagram illustrating an example audio scene in which offset-based rescaling can be performed according to various aspects of the technology. In a particular implementation, source device 12 corresponds to an encoder device that receives audio data (which refers to scene-based audio data 11) from multiple audio sources, such as an audio stream set 11, and encodes the audio data for transmission to audio playback system 16A via bit stream 21. In one example, the audio data encoded by source device 12 and included in bit stream 21 includes immersive sound data and corresponds to at least one of 2D audio data representing a two-dimensional (2D) sound field or 3D audio data representing a three-dimensional (3D) sound field. As used herein, "immersive sound data" includes a set of one or more immersive sound coefficients representing a sound field. In another example, the audio data is in a conventional channel-based audio channel format, such as a 5.1 surround sound format. In yet another example, the audio data includes audio data in an object-based format.

[0181] Source device 12 also receives audio from source 1312 (in Figure 12The example shown uses associated source metadata (e.g., source location and orientation data) to assign audio sources 1312 to one or more groups based on one or more source interval metrics, assign rendering modes to each group, and transmit group metadata associated with the one or more groups to the audio playback system 16A. The group metadata may include indications of which audio sources are assigned to each group, source interval metrics, rendering modes, other data corresponding to the audio source groups, or combinations thereof. In some implementations, one or more components of the group metadata are sent to the audio playback system 16A as bits in bitstream 21. In some examples, one or more components of the group metadata may be transmitted to the audio playback system 16A via one or more syntax elements (such as one or more elements of a defined bitstream syntax) to achieve efficient storage and streaming of the group metadata.

[0182] The audio streaming device 12 includes one or more processors configured to perform operations associated with audio processing. For example, the one or more processors are configured to acquire a set of audio streams 11 associated with a set of audio sources 1312 during audio encoding operations. For instance, the audio sources 1312 may correspond to microphones (such as microphone 5) that may be integrated into or coupled to the audio streaming device 12. For example, in some specific implementations, the audio streaming device 12 includes one or more microphones 5 coupled to one or more processors and configured to provide microphone data representing the sound of at least one audio source in the audio sources 1312.

[0183] One or more processors are configured to obtain group assignment information indicating that a specific audio source in the audio source set 1312 is assigned to a specific audio source group, which is associated with a source spacing condition. According to one aspect, one or more processors are configured to receive source location information indicating the location of each audio source in the audio source 1312, and to perform interval-based source grouping based on the source spacing condition to generate the group assignment information. In one example, audio sources 1312 (and / or audio streams 11 of audio sources 1312) that satisfy the source spacing condition are assigned to a first group (or distributed across multiple first groups), and audio sources 1312 (and / or audio streams 11 of audio sources 1312) that do not satisfy the source spacing condition (or satisfy another source spacing condition) are assigned to a second group (or distributed across multiple second groups). The group assignment information includes data indicating which audio source 1312 is assigned to which group. For example, each group may be represented by a data structure that includes a group identifier (“groupID”) for that group, and may also include an indication of which audio sources 1312 (and / or audio streams 11) are assigned to that group. In another example, group assignment information may include a value associated with each audio source in the audio sources 1312, indicating which group (if any) the audio source 1312 belongs to.

[0184] According to one aspect, the source spacing condition corresponds to whether a source spacing metric associated with the spacing between audio sources 1312 satisfies one or more thresholds. In a particular example, one or more processors of the source device 12 are configured to generate group assignment information based at least in part on a comparison of one or more source spacing metrics with thresholds. For illustration, one or more source spacing metrics may include the distance between audio sources 1312, the source localization density of a particular audio source 1312, or a combination thereof.

[0185] One or more processors of source device 12 are configured to generate output data that includes an encoded version of audio stream set 11 and group assignment information. For example, source device 12 may include a modem coupled to one or more processors and configured to transmit output data to a decoder device, such as by transmitting group assignment information and an encoded version of audio stream 11 to audio playback system 16A via bitstream 21.

[0186] Optionally, in some implementations, one or more processors of source device 12 are further configured to determine a rendering mode for each particular group of audio sources and include an indication of the rendering mode in the output data. In one example, one or more processors of source device 12 are configured to select a rendering mode from a plurality of rendering modes supported by decoder devices such as headphone 104 (and / or different audio playback devices) . For example, the plurality of rendering modes may include a baseline rendering mode that performs signal processing, source direction analysis, and source interpolation in the frequency domain and a mode that performs distance-weighted time-domain interpolation. The baseline rendering mode may correspond to a first mode, and the distance-weighted time-domain interpolation may correspond to a second mode, both of which are supported by renderer 22 of audio playback system 16A.

[0187] Rendering mode assignment is generated based on groups, indicating which rendering mode is assigned to which group. For example, the rendering mode of a group can be included as a data value in the group's data structure. For illustration, a first data structure for the first group 1392 may include a group identifier having a value "1" indicating the first group 1392, and a rendering mode indicator having a first value indicating the first mode (e.g., a Boolean value "0" when only two rendering modes are supported, or an integer value "1" when more than two modes are supported). A second data structure for the second group 1394 may include a group identifier having a value "2" indicating the second group 1394, and a rendering mode indicator having a second value indicating the second mode (e.g., a Boolean value "1" or an integer value "2").

[0188] Table 1 shows an illustrative, non-limiting example of the bitstream syntax associated with a high-order atmos group (hoaGroup). In Table 1, the value of the parameter hoaGroupLC for each hoaGroup is read from the bitstream. The hoaGroupLC parameter is a Boolean value: a "0" value indicates that the baseline rendering mode will be used for the group, and a "1" value indicates that a low-complexity rendering mode will be used.

[0189]

[0190] In a particular implementation, audio playback system 16A corresponds to a decoder device that receives encoded audio data (e.g., audio stream 11) from audio streaming device 12 via audio bitstream 21. Audio playback system 16A also obtains metadata associated with the audio data from audio streaming device 12, such as source metadata (e.g., source location and orientation data) and group metadata (e.g., group assignment information, source spacing metrics, rendering modes, or combinations thereof). In some implementations, one or more components of group metadata, source metadata, or both are extracted from bits transmitted in audio bitstream 21. In some examples, one or more components of group metadata, source metadata, or both may be received from audio streaming device 12 via one or more syntax elements.

[0191] The audio playback system 16A renders received audio data to generate an output speaker feed 25 (and / or other audio signals) based on listener positioning 1396, group assignment information, and rendering mode information. For example, the audio playback system 16A may select one or more audio sources 1312 for rendering based on listener positioning 1396 relative to various audio sources 1312, based on the sound type represented by various audio streams 11 (e.g., determined by one or more classifiers at the audio streaming device 12 and / or the audio playback system 16A), or both. The selected audio sources 1312 may be included in one or more groups, and the audio stream 11 of the selected audio source 1312 in each group may be rendered according to the rendering mode assigned to that group.

[0192] The audio playback system 16A includes one or more processors configured to perform operations associated with audio processing. Figure 1A (Not shown in the example). For illustration, one or more processors of the audio playback system 16A are configured to acquire a set of audio streams 11 (as represented by the immersive audio element 15) associated with the set of audio sources 1312 during an audio decoding operation. At least one audio stream in the set of audio streams 11 is received from an encoder device (e.g., audio streaming device 12) via an audio bitstream 21. In one example, the audio playback system 16A includes a modem coupled to one or more processors and configured to receive at least one audio stream 11 from the set of audio streams 11 via the audio bitstream 21 from the audio streaming device 12.

[0193] One or more processors of the audio playback system 16A are configured to receive group assignment information that indicates a specific audio source in the audio source set 1312 is assigned to a specific audio source group associated with a source spacing condition. The group assignment information may be received via audio bitstream 21 (e.g., in group metadata). In some implementations, the audio playback system 16A also updates the received group assignment information.

[0194] One or more processors of the audio playback system 16A are configured to obtain listener positioning 1396 associated with the posture of a user (also referred to as a “listener”) of the audio playback system 16A. For example, in some specific embodiments, the audio playback system 16A includes a wearable device 800 that includes or is coupled to one or more sensors. Figure 1A The tracking device 306 shown in the example is configured to generate sensor data indicating the movement, pose, or a combination thereof of the audio playback system 16A. As used herein, the “pose” of the audio playback system 16A (or the user’s head) indicates the position and orientation of the audio playback system 16A (or the user’s head), collectively referred to as listener positioning 1396. One or more processors of the audio playback system 16A may use listener positioning 1396 to select which audio stream 11 (as represented by the immersive audio element 15) to render based on the listener’s position, and may also use listener positioning 1396 during rendering to apply rotation, multi-source interpolation, or a combination thereof based on the listener’s orientation and / or position in the audio scene 1390.

[0195] One or more sensors include one or more inertial sensors such as accelerometers, gyroscopes, compasses, positioning sensors (e.g., Global Positioning System (GPS) receivers), magnetometers, inclinometers, optical sensors, and one or more other sensors for detecting acceleration, position, velocity, angular orientation, angular velocity, angular acceleration, or any combination thereof for the audio playback system 16A. In one example, one or more sensors include GPS, electronic maps, and electronic compasses that use inertial and magnetic sensor technology to determine orientation, such as a 3-axis magnetometer for measuring the Earth's geomagnetic field and a 3-axis accelerometer for providing a horizontal reference to the Earth's magnetic field vector based on the direction of gravity. In some examples, one or more sensors include one or more optical sensors (e.g., cameras) to track movement individually or in combination with one or more other sensors (e.g., inertial sensors).

[0196] The resulting output speaker feed 25 is provided for playback by the speaker. According to some aspects, the speaker is a headphone speaker, and the output speaker feed 25 corresponds to a binaural signal. For example, the renderer 22 may be configured to perform one or more sound field rotations based on the orientation of the audio source 1312 and / or the interpolated audio source and the listener's orientation, and to perform binauralization using a head-related transfer function (HRTF) to generate a realistic representation of the audio scene 1390 for a headphone-wearing user and based on the listener's specific position and orientation relative to the audio source 1312 within the audio scene 1390.

[0197] While the above examples describe rendering audio associated with five audio sources 1312 grouped into two groups, it should be understood that the techniques of the present invention can be used with any number of audio sources 1312 grouped into any number of groups. According to some specific implementations, as a non-limiting example, rendering at the audio playback system 16A may be limited to a set number of audio sources 1312 to be rendered per group, such as three audio sources per group. Rendering mode selection may include comparing the position of each audio source 1312 in the group with the listener's position to select a set number (e.g., three) of audio sources from the group that are closest to the listener's position. Rendering mode selection (as implemented by the renderer generation unit 32) may also be based on listener positioning 1396 to select which group to render from multiple groups capturing the same sound (e.g., a waterfall).

[0198] Although the examples included herein describe audio stream 11 as corresponding to audio data from a corresponding microphone, in other examples, one or more audio streams in audio stream 11 may correspond to a media file, audio generated at a game engine, one or more other sources of sound information, or a combination thereof. For illustration, audio streaming device 12 may obtain one or more audio streams in audio stream 11 from a storage device coupled to one or more processors or from a game engine included in one or more processors. As another example, in addition to receiving audio stream 11 via audio bitstream 21, audio playback system 16A may also obtain one or more audio streams locally, such as from a microphone coupled to audio playback system 16A, from a storage device coupled to one or more processors, or from a game engine included in one or more processors.

[0199] Although in some examples, for illustrative purposes, the audio playback system 16A is described as a headphone device, in other specific implementations, the audio playback system 16A (and / or the audio streaming device 12) is implemented as another type of device. In some specific implementations, the audio playback system 16A (e.g., one or more processors), the audio streaming device 12 (e.g., one or more processors), or both are integrated into a head-mounted device, such as... Figure 5A and / or Figure 5BAs depicted in the illustration. In the exemplary examples, the head-mounted device corresponds to at least one of a virtual reality head-mounted device, a mixed reality head-mounted device, or an augmented reality head-mounted device. In some embodiments, the audio playback system 16A (and / or the audio streaming device 12) is integrated into at least one of a mobile phone or tablet computer device or a camera device. In some embodiments, the audio playback system 16A (and / or the audio streaming device 12) is integrated into a wireless speaker and a voice activation device. In some embodiments, the audio playback system 16A (and / or the audio streaming device 12) is integrated into a vehicle.

[0200] While the various examples described for System 10 and also for the system depicted in the following figures correspond to specific implementations in which the output speaker feed 25 is a binaural output signal, in other implementations, the output speaker feed 25 has a format other than binaural. As an illustrative and non-limiting example, in some implementations, the output speaker feed 25 corresponds to an output stereo signal for playback (or for transmission to another device) at a loudspeaker integrated into or coupled to the audio playback system 16A. In other implementations, the output speaker feed 25, provided by one or more processors, may have one or more other formats and is not limited to binaural or stereo. Further information regarding packet-based rendering is described in U.S. Provisional Application No. 63 / 512,482, filed July 7, 2023.

[0201] Return to Figure 1A As another example, the renderer initializer 26 can obtain a virtual origin for the virtual environment reproduced within the playback position (which can be defined by the listener, automatically defined by the audio playback system 16A, or defined by other automatic or preset methods). The renderer initializer 26 can obtain an offset 27 for the playback position based on the source origin and the virtual origin.

[0202] Scene manager 23 may receive offset 27 (again, possibly along with a rescaling factor) as additional metadata. That is, scene manager 23 may present an application programming interface (API) or other type of interface (including a hardware-based interface when implemented only in hardware) that is extended to facilitate the definition of offset 27 (and again, the rescaling factor). Scene manager 23 may perform rescaling relative to (e.g., decoded from audio bitstream 21) the immersive audio element 15 based on the obtained offset 27, using scene-based grouping or within scene-based grouping. More information on rescaling can be found in U.S. Patent Application No. 18 / 045,839, filed October 12, 2022, entitled “SCALING AUDIO SOURCES IN EXTENDED REALITY SYSTEMS”. Audio playback system 16A may then reproduce the immersive audio element 15 based on offset 27 to obtain output audio signal 25 (which refers to another way of referring to speaker feed 25).

[0203] Alternatively, renderer initializer 26 may locally offset the source origin and perform rescaling to initialize audio renderer 22. Initializing audio renderer 22 may result in adapting audio renderer 22 to configure various translations, thereby expanding or reducing the rendering of the sound field. In this case, renderer initializer 26 may adapt transformations and / or translations (e.g., coefficients involved in translation and / or rendering modes) to change the rendering of immersive sound audio element 15 to reposition (or in other words, scale or rescale) the position of immersive sound audio element 15 to potentially more effectively reproduce the sound field expected by the content consumer within the physical location of the virtual environment being presented.

[0204] Using an offset 27 provided for scaling according to various aspects of the techniques described in this disclosure (which may be referred to as "offset scaling techniques"), the audio playback system 16A can improve the reproduction of the sound field to modify the origin used during scaling to fit the size of the playback space. In achieving this scaling, given that the extended reality scene more closely matches the geometry of the playback location (which may also be referred to as the "playback space"), the audio playback system 16A can improve the immersive experience for the user when consuming the extended reality scene. The user can then safely experience the entire extended reality scene within the limits of the permitted playback space. At this point, these techniques improve the operation of the audio playback system 16A itself (which may represent an example of an XR playback device). Relative to... Figure 14 and Figure 15 The example describes additional information about offset-based rescaling.

[0205] Figure 14This is a more detailed illustration of the renderer initializer of Figure 1 configured to implement offset-based scaling techniques. Figure 14 In the example, renderer initializer 26 may include a plurality of different initializers 4000A to 4000C (which may be collectively referred to as “initializer 4000”), which are configured to perform various aspects of the offset-based scaling techniques described in this disclosure.

[0206] The renderer initializer 26 may invoke the scene-based audio initializer 4000A in response to an audio bitstream 21 specifying that the bitstream includes HOA audio data (or other scene-based audio formats). The audio bitstream 21 may define a source origin for representing the sound field in the virtual environment where the source location is reproduced. The scene-based audio initializer 4000A may operate relative to the scene-based audio data to determine a rescaling offset 4002A to the origin based on the playback position, in a manner described above with respect to Figure 1. The scene-based audio initializer 4000A may also obtain and output a rescaling factor 4002B, as described in more detail below. The scene-based audio initializer 4000A may rescale the offset 4002A (which may represent...) Figure 1A The example shown in the example has an offset of 27) and a rescaling factor of 4002B as additional metadata 4004, which is output to the multipoint audio element processor 4800 (which is relative to the example above). Figure 16 (An example is described in more detail). In other words, the Multipoint Audio Element (MPAE) processor 4800 can represent, as described above, relative to... Figure 1A The example shown is a part of Scene Manager 23.

[0207] The renderer initializer 26 may also include an object initializer 4000B, which can be invoked by the renderer initializer 26 when the audio bitstream 21 indicates that the immersive audio element 15 includes at least one audio object. The object initializer 4000B obtains the source location and, based on the source location, obtains a rescaling offset 4002A (and potentially, a rescaling factor 4002B), which is again output as metadata to the MPAE processor 4800.

[0208] The renderer initializer 26 may also include a channel-based audio initializer 4000C, which the renderer initializer 26 may invoke when the audio bitstream 21 indicates that the immersive audio element 15 includes at least one channel-based audio element. The object initializer 4000B obtains the source location and, based on the source location, obtains a rescaling offset 4002A (and potentially, a rescaling factor 4002B), which is again output as metadata to the MPAE processor 4800.

[0209] In each case, the MPAE processor 4800 can operate as described below to render speaker feed 25 that can potentially be better adapted to the playback position, thereby creating a more immersive user experience. The following is relative to... Figure 15 The process for obtaining this additional metadata (e.g., rescaling offset 4002A and rescaling factor 4002B) is described, while Figure 16 An example of the MPAE processor 4800 is shown in more detail.

[0210] In other words, when performing automatic rescaling of an XR scene, it may be important to determine which origin to use when applying the rescaling translation to the positioning of the audio source element (which refers to the immersive audio element 15). Contextual understanding of the XR playback space (which refers to the playback position) is used to determine whether the anchor positioning is used as the new origin for rescaling or whether another origin is used. Other points could be the virtual world coordinate origin, the real-world playback space origin, or the listener positioning 1396. As discussed throughout this disclosure, when performing rescaling, an offset relative to the new origin is applied to the immersive audio element 15.

[0211] Figure 15 Examples of providing through Figures 1A to 14 The example shown is a flowchart illustrating example operations of a renderer initializer when performing various aspects of the offset-based scaling techniques described in this disclosure. Renderer initializer 153 may first initialize or otherwise obtain one or more scaling factors (4100). For example, renderer initializer 26 may initialize a rescaling offset 4002A (which is another way of referring to rescaling offset 4002A) and a rescaling factor 4002B to 0.0 and 1.0, respectively. As an example, renderer initializer 26 may initialize two 3D vectors (rescaleOffset and rescale) to (respectively) store the offset and scaling factor for each dimension.

[0212] Renderer initializer 26 may then perform a parent retrieval (4102). During the parent retrieval, renderer initializer 26 may retrieve the parent (e.g., HOA group) of the panoramic sound audio element 15 to find the source space through the region associated with the HOA group (discussed in more detail above). Although described relative to HOA groups, renderer initializer 26 may perform a parent retrieval relative to a single panoramic sound audio element 15, which may not represent a HOA group but any form of audio element or audio group, whether defined as HOA audio data or other scene-based audio data, audio objects, and / or channels.

[0213] The renderer initializer 26 can then implement anchor looping (4104) (e.g., when configured to use anchor positioning, but different reference origins can also be utilized as noted elsewhere). During anchor looping (which is looping because anchor looping iterates over each of the Atmos audio elements in Atmos audio element 15), the renderer initializer 26 can find the source space by the region associated with the corresponding Atmos audio element 15 (again, assuming the HOA group as an example, but not limited to HOA audio data or other scene-based audio data). For each anchor, the renderer initializer 26 can verify that the name of the associated Atmos audio element in Atmos audio element 15 matches the name of the parent and verify whether automatic rescaling is enabled.

[0214] After completing the anchor loop (and verifying name matching), renderer initializer 26 may then perform geometry retrieval (4106). During geometry retrieval, renderer initializer 26 may retrieve the current region and all geometry associated with the scene. After retrieving the current region and all geometry associated with the scene, renderer initializer 26 may perform geometry loop (4108), where renderer initializer 26 may loop over all geometry to identify regions for the panoramic sound audio elements 15 (e.g., HOA groups) and regions for the listener space.

[0215] The renderer initializer 26 can then calculate the rescaling offset 4002A (4110) based on the reference origin of the immersive audio element 15. In this example, the reference origin may refer to the anchor point, but in other examples, the renderer initializer 26 may be configured to select the listener space origin (e.g., when no anchor point is defined), the listener location 1396, or the virtual world origin.

[0216] After and / or simultaneously calculating the rescaling offset 4002A, renderer initializer 26 obtains the bounding box and tolerance (4112). Renderer initializer 26 obtains the tolerance level, retrieves the parent of the geometry, and obtains the bounding boxes of both the listener space and the audio element region.

[0217] Renderer initializer 26 may continue to calculate or otherwise obtain the rescaling factor 4002B (4114). When the rescaling factor 4002B is obtained, renderer initializer 26 may loop through the bounding box of the region, thereby calculating a new scaling factor 4002B and offset for each dimension (e.g., x, y, z). Renderer initializer 26 may calculate or otherwise obtain the rescaling factor 4002B based on the ratio of the listener space to the dimension of the audio element region, adjusted by the tolerance level. Renderer initializer 26 may be configured to cap the scaling factor or otherwise limit it to 1.0.

[0218] The renderer initializer 26 can then apply a rescaling factor 4002B (4116), whereby the renderer initializer 26 adjusts the immersive audio element 15 based on the final rescaling factor 4002B and the rescaling offset 4002A. In one example, the renderer initializer 26 may subtract the rescaling offset 4002A (e.g., rescaleOffset_x, rescaleOffset_y, rescaleOffset_z) from the source coordinates for the audio element 15. In this example, the renderer initializer 26 may multiply the result by the corresponding (e.g., for each x, y, z dimension) rescaling factor 4002B and add the corresponding (e.g., for each x, y, z dimension) back to the rescaling coordinates. Further details regarding the acquisition and application of the rescaling factor are described below in more detail in the aforementioned U.S. Patent Application No. 18 / 045,839, filed October 12, 2022, entitled “SCALING AUDIO SOURCES INEXTENDED REALITY SYSTEMS”. The renderer initializer 26 may then output the rescaled audio element (4118) as the adjusted audio element.

[0219] Figures 7A to 7C This is an example Figure 1A and Figure 1B The example shown illustrates various aspects of the extended reality system's operation when performing tolerance-modified rescaling techniques. First, refer to... Figure 7A An example is shown in real-world space 801A, where the scene manager 23 of the extended reality system 10 (in) Figure 1A and / or Figure 1B(As shown in the example) A tolerance of 50 is obtained to indicate a strictness of 100% (in at least two of the three dimensions, namely the x-axis and y-axis, with the depth (z-axis) not shown). Strictness can be defined as one (1) minus the tolerance of 50 (or in other words, 1 - tolerance). Strictness can represent the percentage of source locations contained in real-world space.

[0220] With 100% strictness, there is no tolerance (0%) for the vertical or height axis (e.g., the y-axis). Object 802 (which is depicted as a sofa in this example) is excluded from real-world space, and source position 804A is restricted to the top of object 802. Although source position 804A is larger than real-world space 801A (meaning, in this example, extending towards the floor of real-world space 801A), scene manager 23 can adapt the audio element to be above object 802 due to the potential incomplete or irregular representation of real-world space 801A caused by object 802.

[0221] exist Figure 7B In the example, real-world space 801B represents a physical location where the extended reality system 10 is configured to reproduce an audio sound field represented by audio bitstream 21. In this example, the strictness is set to 50%, and the tolerance is also at 50%, which results in the source location 804B being restricted to 50% of the object 802, thereby restricting the source location 804B to 50% of the height of the object 802.

[0222] exist Figure 7C In the example, real-world space 801C represents a physical location where the extended reality system 10 is configured to reproduce the audio sound field represented by audio bitstream 21. In this example, the strictness is set to 0%, and the tolerance is also at %1000, which causes the source location 804C to be constrained to 0% of the height of object 802, thus constraining the source location 804C to 0% of the height of object 802. Scene manager 23 may not adapt to source location 804C, resulting in the source dimension (i.e., the y-axis) being the same and not being scaled due to the tolerance being defined as 100% 50.

[0223] Figures 8A to 8C This is an example Figure 1A and Figure 1B Additional illustrations show sample operations of the extended reality system in performing various aspects of tolerance-modified rescaling techniques, as shown in the examples. First, refer to... Figure 8A An example is shown in real-world space 900A, where the scene manager 23 of the extended reality system 10 (in) Figure 1A and / or Figure 1B(As shown in the example) A tolerance of 50 is obtained to indicate a strictness of 100% (in at least two of the three dimensions, namely the x-axis and y-axis, with the depth (z-axis) not shown). Strictness can be defined as one (1) minus the tolerance of 50 (or in other words, 1 - tolerance). Strictness can represent the percentage of source locations contained in real-world space.

[0224] With 100% strictness, there is no tolerance (0%) for the vertical axis or height axis (e.g., the y-axis). Objects 902A (described as a sofa in this example) and 902B (a floor lamp in this example) are excluded from real-world space, and source location 904A is restricted to the top of object 902B (meaning the highest object 902A and / or 902B). Although source location 904A is larger than real-world space 900A (meaning, in this example, extending towards the floor of real-world space 900A), scene manager 23 can adapt the audio element to be above object 902B due to the potential incomplete or irregular representation of real-world space 900A caused by object 902B.

[0225] exist Figure 8B In the example, real-world space 900B represents a physical location where the extended reality system 10 is configured to reproduce an audio sound field represented by audio bitstream 21. In this example, the strictness is set to 50%, and the tolerance is also at 50%, which results in the source location 904B being constrained to 50% of the object 902B, thereby constraining the source location 904B to 50% of the height of the object 902B.

[0226] exist Figure 8C In the example, real-world space 900C represents a physical location where the extended reality system 10 is configured to reproduce the audio sound field represented by audio bitstream 21. In this example, the strictness is set to 0%, and the tolerance is also at %1000, which causes the source location 904C to be constrained to 0% of the height of object 902B, thus constraining the source location 904C to 0% of the height of object 902B. Scene manager 23 may not adapt to source location 904C, resulting in the source dimension (i.e., the y-axis) being the same and not being scaled due to the tolerance being defined as 100% 50.

[0227] Figure 9A and Figure 9B This is an example Figure 1A and Figure 1B Further illustrations of example operations of the extended reality system shown in the example, when performing various aspects of tolerance-modified rescaling techniques. First refer to... Figure 9A An example is shown, illustrating a real-world space 1000A, in which the scene manager 23 of the extended reality system 10 (in...) Figure 1Aand / or Figure 1B (As shown in the example) a tolerance of 50 is obtained with an indication of strictness of 100% (in at least two of the three dimensions, namely the x-axis and y-axis, with the depth (z-axis) not shown).

[0228] In this example, the sound field representation generator 302 specifies a tolerance of 50 as a minimum / maximum value across three dimensions, where, in this example, the minimum / maximum value is defined differently for the height dimension than for the width dimension to accommodate the sloping wall. Figure 9A In the example, the minimum / maximum value of the width dimension at the maximum top height differs from the minimum / maximum value of the width dimension at the minimum floor height. Scene Manager 23 can generate a real-world space 1000A based on a rescaling factor modified by a tolerance of 50% for the source location 1004A.

[0229] With 100% strictness, there is no tolerance (0%) for the vertical axis or height axis (e.g., the y-axis), and the source location is controlled. In this case, the sound field extends 100% beyond the walls of a 900 Å real-world space, making certain spatial extents of the sound field inaccessible within a 1000 Å real-world space. In this example, the source location includes diagonal and / or slanted lines in a polyhedral configuration.

[0230] Next reference Figure 9B For example, real-world space 1000B represents a physical location where the extended reality system 10 is configured to reproduce an audio sound field represented by audio bitstream 21. In this example, the strictness is set to 50%, and the tolerance is also at 50%, which results in source location 1004B being constrained to 50% of real-world space 1000B (meaning source location 1004B is constrained to 50% of the width of real-world space 1000B). In this example, source location 1004B is constrained by 50% of real-world space 1000B, where the starting point of source location 1004B (starting from the floor) extends only 50% beyond real-world space 1000B.

[0231] Figure 10 This is an example Figure 1A and Figure 1B This is another illustration of example operation of the extended reality system shown in the example, demonstrating various aspects of the rescaling technique with tolerance modification. Real-world space 1100 represents physical location / space 1104, in which the extended reality system 10 is configured to reproduce the audio sound field represented by audio bitstream 21. In this example, the tolerance is set to 0% for the maximum y-dimension (height) and to 100% for the minimum y-dimension, thus allowing variation between the minimum and maximum values ​​in the y-dimension.

[0232] Figures 11A to 11C This is a diagram illustrating the syntax table for various aspects of rescaling techniques used to implement tolerance modifications. Figure 11A Syntax table 1200A, as shown in the example, provides syntax elements (i.e., id, lsdf_ref, audioSourceRescale, autoRescale, and tolerance in this example). Syntax table 1200A provides syntax elements for implementing / controlling spatial transformations used to place objects or other audio elements in an augmented reality (AR) setting (or other XR setting), where the positioning and orientation are fixed and can only be defined once before the scene begins.

[0233] `id` refers to the identifier, while `lsdf_ref` refers to the identifier of the corresponding LSDF anchor. `audioSourceRescale` refers to the rescaling factor for the location and dimensions of all audio source (object, channel, HOA source, HOA group) nodes to be applied to the transformations in the x, y, and z dimensions. `autoRescale` is a Boolean value that allows the renderer to determine the rescaling factor by comparing the acoustic environment region with the acoustic environment region defined in the LSDF, while `tolerance` is a control property for `autoRescale` that defines the percentage of scene area retained outside the LSDF region for each x, y, and z dimension.

[0234] Figure 11B The example shown in syntax table 1200B is similar to syntax table 1200A, except that tolerance refers to the control properties (maximum / minimum x-direction, maximum / minimum y-dimension, and maximum / minimum z-dimension) that define the percentage of scene area retained outside the LSDF area. Figure 11C The example shown in syntax table 1200C is similar to syntax tables 1200A and 1200B, except that the tolerance is replaced by strictness (which can be defined in the x, y, and z dimensions, or it can be the minimum / maximum of each of the x, y, and z dimensions).

[0235] Figure 1B This is a block diagram illustrating another example system 100 configured to perform various aspects of the techniques described in this disclosure. System 100 is similar to... Figure 1A The system 10 shown is different. Figure 1A The audio renderer 22 shown is replaced by a binaural renderer 102, which is capable of performing binaural rendering using one or more HRTFs or other functions that can render to the left and right speaker feeds 103.

[0236] The audio playback system 16B can output left and right speaker feeds 103 to headphones 104, which can represent another example of a wearable device and can be coupled to additional wearable devices (such as watches, the aforementioned VR headsets, smart glasses, smart clothing, smart rings, smart bracelets, or any other type of smart jewelry, including smart necklaces) to facilitate sound field reproduction. Headphones 104 can be coupled to additional wearable devices wirelessly or via a wired connection.

[0237] Additionally, the headphones 104 can be connected via a wired connection (such as a standard 3.5mm audio jack, a Universal System Bus (USB) connection, an optical audio jack, or other forms of wired connection) or wirelessly (such as via Bluetooth). ™ (Connectivity, wireless network connection, etc.) is coupled to the audio playback system 16. The headphones 104 can reproduce the sound field represented by the immersive sound coefficient 11 based on the left and right speaker feeds 103. The headphones 104 may include a left headphone speaker and a right headphone speaker, which are powered (or in other words, driven) by the corresponding left and right speaker feeds 103.

[0238] Although for such Figure 5A and Figure 5B The VR device illustrated in the example is described, but these technologies can be implemented by other types of wearable devices, including watches (such as so-called "smartwatches"), glasses (such as so-called "smart glasses"), headphones (including wireless headphones coupled via a wireless connection or smart headphones coupled via a wired or wireless connection), and any other type of wearable device. Therefore, these technologies can be implemented by any type of wearable device through which a user can interact with the wearable device while it is being worn.

[0239] Figure 2 This is a block diagram illustrating various aspects of implementing rescaling techniques to promote increased immersion in an example physical space while consuming expanded realistic scenes. Figure 2 The example shows source space 500A and physical playback space (PPS) 500B.

[0240] Source space 500A represents the concert venue (in this example), and stage 502 extends from the concert venue via a loudspeaker (for illustrative purposes, in...). Figure 2 (Not shown in the example) The audio sound field 504 is emitted. The source space 500A also includes microphones 506A and 506B (“microphone 506”) that capture audio data representing the audio sound field 504. The content capture device 300 can capture audio data representing the audio sound field 504 in various audio formats, such as scene-based formats (which can be defined via HOA audio formats), object-based formats, channel-based formats, etc.

[0241] The sound field representation generator 302 can generate an audio bitstream 21 of specified audio elements (one or more of the aforementioned audio formats) based on audio data. The sound field representation generator 302 can specify the aforementioned syntax elements in the audio bitstream 21, such as a rescaling syntax element that specifies the rescaling factor relative to (typically denoted as "wrt") the positioning (or in other words, location) of the audio element, and an auto-rescaling syntax element that specifies whether (e.g., a boolean value) the rescaling factor is generated based on the environment dimension (e.g., the dimension of PPS 500B).

[0242] exist Figure 2 In the example, source space 500A is a square with dimensions of 20M in width and 20M in length. Although not shown, source space 500A may also include a height dimension (e.g., 20M). Sound field representation generator 302 may specify one or more syntax elements in audio bitstream 21 that specify one or more of the width, space, and / or height of source space 500A. Sound field representation generator 302 may use any form of syntax element (including reference syntax elements indicating that one or more dimensions are the same as the other dimensions of source space 500A) to specify one or more syntax elements specifying one or more of the width, space, and / or height of source space 500A.

[0243] Microphone 506 can capture audio data representing the audio sound field 504 in a manner that preserves how the audio sound field 504 arrives at each individual microphone 506, including its orientation, angle of incidence, etc. Microphone 506 can represent an Eigenmike capable of capturing audio data in 360 degrees to fully represent the audio sound field 504. ® Or other 3D microphones. At this point, microphone 506 can be considered... Figure 1A and Figure 1B The example shown is microphone 5. Microphone 506 can therefore generate audio data that reflects a particular aspect of the audio sound field 504, including the angle of incidence 508A (for microphone 506A, but there is also an angle of incidence 508B which is not shown for microphone 506B for illustrative purposes).

[0244] In this way, the sound field generator device 302 can generate an audio bitstream 21 to represent the audio elements captured by the microphone 506 in 3D, making the reproduction of the audio sound field 504 possible in a 6DOF system that provides a highly immersive experience. However, the PPS 500B has a significantly different dimension, as the PPS 500B has a width of 6M and a length of 6M (and possibly a height of 6M or some other height).

[0245] In this example, the microphones in source space 500A are located approximately 10 meters apart, while PPS 500B includes loudspeakers 510A and 510B (“loudspeaker 510”) spaced approximately 3 meters apart. Audio playback system 16A can receive audio bitstream 21 and parse (e.g., by invoking audio decoding device 24) audio elements representing audio sound field 504 from audio bitstream 21. Audio playback system 16A can also (e.g., by invoking audio decoding device 24) parse the various syntax elements indicated above for relative rescaling and automatic rescaling of elements.

[0246] When initializing the audio playback system 16A for playback in the PPS 500B, a user of the content consumer device 14 can input the dimensions of the PPS 500B (e.g., one or more of height, length, and width) via, for example, a user interface or by allowing the user to move around the PPS 500B. In some examples, the content consumer device 14 can automatically determine the dimensions of the PPS 500B (e.g., using a camera, infrared sensing device, radar, lidar, ultrasound, etc.).

[0247] In any case, the audio playback system 16A can obtain both the dimensions of the source space 500A and the dimensions of the PPS 500B. The audio playback system 16A can then process syntax elements for relative and automatic rescaling to identify how and which audio elements specified in the audio bitstream 21 will be rescaled. Assuming that in this example the audio sound field 504 will be rescaled as indicated by the syntax elements, the audio playback system 16A can determine the rescaling factor 512 based on the dimensions of the source space 500A and the dimensions of the PPS 500B.

[0248] exist Figure 2 In the example, the audio playback system 16A can determine the rescaling factor as the width of PPS 500B (6M) divided by the width of the source space 500A (20M), resulting in a rescaling factor of 0.3, or in other words, 30%. The audio playback system 16A can then modify the position of the audio elements representing the audio sound field 504 captured by the microphone 506A to obtain a modified position of the audio sound field 504 captured by the microphone 506A that is 30% closer to a reference position (e.g., a reference position of the center (0, 0, 0) of the XR scene (or in other words, the XR world)) than the position of the microphone 506A relative to the center of the XR scene / world.

[0249] In other words, the audio playback system 16A can multiply the position of an audio element representing an audio sound field 504, as captured by microphone 506A, by a rescaling factor to obtain a modified position of the audio element representing the audio sound field 504, as captured by microphone 506A. Assuming that the approximate dimension of source space 500A is uniform compared to the dimension of PPS 500B (meaning the width and length of source space 500A are proportionally linearly equivalent to the dimension of PPS 500B), the audio playback system 16A can apply a rescaling factor to modify the position of the audio element representing the audio sound field 504, as captured by microphone 506A, without changing the angle of incidence 508B during reproduction / playback by loudspeaker 510.

[0250] Using scaling provided by various aspects of the techniques described in this disclosure (which refers to another way of rescaling), the audio playback system 16A (which refers to another way of XR playback system, playback system, etc.) can improve the reproduction of the sound field to modify the position of the audio source (which refers to another way of audio elements resolved from audio bitstream 21) to fit the size of the PPS 500B. In implementing such scaling, the audio playback system 16A can improve the immersive experience for the user when consuming an XR scene, provided that the XR scene more closely matches the playback space. The user can then safely experience the entire XR scene within the limits of the allowed playback space. In this respect, these techniques can improve the operation of the audio playback system 16A itself.

[0251] Figure 3A and Figure 3B This is a block diagram illustrating various aspects of implementing rescaling techniques to promote increased immersion in physical spaces while expanding the consumer's realistic scene. First, refer to... Figure 3A For example, source space 600A is the same or substantially similar to source space 500A in terms of dimensions (20M width by 20M length), while PPS 600B is different from PPS 500B in terms of dimensions (PPS 600B's 10M width by 20M length is compared to PPS 500B's 6M width by 6M length).

[0252] The audio playback system 16B can determine the aspect ratio of the source space 600A and the PPS 600B. Specifically, the audio playback system 16A can determine the source aspect ratio of the source space 600A based on the source width (20M) and source length (20M), resulting in a source aspect ratio of 1:1 for the source space 600A (in this example). The audio playback system 16A can also determine the aspect ratio of the PPS 600B based on a playback width of 10M and a playback length of 20M, resulting in a playback aspect ratio of 1:2 for the PPS 600B (in this example).

[0253] The audio playback system 16 can determine the difference between the playback aspect ratio (e.g., 1:1) and the source aspect ratio (e.g., 1:2). The audio playback system 16 can compare the difference between the playback aspect ratio and the source aspect ratio with a threshold difference. When the difference between the playback aspect ratio and the source aspect ratio exceeds the threshold difference, the audio playback system 16 can perform audio warping on audio elements representing an audio sound field 604 captured by microphone 606A (which could represent an example of microphone 506A, and microphone 606B could represent an example of microphone 506B) as in this example.

[0254] More information on audio warping can be found in the following references: 1) Zotter, F. et al., “Warping of the Recording Angle in Ambisonics,” presented at the First International Conference on Spatial Audio in Detmold, 2011; 2) Zotter, F. et al., “Warping of 3D Ambisonic Recordings,” presented at the Holographic Sound Workshop in Lexington, 2011; and 3) Kronlachner, Matthias et al., “Spatial Transformations for the Enhancement of Ambisonic Recordings,” presented in the proceedings of the Second International Conference on Spatial Audio in Erlangen, 2014. Audio warping can involve warping of the recording angle and directional loudness modification of the HOA.

[0255] That is, audio warping can involve the application of the following mathematical equation, where substitution is used to simplify the subsequent warping curve so as to express the manipulation of angle υ as follows:

[0256] µ=sin(υ), original,

[0257] Warp,

[0258] Orientation and deflection from the equator can occur using the following equation, which is another useful warp curve for maintaining equatorial altitude. The following equation is neutral for β=0, pushes surround sound content away from the equator to the poles for β>0, or pulls surround sound content towards the equator for β<0:

[0259]

[0260] The rescaling factor discussed in this disclosure can be defined as β = -(rescaling factor).

[0261] When rescaling occurs in the height dimension, the audio playback system 16A can apply the following warping toward the poles as illustrated in the following equation.

[0262]

[0263] This operator is for It is neutral, and depends on... The sign indicates that the original equator υ=0 has been raised or lowered. In other words, if the height of the XR space is small, the audio playback system 16A can push the sound towards the acoustic pole and thus set... =-(height rescaling factor).

[0264] Figure 3A The example illustrates how the audio playback system 16A can distort the reproduction of the audio sound field 604 produced by loudspeakers 610A and 610B (which may represent an example of loudspeaker 510 of the audio playback system 16A) by rescaling between different source aspect ratios and playback aspect ratios. Figure 3A As shown in the example, the angle of incidence 608A represents the angle of incidence of the audio elements of the audio sound field 604 captured by microphone 606A at stage 602 (which is Figure 2 (Another example of stage 502 shown in the example) is centered. However, before the reproduction is performed by the loudspeaker 610, the audio playback system 16A may cause a reproduction when rescaling the position of the audio element, which results in the incident angle 608B of the presented audio element being from the far right.

[0265] In this example, the audio playback system 16A can calculate a rescaling factor 612 that includes no warping (“no warping”). The captured sound field 612A is represented by a dashed circle to show that the audio sound field 604 is captured without warping. The reproduced sound field 612B is shown as a dashed circle to indicate that the reproduction of the audio sound field 604 (from the captured audio elements) is also not warped.

[0266] Next reference Figure 3B For example, when the difference between the playback aspect ratio and the source aspect ratio exceeds a threshold difference, the audio playback system 16A can perform audio warping for audio elements representing an audio sound field 604 as captured by microphone 606A in this example. In this example, the audio playback system 16A determines the width that causes the difference between the source aspect ratio and the playback aspect ratio. The audio playback system 16A can then perform audio warping based on the width difference between the source space 600A and PPS 600B to preserve the angle of incidence 608A of the audio elements representing the audio sound field 604 as captured by microphone 606A.

[0267] When performing audio warping, the audio playback system 16A can remove the coefficients associated with the zero-order spherical basis functions (or, in other words, the zero-order basis functions) when the audio element is defined using a higher-order atmos (HOA) conforming to a scene-based audio format, which has coefficients associated with zero-order and additional higher-order spherical basis functions such as first-order, second-order, etc.). The audio playback system 16A can then perform audio warping on the modified HOA coefficients (e.g., obtained after removing the coefficients associated with the zero-order basis functions) to preserve the incident angle 608A of the audio element and obtain the warped HOA coefficients. The audio playback system 16A can then render the warped HOA coefficients and the coefficients corresponding to the zero-order HOA coefficients based on the modified position (in this example, due to rescaling as discussed herein) to obtain one or more speaker feeds 25.

[0268] exist Figure 3B In the example, the audio playback system 16A may determine or otherwise obtain a rescaling factor 614 that includes the warping of the width (not length) difference between the source space 600A and PPS 600B. Audio warping results in a distorted reproduced sound field 612C to preserve the perceived angle of incidence 608A of the audio element as arriving from the center of stage 602. The angle of incidence 608C is warped to preserve the angle of incidence 608A against the background of PPS 600B and assumes that the reproduction of the audio sound field 604 has been rescaled. At this point, the audio playback system 16A may perform audio warping on the audio element based on the playback dimension and the source dimension to preserve the angle of incidence 608A of the audio element (during reproduction against the background of PPS 600B).

[0269] Figure 4A and Figure 4B This is a flowchart illustrating exemplary operation of the extended reality system shown in the example of Figure 1 when performing various aspects of the rescaling techniques described in this disclosure. First, refer to... Figure 4A For example, the sound field representation generator 302 can specify a syntax element (700) in the audio bitstream 21 that indicates a rescaling factor for audio elements. The rescaling factor can indicate how the position of an audio element will be rescaled relative to other audio elements. The sound field representation generator 302 can also specify a syntax element that indicates that automatic rescaling will be performed for the duration the audio element is present for playback. The sound field representation generator 302 can then output the audio bitstream 21 (702).

[0270] Next reference Figure 4BFor example, audio playback system 16A can receive audio bitstream 21. Audio playback system 16A can obtain playback dimensions (one or more playback dimensions) (750) associated with the physical space in which playback of audio bitstream 21 will occur. That is, audio playback system 16A can interface with a user operating audio playback system 16A to identify playback dimensions (e.g., through an interactive user interface in which the user moves around the room to identify playback dimensions) (and / or via a remote camera installed to capture playback of XR scenes, via a head-mounted camera for capturing playback / interaction with XR scenes, etc.).

[0271] In some cases, the audio playback system 16A may also extract from the audio bitstream 21 a syntax element indicating one or more source dimensions associated with the source space for the XR scene. This syntax element may be specified in a side channel of metadata or in other extensions to the audio bitstream 21. In any case, the audio playback system 16A may parse the source dimensions from the audio bitstream 21 (e.g., as syntax elements) or otherwise obtain the source dimensions associated with the source space for the XR scene (752).

[0272] The audio playback system 16A can then modify the position associated with the audio element based on the playback dimension and the source dimension to obtain the modified position of the audio element (754). At this point, the audio decoding device 24 can resolve the audio element from the audio bitstream 21, which is represented in the example of Figure 1 as a panoramic audio element 15 (which may represent one or more audio elements). Although described relative to the panoramic audio element 15, various other audio data formats can be rescaled according to various aspects of the technology, including object-based formats, channel-based formats, etc.

[0273] In performing audio rescaling, the audio playback system 16A can determine whether the difference between the source dimension and the playback dimension exceeds a threshold difference (e.g., the difference between the source dimension and the playback dimension is greater than 1%, 5%, 10%, 20%, etc.). If the difference between the source dimension and the playback dimension exceeds different thresholds (e.g., defined as 1%, 5%, 10%, 20%, etc.), the audio playback system 16A can then determine a rescaling factor. The audio playback system 16A can determine the rescaling factor as a function of the playback dimension divided by the source dimension.

[0274] In the case of linear rescaling, the audio playback system 16A can determine the rescaling factor as the playback dimension divided by the source dimension. For example, assuming the source dimension is 20M and the playback dimension is 6M, the audio playback system 16A can calculate the rescaling factor as 6M / 20M, which equals 0.3 or 30% as the rescaling factor. The audio playback system 16A can apply the audio factor (e.g., 30%) when the scene manager 23 is invoked.

[0275] The audio playback system 16A can invoke the scene manager 23, passing a rescaling factor to it. The scene manager 23 can apply the rescaling factor when processing audio elements extracted from the audio bitstream 21. The scene manager 23 can modify the metadata defining the position of the audio element in the XR scene, rescaling the metadata defining the position of the audio object to obtain the modified position of the audio object. The scene manager 23 can pass the modified position to the audio renderer 22, which can render the audio element to the speaker feed 25 (756) based on the modified position of the audio element. The audio renderer 22 can then output the speaker feed 25 for playback by the audio playback system 16A (which may include one or more loudspeakers configured to reproduce the sound field based on the speaker feed 25) (758).

[0276] Audio decoding device 24 can parse the aforementioned syntax element indicating the rescaling factor and the audio rescaling to be performed. When the rescaling factor is specified as described above, audio playback system 16A can directly obtain the rescaling factor from audio bitstream 21 (meaning that, in this case, the rescaling factor is not calculated as a function of the playback dimension divided by the source dimension). Audio playback system 16A can then apply the rescaling factor as described above to obtain the location of the modified audio element.

[0277] When processing a syntax element that indicates automatic rescaling, audio playback system 16A may suppress rescaling of audio elements associated with the auto-rescaling syntax element that indicates automatic rescaling will not be performed. Otherwise, audio playback system 16A may use various aspects of the techniques described in this disclosure to process audio elements associated with an audio rescaling syntax element that indicates automatic rescaling will be performed for the purpose of performing rescaling. The auto-rescaling syntax element may configure audio playback system 16A to continuously apply a rescaling factor to the associated audio elements as long as the associated audio elements are present in the XR scene.

[0278] Figure 6Examples of a wireless communication system 100 supporting audio streaming according to various aspects of this disclosure are illustrated. The wireless communication system 100 includes a base station 105, a user interface unit (UE) 115, and a core network 130. In some examples, the wireless communication system 100 may be a Long Term Evolution (LTE) network, an LTE-Advanced (LTE-A) network, an LTE-A Pro network, or a New Radio (NR) network. In some cases, the wireless communication system 100 may support enhanced broadband communication, ultra-reliable (e.g., mission-critical) communication, low-latency communication, or communication with low-cost and low-complexity devices.

[0279] Base station 105 can wirelessly communicate with UE 115 via one or more base station antennas. Base station 105 described herein may include, or may be referred to by those skilled in the art as, a base transceiver, wireless base station, access point, wireless transceiver, Node B, evolved Node B (eNB), next-generation Node B, or gigabit Node B (any of which may be referred to as gNB), home Node B, home evolved Node B, or some other suitable term. Wireless communication system 100 may include different types of base station 105 (e.g., macro cell base station or small cell base station). UE 115 described herein may be able to communicate with various types of base station 105 and network equipment (including macro eNB, small cell eNB, gNB, relay base station, etc.).

[0280] Each base station 105 may be associated with a specific geographic coverage area 110 in which communication with each UE 115 is supported. Each base station 105 may provide communication coverage to the corresponding geographic coverage area 110 via a communication link 125, and the communication link 125 between the base station 105 and the UE 115 may utilize one or more carriers. The communication link 125 shown in the wireless communication system 100 may include: an uplink transmission from the UE 115 to the base station 105 or a downlink transmission from the base station 105 to the UE 115. Downlink transmission may also be referred to as forward link transmission, and uplink transmission may also be referred to as reverse link transmission.

[0281] The geographic coverage area 110 for base station 105 can be divided into sectors, which constitute part of the geographic coverage area 110, and each sector can be associated with a cell. For example, each base station 105 can provide communication coverage for macro cells, small cells, hotspots, or other types of cells, or various combinations thereof. In some examples, base station 105 can be mobile and thus provide communication coverage to mobile geographic coverage areas 110. In some examples, different geographic coverage areas 110 associated with different technologies can overlap, and overlapping geographic coverage areas 110 associated with different technologies can be supported by the same base station 105 or different base stations 105. The wireless communication system 100 can include, for example, heterogeneous LTE / LTE-A / LTE-A Pro or NR networks, wherein different types of base stations 105 provide coverage to various geographic coverage areas 110.

[0282] UE 115 may be distributed throughout the wireless communication system 100, and each UE 115 may be stationary or mobile. UE 115 may also be referred to as a mobile device, wireless device, remote device, handheld device, subscriber device, or some other suitable term, wherein "device" may also be referred to as a unit, station, terminal, or client. UE 115 may also be a personal electronic device, such as a cellular phone, personal digital assistant (PDA), tablet computer, laptop computer, or personal computer. In the examples of this disclosure, UE 115 may be any audio source described in this disclosure, including VR headsets, XR headsets, AR headsets, vehicles, smartphones, microphones, microphone arrays, or any other device including a microphone, or capable of transmitting captured and / or synthesized audio streams. In some examples, the synthesized audio stream may be an audio stream stored in memory or previously created or synthesized. In some examples, UE 115 may also refer to a wireless local loop (WLL) station, Internet of Things (IoT) device, Internet of Everything (IoE) device, or MTC device, which can be implemented in various items such as appliances, vehicles, meters, etc.

[0283] Some UE 115s (such as MTC or IoT devices) may be low-cost or low-complexity devices and may provide automated communication between machines (e.g., via machine-to-machine (M2M) communication). M2M communication or MTC may refer to data communication technologies that allow devices to communicate with each other or with base station 105 without human intervention. In some examples, M2M communication or MTC may include communication from devices that exchange and / or use audio metadata indicating privacy restrictions and / or password-based privacy data to switch, block various audio streams and / or audio sources and / or empty various audio streams and / or audio sources, as will be described in more detail below.

[0284] In some cases, UE 115 may also be able to communicate directly with other UE 115 (e.g., using peer-to-peer (P2P) or device-to-device (D2D) protocols). One or more UEs in a group of UEs 115 utilizing D2D communication may be within the geographic coverage area 110 of base station 105. Other UEs 115 in such a group may be outside the geographic coverage area 110 of base station 105, or otherwise unable to receive transmissions from base station 105. In some cases, multiple groups of UEs 115 communicating via D2D communication may utilize a one-to-many (1:M) system, where each UE 115 transmits to every other UE 115 in the group. In some cases, base station 105 facilitates the scheduling of resources for D2D communication. In other cases, D2D communication is performed between UEs 115 without involving base station 105.

[0285] Base station 105 can communicate with core network 130 and with each other. For example, base station 105 can interface with core network 130 via backhaul link 132 (e.g., via S1, N2, N3 or other interfaces). Base station 105 can communicate with each other directly (e.g., directly between base stations 105) or indirectly (e.g., via core network 130) on backhaul link 134 (e.g., via X2, Xn or other interfaces).

[0286] In some cases, wireless communication system 100 may utilize both licensed and unlicensed radio frequency spectrum bands. For example, wireless communication system 100 may employ Licensed Assisted Access (LAA), LTE Unlicensed (LTE-U) radio access technology, or NR technology in unlicensed frequency bands (such as the 5 GHz ISM band). When operating in unlicensed radio spectrum bands, wireless devices (such as base station 105 and UE 115) may employ a Listen-Before-Talk (LBT) procedure to ensure the frequency channel is open before transmitting data. In some cases, operation in unlicensed frequency bands may be based on carrier aggregation configurations that combine component carriers operating in licensed frequency bands (e.g., LAA). Operation in unlicensed spectrum may include downlink transmission, uplink transmission, peer-to-peer transmission, or a combination of these. Duplexing in unlicensed spectrum may be based on Frequency Division Duplex (FDD), Time Division Duplex (TDD), or a combination of both.

[0287] Figure 21 This is a flowchart illustrating example operations of the playback system shown in Figure 1 when performing various aspects of an audio source rescaling technique. Playback system 16A may store metadata specified for two or more audio elements, identifying the corresponding area (2100) where each of the two or more audio elements resides within a virtual environment representing the source location. Playback system 16A may determine that a listener has moved between the two or more audio elements (2102), and in response to determining that a listener has moved between the two or more audio elements and based on the corresponding area, rescale the two or more audio elements to obtain at least one rescaled audio element (2104). Playback system 16A may then reproduce the at least one rescaled audio element to obtain an output audio signal (2106).

[0288] Figure 22 This is a flowchart illustrating example operations of the sound field representation generator shown in Figure 1 when performing various aspects of audio source rescaling techniques. The sound field representation generator 302 may first encode two or more audio elements to obtain an audio bitstream 21 (2200), and specify syntax elements in the audio bitstream 21 that indicate how rescaling should be performed when a listener moves between the two or more audio elements (2202). The sound field representation generator 302 may output the audio bitstream 21 (2204).

[0289] Figure 23This is a flowchart illustrating another example operation of the playback system shown in Figure 1 when performing various aspects of an audio source rescaling technique. Playback system 16A may first store audio elements residing within a virtual environment representing the source location (2300), and determine that the listener has moved between virtual rooms within the virtual environment and between physical rooms within the playback location (2302). Playback system 16A may then, in response to determining that the listener has moved between virtual rooms within the virtual environment and between physical rooms within the playback location, rescale the audio elements to obtain rescaled audio elements (2304), and reproduce the rescaled audio elements to obtain an output speaker feed 25 (and / or other audio signals) (2306).

[0290] In this way, various aspects of the technology can be implemented in the following embodiments.

[0291] Example 1A. A device configured to scale audio between a source location and a playback location, the device comprising: a memory configured to store metadata specified for two or more audio elements, the metadata identifying a corresponding area in which each of the two or more audio elements resides within a virtual environment representing the source location; and processing circuitry communicatively coupled to the memory and configured to: determine that a listener has moved between the two or more audio elements; in response to determining that the listener has moved between the two or more audio elements and based on the corresponding area, rescale the two or more audio elements to obtain at least one rescaled audio element; and reproduce the at least one rescaled audio element to obtain an output audio signal.

[0292] Example 2A. The device according to Example 1A, wherein the processing circuitry is further configured to determine that the corresponding region is a different region, and wherein the processing circuitry is configured to rescale the two or more audio elements to obtain the at least one rescaled audio element in response to determining that the listener has moved between the two or more audio elements and determining that the corresponding region is a different region.

[0293] Example 3A. The device according to Example 2A, wherein the processing circuitry is configured to rescale the two or more audio elements relative to the two or more audio elements in which the listener has moved, in response to determining that the listener has moved between the two or more audio elements and determining that the corresponding area is a different area.

[0294] Example 4A. The device according to Example 2A, wherein the processing circuitry is configured to rescale the two or more audio elements relative to the two or more audio elements from which the listener has moved away, in response to determining that the listener has moved between the two or more audio elements and determining that the corresponding area is a different area.

[0295] Example 5A. The device according to any one of Examples 2A to 4A, wherein the processing circuitry is configured to individually rescale the two or more audio elements in different ways to obtain the at least one rescaled audio element.

[0296] Example 6A. The device according to Example 1A, wherein the processing circuitry is configured to: determine that the corresponding regions overlap; and in response to determining that the listener has moved between the two or more audio elements and determining that the corresponding regions overlap, rescale the two or more audio elements to obtain corresponding two or more scaled audio elements.

[0297] Example 7A. The device according to Example 6A, wherein the processing circuitry is configured to rescale the two or more scaled audio elements by the same amount.

[0298] Example 8A. The device according to Example 6A, wherein the processing circuitry is configured to rescale the two or more audio elements by the same amount and avoid rescaling the two or more audio elements individually.

[0299] Example 9A. The device according to any one of Examples 1A to 8A, wherein the processing circuitry is further configured to obtain a syntax element from a bitstream specifying the two or more audio elements, the syntax element indicating how to perform rescaling when the listener moves between the two or more audio elements, and wherein the processing circuitry is configured to rescale the two or more audio elements to obtain the at least one rescaled audio element in response to determining that the listener has moved between the two or more audio elements and based on the two or more regions and the syntax element.

[0300] Example 10A. The device according to any one of Examples 1A to 9A, wherein the processing circuitry is further configured to obtain a rescaling factor that changes over time during the reproduction of the virtual scene, and wherein the processing circuitry is configured to dynamically rescale the two or more audio elements based on the two or more regions and the rescaling factor that changes over time to obtain the at least one rescaled audio element in response to determining that the listener has moved between the two or more audio elements.

[0301] Example 10.1A. The device according to any one of Examples 1A to 10A, wherein the two or more audio elements include at least one audio object.

[0302] Example 10.2A. The device according to any one of Examples 1A to 10A, wherein the two or more audio elements comprise one or more sets of high-order immersive coefficients.

[0303] Example 11A. A method for scaling audio between a source location and a playback location, the method comprising: storing metadata specified for two or more audio elements by a processing circuit, the metadata identifying a corresponding area in which each of the two or more audio elements resides within a virtual environment representing the source location; determining, by the processing circuit, that a listener has moved between the two or more audio elements; rescaling, by the processing circuit, the two or more audio elements based on the corresponding area, in response to determining that the listener has moved between the two or more audio elements to obtain at least one rescaled audio element; and reproducing, by the processing circuit, the at least one rescaled audio element to obtain an output audio signal.

[0304] Example 12A. The method according to Example 11A, the method further includes determining that the corresponding region is a different region, and wherein rescaling the two or more audio elements includes rescaling the two or more audio elements to obtain the at least one rescaled audio element in response to determining that the listener has moved between the two or more audio elements and determining that the corresponding region is a different region.

[0305] Example 13A. The method according to Example 12A, wherein rescaling the two or more audio elements includes rescaling the two or more audio elements relative to the two or more audio elements in which the listener has moved in response to determining that the listener has moved between the two or more audio elements and determining that the corresponding regions are different regions.

[0306] Example 14A. The method according to Example 12A, wherein rescaling the two or more audio elements includes rescaling the two or more audio elements relative to the two or more audio elements from which the listener has moved in response to determining that the listener has moved between the two or more audio elements and determining that the corresponding area is a different area.

[0307] Example 15A. The method according to any one of Examples 12A to 14A, wherein rescaling the two or more audio elements comprises individually rescaling the two or more audio elements in a manner different from each other to obtain the at least one rescaled audio element.

[0308] Example 16A. The method according to Example 11A, the method further includes determining the corresponding region overlap, wherein rescaling the two or more audio elements includes rescaling the two or more audio elements to obtain corresponding two or more scaled audio elements in response to determining that the listener has moved between the two or more audio elements and determining that the corresponding region overlaps.

[0309] Example 17A. The method according to Example 16A, wherein rescaling the two or more audio elements includes rescaling the two or more scaled audio elements by the same amount.

[0310] Example 18A. The method according to Example 16A, wherein rescaling the two or more audio elements includes rescaling the two or more audio elements by the same amount, and avoids rescaling the two or more audio elements individually.

[0311] Example 19A. The method according to any one of Examples 11A to 18A, the method further comprising obtaining a syntax element from a bitstream specifying the two or more audio elements, the syntax element indicating how to perform rescaling when the listener moves between the two or more audio elements, wherein rescaling the two or more audio elements includes rescaling the two or more audio elements in response to determining that the listener has moved between the two or more audio elements and based on the two or more regions and the syntax element to obtain the at least one rescaled audio element.

[0312] Example 20A. The method according to any one of Examples 11A to 19A, the method further comprising obtaining a rescaling factor that changes over time during the reproduction of the virtual scene, wherein rescaling the two or more audio elements comprises dynamically rescaling the two or more audio elements based on the two or more regions and the rescaling factor that changes over time to obtain the at least one rescaled audio element in response to determining that the listener has moved between the two or more audio elements.

[0313] Example 20.1A. The method according to any one of Examples 11A to 19A, wherein the two or more audio elements include at least one audio object.

[0314] Example 20.2A. The method according to any one of Examples 11A to 20A, wherein the two or more audio elements comprise one or more sets of high-order immersive coefficients.

[0315] Example 21A. A non-transitory computer-readable storage medium storing instructions that, when executed, cause processing circuitry to: store metadata specifying two or more audio elements, the metadata identifying a corresponding area in a virtual environment representing the source location where each of the two or more audio elements resides; determine that a listener has moved between the two or more audio elements; in response to determining that the listener has moved between the two or more audio elements and based on the corresponding area, rescale the two or more audio elements to obtain at least one rescaled audio element; and reproduce the at least one rescaled audio element to obtain an output audio signal.

[0316] Example 1B. An apparatus configured to encode an audio bitstream, the apparatus comprising: a memory configured to store two or more audio elements; and processing circuitry coupled to the memory and configured to: encode the two or more audio elements to obtain an audio bitstream; specify syntax elements in the audio bitstream indicating how to perform rescaling when a listener moves between the two or more audio elements; and output the audio bitstream.

[0317] Example 2B. The device according to Example 1B, wherein the syntax element indicates that rescaling is to be performed relative to the listener moving to one of the two or more audio elements therein.

[0318] Example 3B. The device according to Example 1B, wherein the syntax element indicates to perform rescaling relative to one of the two or more audio elements removed from the listener.

[0319] Example 4B. The device according to any one of Examples 1B to 3B, wherein the syntax element includes a first syntax element, and wherein the processing circuitry is further configured to: specify a second syntax element in the audio bitstream, the second syntax element indicating a rescaling factor for the audio element, the rescaling factor indicating how the source location of the audio element should be rescaled relative to other audio elements within the playback position; specify a third syntax element in the audio bitstream, the third syntax element indicating a tolerance for applying the rescaling factor; and output the audio bitstream.

[0320] Example 5B. The device according to Example 4B, wherein the third syntax element defines the percentage of the audio scene region that is retained outside the playback area per width dimension, per height dimension, and per depth dimension.

[0321] Example 6B. The device according to Example 4B, wherein the third syntax element defines the maximum and minimum percentage of the scene region that is retained outside the playback area per width dimension, per height dimension, and per depth dimension.

[0322] Example 7B. The device according to any one of Examples 4B to 6B, wherein the processing circuitry is further configured to specify a fourth syntax element in the audio bitstream, the fourth syntax element indicating center alignment, wherein the center alignment indicates that the center of the source dimension will be aligned with the center of the playback dimension.

[0323] Example 8B. The device according to any one of Examples 4B to 7B, wherein the processing circuitry is further configured to specify a fifth syntax element in the audio bitstream, the fifth syntax element indicating rotation, wherein the rotation indicates that the source dimension is to be rotated forward relative to the playback dimension.

[0324] Example 9B. The device according to any one of Examples 4B to 8B, wherein the processing circuitry is further configured to specify a sixth syntax element in the audio bitstream, the sixth syntax element indicating to perform automatic rescaling for the audio element.

[0325] Example 9.1B. The device according to any one of Examples 1B to 9B, wherein the two or more audio elements include at least one audio object.

[0326] Example 9.2B. The device according to any one of Examples 1B to 9B, wherein the two or more audio elements comprise one or more sets of high-order immersive coefficients.

[0327] Example 10B. A method for encoding an audio bitstream, the method comprising: encoding two or more audio elements to obtain an audio bitstream; specifying a syntax element in the audio bitstream, the syntax element indicating how to perform rescaling when a listener moves between the two or more audio elements; and outputting the audio bitstream.

[0328] Example 11B. The method according to Example 10B, wherein the syntax element indicates that a rescaling is to be performed relative to the listener moving to one of the two or more audio elements therein.

[0329] Example 12B. The method according to Example 10B, wherein the syntax element indicates that rescaling should be performed relative to one of the two or more audio elements removed from the listener.

[0330] Example 13B. The method according to any one of Examples 10B to 12B, wherein the syntax element includes a first syntax element, and wherein the method further includes: specifying a second syntax element in the audio bitstream, the second syntax element indicating a rescaling factor for the audio element, the rescaling factor indicating how the source location of the audio element should be rescaled relative to other audio elements within the playback position; specifying a third syntax element in the audio bitstream, the third syntax element indicating a tolerance for applying the rescaling factor; and outputting the audio bitstream.

[0331] Example 14B. The method according to Example 13B, wherein the third syntax element defines the percentage of the audio scene region that is retained outside the playback area per width dimension, per height dimension, and per depth dimension.

[0332] Example 15B. According to the method of Example 13B, wherein the third syntax element defines the maximum and minimum percentage of the scene region that is retained outside the playback area per width dimension, per height dimension, and per depth dimension.

[0333] Example 16B. The method according to any one of Examples 13B to 15B, the method further comprising specifying a fourth syntax element in the audio bitstream, the fourth syntax element indicating center alignment, wherein the center alignment indicates that the center of the source dimension will be aligned with the center of the playback dimension.

[0334] Example 17B. The method according to any one of Examples 13B to 16B, the method further comprising specifying a fifth syntax element in the audio bitstream, the fifth syntax element indicating rotation, wherein the rotation indicates that the source dimension is rotated forward relative to the playback dimension.

[0335] Example 18B. The method according to any one of Examples 13B to 18B, the method further comprising specifying a sixth syntax element in the audio bitstream, the sixth syntax element indicating to perform automatic rescaling for the audio element.

[0336] Example 18.1B. The method according to any one of Examples 10B to 18B, wherein the two or more audio elements include at least one audio object.

[0337] Example 18.2B. The method according to any one of Examples 10B to 18B, wherein the two or more audio elements comprise one or more sets of high-order immersive coefficients.

[0338] Example 19B. A non-transitory computer-readable storage medium storing instructions that, when executed, cause processing circuitry to: encode two or more audio elements to obtain an audio bitstream; specify syntax elements in the audio bitstream indicating how to perform rescaling when a listener moves between the two or more audio elements; and output the audio bitstream.

[0339] Example 1C. An apparatus configured to scale audio between a source location and a playback location, the apparatus comprising: a memory configured to store audio elements residing within a virtual environment representing the source location; and processing circuitry communicatively coupled to the memory and configured to: determine that a listener has moved between virtual rooms within the virtual environment and between physical rooms within the playback location; in response to determining that a listener has moved between virtual rooms within the virtual environment and between physical rooms within the playback location, rescale the audio elements to obtain rescaled audio elements; and reproduce the rescaled audio elements to obtain an output audio signal.

[0340] Example 2C. The device according to Example 1C, wherein the processing circuitry is further configured to: determine whether the rescaled audio element is smaller than the audio element; and in response to determining that the rescaled audio element is smaller than the audio element, reduce the speed associated with the audio element to reduce the Doppler effect.

[0341] Example 3C. The device according to Example 1C, wherein the processing circuitry is further configured to: determine whether the rescaled audio element is smaller than the audio element; and in response to determining that the rescaled audio element is smaller than the audio element, increase the speed associated with the audio element to maintain the Doppler effect.

[0342] Example 4C. The device according to Example 1C, wherein the processing circuitry is further configured to: determine whether the rescaled audio element is smaller than the audio element; and in response to determining that the rescaled audio element is not smaller than the audio element, increase the speed associated with the audio element to increase the Doppler effect.

[0343] Example 5C. The device according to any one of Examples 1C to 4C, wherein the processing circuitry is configured to: determine that the listener has moved between the physical rooms; and in response to determining that the listener has moved between the physical rooms, rescale the audio element to accommodate the new space in which the listener has moved to and replace the old space in the physical room from which the listener has moved out.

[0344] Example 6C. The device according to Example 1C, wherein the processing circuitry is configured to: determine that the listener has moved between the virtual rooms; and in response to determining that the listener has moved between the virtual rooms, rescale the audio elements to accommodate the new space in which the listener has moved to in the virtual room and replace the old space in the virtual room from which the listener has moved out.

[0345] Example 7C. The device according to Example 6C, wherein the new space of the virtual room includes a new space defined relative to the minimap of the virtual room.

[0346] Example 8C. The device according to any one of Examples 1C to 7C, wherein the processing circuitry is further configured to identify a change in the playback position, and wherein the processing circuitry is configured to, in response to determining that the listener has performed any one or both of the following: moving between the virtual rooms; and moving between the physical rooms, rescale the audio element based on the identified change in the playback position to obtain the rescaled audio element.

[0347] Example 9C. The device according to any one of Examples 1C to 8C, wherein the processing circuitry is further configured to: obtain a rescaling factor that changes over time during the reproduction of the virtual scene, and wherein the processing circuitry is configured to, in response to determining that the listener has performed any one or both of the following: moving between the virtual rooms; and moving between the physical rooms, dynamically rescaling the audio element based on the rescaling factor that changes over time to obtain the rescaled audio element.

[0348] Example 9.1C. The device according to any one of Examples 1B to 9B, wherein the audio element includes at least one audio object.

[0349] Example 9.2C. The device according to any one of Examples 1B to 9B, wherein the audio element comprises one or more sets of high-order immersive coefficients.

[0350] Example 10C. A method for scaling audio between a source location and a playback location, the method comprising: storing audio elements residing in a virtual environment representing the source location; determining that a listener has moved between virtual rooms within the virtual environment and between physical rooms within the playback location; in response to determining that the listener has moved between the virtual rooms within the virtual environment and between the physical rooms within the playback location, rescaling the audio elements to obtain rescaled audio elements; and reproducing the rescaled audio elements to obtain an output audio signal.

[0351] Example 11C. The method according to Example 10C, the method further comprising: determining whether the rescaled audio element is smaller than the audio element; and in response to determining that the rescaled audio element is smaller than the audio element, reducing the speed associated with the audio element to reduce the Doppler effect.

[0352] Example 12C. The method according to Example 10C, the method further comprising: determining whether the rescaled audio element is smaller than the audio element; and in response to determining that the rescaled audio element is smaller than the audio element, increasing the speed associated with the audio element to maintain the Doppler effect.

[0353] Example 13C. The method according to Example 10C, the method further comprising: determining whether the rescaled audio element is smaller than the audio element; and in response to determining that the rescaled audio element is not smaller than the audio element, increasing the speed associated with the audio element to increase the Doppler effect.

[0354] Example 14C. The method according to any one of Examples 10C to 13C, wherein rescaling the audio element includes, in response to determining that the listener has moved between the physical rooms, rescaling the audio element to accommodate the new space in which the listener has moved to and replacing the old space in the physical room from which the listener has moved out.

[0355] Example 15C. The method according to Example 10C, wherein rescaling the audio element includes, in response to determining that the listener has moved between the virtual rooms, rescaling the audio element to accommodate the new space in which the listener has moved to in the virtual room and replacing the old space in the virtual room from which the listener has moved out.

[0356] Example 16C. The method according to Example 15C, wherein the new space of the virtual room includes a new space defined relative to the minimap of the virtual room.

[0357] Example 17C. The method according to any one of Examples 10C to 16C, the method further comprising identifying a change in the playback position, wherein rescaling the audio element comprises, in response to determining that the listener has moved between any one or both of the following: moving between virtual rooms; and moving between physical rooms, and rescaling the audio element based on the identified change in the playback position to obtain the rescaled audio element.

[0358] Example 18C. The method according to any one of Examples 10C to 17C, the method further comprising obtaining a rescaling factor that changes over time during the reproduction of the virtual scene, wherein rescaling the audio element comprises, in response to determining that the listener has performed any one or both of the following: moving between the virtual rooms; and moving between the physical rooms, dynamically rescaling the audio element based on the rescaling factor that changes over time to obtain the rescaled audio element.

[0359] Example 18.1C. The method according to any one of Examples 10C to 18C, wherein the two or more audio elements include at least one audio object.

[0360] Example 18.2C. The method according to any one of Examples 10C to 18C, wherein the two or more audio elements comprise one or more sets of high-order immersive coefficients.

[0361] Example 19C. A non-transitory computer-readable storage medium storing instructions that, when executed, cause processing circuitry to: store an audio element residing in a virtual environment representing the source location; determine that a listener has moved between virtual rooms within the virtual environment and between physical rooms within the playback location; in response to determining that a listener has moved between virtual rooms within the virtual environment and between physical rooms within the playback location, rescale the audio element to obtain a rescaled audio element; and reproduce the rescaled audio element to obtain an output audio signal.

[0362] It should be recognized that, based on the examples, certain actions or events of any technique described herein may be performed in a different sequence, and may be added, combined, or omitted entirely (e.g., not all actions or events described are necessary for implementing the technique). Furthermore, in some examples, actions or events may be performed concurrently (e.g., through multithreading, interrupt handling, or multiple processors) rather than sequentially.

[0363] In some examples, the VR device (or streaming device) can use a network interface coupled to the VR / streaming device's memory to communicate and exchange messages with external devices, where the exchanged messages are associated with multiple available representations of the sound field. In some examples, the VR device can use an antenna coupled to the network interface to receive wireless signals that include data packets, audio packets, visual packets, or transport protocol data associated with multiple available representations of the sound field. In some examples, one or more microphone arrays can capture the sound field.

[0364] In some examples, multiple available representations of a sound field stored in a memory device may include multiple object-based representations of the sound field, a high-order immersive representation of the sound field, a mixed-order immersive representation of the sound field, a combination of an object-based representation of the sound field and a high-order immersive representation of the sound field, a combination of an object-based representation of the sound field and a mixed-order immersive representation of the sound field, or a combination of a mixed-order representation of the sound field and a high-order immersive representation of the sound field.

[0365] In some examples, one or more of the multiple available representations of the sound field may include at least one high-resolution region and at least one low-resolution region, and wherein the representation selected based on the steering angle provides higher spatial accuracy for at least one high-resolution region and lower spatial accuracy for the low-resolution region.

[0366] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium, including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0367] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies (such as infrared, radio, and microwave), then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead refer to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs utilize lasers to optically reproduce data. The combinations described above should also be included within the scope of computer-readable media.

[0368] Instructions can be executed by one or more processors, which include fixed-function processing circuitry and / or programmable processing circuitry, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Furthermore, these techniques can be fully implemented in one or more circuit or logic elements.

[0369] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs) or a set of ICs (e.g., chipsets), robots, and vehicles (such as automobiles, farm equipment, aircraft, etc.). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but implementation by different hardware units is not necessarily required. Rather, as described above, various units can be combined in a codec hardware unit, or various units can be provided by a collection of interoperable hardware units (including one or more processors as described above) combined with appropriate software and / or firmware.

[0370] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A device configured to scale audio between a source location and a playback location, the device comprising: A memory configured to store audio elements residing within a virtual environment representing the source location; and Processing circuitry, communicatively coupled to the memory and configured to: Determine that the listener has moved between virtual rooms within the virtual environment; and between physical rooms within the playback location. In response to determining that the listener has moved between the virtual rooms within the virtual environment; and between the physical rooms within the playback location, the audio element is rescaled to obtain a rescaled audio element; and Reproduce the rescaled audio elements to obtain the output audio signal.

2. The device according to claim 1, wherein the processing circuit is further configured to: Determine whether the rescaled audio element is smaller than the audio element; and In response to determining that the rescaled audio element is smaller than the audio element, the speed associated with the audio element is reduced to reduce the Doppler effect.

3. The device according to claim 1, wherein the processing circuit is further configured to: Determine whether the rescaled audio element is smaller than the audio element; and In response to determining that the rescaled audio element is smaller than the audio element, the speed associated with the audio element is increased to maintain the Doppler effect.

4. The device according to claim 1, wherein the processing circuit is further configured to: Determine whether the rescaled audio element is smaller than the audio element; and In response to determining that the rescaled audio element is not smaller than the audio element, the speed associated with the audio element is increased to increase the Doppler effect.

5. The device according to claim 1, wherein the processing circuit is configured to: Determine that the listener has moved between the physical rooms; and In response to determining that the listener has moved between the physical rooms, the audio element is rescaled to fit the new space in which the listener has moved and to replace the old space in which the listener has moved out.

6. The device according to claim 1, wherein the processing circuit is configured to: Determine that the listener has moved between the virtual rooms; and In response to determining that the listener has moved between the virtual rooms, the audio element is rescaled to fit the new space in which the listener has moved into the virtual room and replaces the old space in which the listener has moved out of the virtual room.

7. The device of claim 6, wherein the new space of the virtual room includes a new space defined relative to a minimap of the virtual room.

8. The device according to claim 1, The processing circuitry is further configured to identify changes in the playback position, and The processing circuitry is configured to, in response to determining that the listener has moved between the virtual rooms and between the physical rooms, rescale the audio element to obtain the rescaled audio element based on the identified change in the playback position.

9. The device according to claim 1, The processing circuitry is further configured to obtain a rescaling factor that changes over time during the reproduction of the virtual environment, and The processing circuitry is configured to dynamically rescale the audio element to obtain the rescaled audio element in response to determining that the listener has moved between the virtual rooms and between the physical rooms, based on the rescaling factor that changes over time.

10. The device of claim 1, wherein the audio element comprises at least one audio object.

11. The device of claim 1, wherein the audio element comprises one or more sets of high-order immersive sound coefficients.

12. A method for scaling audio between a source location and a playback location, the method comprising: Storing audio elements residing within a virtual environment representing the source location; Determine that the listener has moved between virtual rooms within the virtual environment; and between physical rooms within the playback location. In response to determining that the listener has moved between the virtual rooms within the virtual environment; and between the physical rooms within the playback location, the audio element is rescaled to obtain a rescaled audio element; and Reproduce the rescaled audio elements to obtain the output audio signal.

13. The method according to claim 12, further comprising: Determine whether the rescaled audio element is smaller than the audio element; as well as In response to determining that the rescaled audio element is smaller than the audio element, the speed associated with the audio element is reduced to reduce the Doppler effect.

14. The method according to claim 12, further comprising: Determine whether the rescaled audio element is smaller than the audio element; as well as In response to determining that the rescaled audio element is smaller than the audio element, the speed associated with the audio element is increased to maintain the Doppler effect.

15. The method according to claim 12, further comprising: Determine whether the rescaled audio element is smaller than the audio element; as well as In response to determining that the rescaled audio element is not smaller than the audio element, the speed associated with the audio element is increased to increase the Doppler effect.

16. The method of claim 12, wherein rescaling the audio element comprises, in response to determining that the listener has moved between the physical rooms, rescaling the audio element to accommodate the new space in which the listener has moved to and replacing the old space in the physical room from which the listener has moved out.

17. The method of claim 12, wherein rescaling the audio element comprises, in response to determining that the listener has moved between the virtual rooms, rescaling the audio element to accommodate the new space in which the listener has moved to in the virtual room and replacing the old space in the virtual room from which the listener has moved out.

18. The method of claim 17, wherein the new space of the virtual room comprises a new space defined relative to a minimap of the virtual room.

19. The method of claim 12, further comprising identifying a change in the playback position. The rescaling of the audio element includes, in response to determining that the listener has moved between the virtual rooms and between the physical rooms, and based on the identified change in the playback position, rescaling the audio element to obtain the rescaled audio element.

20. A non-transitory computer-readable storage medium having instructions stored thereon, the instructions causing a processing circuit, when executed, to: Stores audio elements residing within a virtual environment representing the source location; Determine that the listener has moved between virtual rooms within the virtual environment; and between physical rooms within the playback location; In response to determining that the listener has moved between the virtual rooms within the virtual environment; and between the physical rooms within the playback location, the audio element is rescaled to obtain a rescaled audio element; and Reproduce the rescaled audio elements to obtain the output audio signal.

Citation Information

Patent Citations

  • Mixed-order ambisonics (MOA) audio data for computer-mediated reality systems

    US10405126B2

  • Mixed-order ambisonics (MOA) audio data for computer-mediated reality systems

    US20190007781A1

  • Scaling audio sources in extended reality systems

    US20240129681A1