Devices, methods, apparatus, media, and products for sound field adjustment

Abstract

A device includes one or more processors configured to obtain sound information from an audio source. The one or more processors are also configured to select a compression mode, in which a representation of the sound information is compressed prior to transmission to a playback device, or a bypass mode, in which the representation of the sound information is not compressed prior to transmission to the playback device, based on a latency criterion associated with the playback device. The one or more processors are also configured to generate audio data that includes a compressed representation of the sound information or an uncompressed representation of the sound information based on the selected one of the compression mode or the bypass mode. The one or more processors are also configured to send the audio data as streaming data to the playback device via a wireless transmission.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to the common provisional patent application No. 63 / 045,297, filed June 29, 2020; provisional patent application No. 63 / 053,331, filed July 17, 2020; and non-provisional patent application No. 17 / 360,895, filed June 28, 2021, the contents of which are expressly incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure generally involves adjusting the sound field. Background Technology

[0004] Technological advancements have led to smaller and more powerful computing devices. For example, a variety of portable personal computing devices exist today, including cordless phones such as mobile and smartphones, tablets, and laptops, which are small, lightweight, and easy for users to carry. These devices can transmit voice and data packets over wireless networks. Furthermore, many of these devices incorporate additional functions such as digital cameras, digital camcorders, digital recorders, and audio file players. Moreover, such devices can process executable instructions, including software applications (such as web browser applications) that can access the internet. Therefore, these devices can include significant computing power.

[0005] One application of this device involves providing users with wireless immersive audio. As an example, a headset worn by the user can receive streaming audio data from a remote server for playback. To illustrate, the headset detects the rotation of the user's head and sends head tracking information to the remote server. The remote server updates the audio scene based on the head tracking information, generates binaural audio data based on the updated audio scene, and sends the binaural audio data to the headset for playback.

[0006] Performing audio scene updates and binaural audio conversion at a remote server allows users to experience immersive audio through headphones with relatively limited processing resources. However, such systems can result in unnaturally high motion-to-sound latency due to the latency associated with sending head tracking information to the remote server, updating audio data based on head rotation, and sending the updated binaural audio data to the headphones. In other words, the time delay between the user's head rotation and the corresponding modified spatial audio played at the user's ears can be unnaturally long, potentially degrading the user experience. Summary of the Invention

[0007] According to a specific implementation of the technology disclosed herein, a device includes one or more processors configured to acquire sound information from an audio source. The one or more processors are further configured to select a compression mode or a bypass mode based on a latency metric associated with a playback device. In the compression mode, a representation of the sound information is compressed before transmission to the playback device; in the bypass mode, a representation of the sound information is not compressed before transmission to the playback device. The one or more processors are further configured to generate audio data that includes either a compressed representation of the sound information or an uncompressed representation of the sound information based on the selected compression mode or bypass mode. The one or more processors are further configured to transmit the audio data as streaming data to the playback device via wireless transmission.

[0008] According to a specific implementation of the technology disclosed herein, a method includes: acquiring audio information from an audio source, and selecting a compression mode or a bypass mode based on a latency standard associated with a playback device. In the compression mode, the representation of the sound information is compressed before transmission to the playback device; in the bypass mode, the representation of the sound information is not compressed before transmission to the playback device. The method also includes generating audio data that includes either a compressed representation of the sound information or an uncompressed representation of the sound information based on the selected compression mode or bypass mode. The method further includes transmitting the audio data as streaming data to the playback device via wireless transmission.

[0009] According to a specific implementation of the technology disclosed herein, an apparatus includes components for acquiring sound information from an audio source. The apparatus also includes components for selecting a compression mode or a bypass mode based on a latency metric associated with a playback device. In compression mode, the representation of the sound information is compressed before transmission to the playback device; in bypass mode, the representation of the sound information is not compressed before transmission to the playback device. The apparatus also includes components for generating audio data that includes either a compressed representation of the sound information or an uncompressed representation of the sound information based on the selected compression mode or bypass mode. The apparatus also includes components for transmitting the audio data as streaming data to the playback device via wireless transmission.

[0010] According to a specific implementation of the technology disclosed herein, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause one or more processors to obtain sound information from an audio source. When executed, the instructions also cause the one or more processors to select a compression mode or a bypass mode based on a latency metric associated with a playback device. In compression mode, the representation of the sound information is compressed before transmission to the playback device; in bypass mode, the representation of the sound information is not compressed before transmission to the playback device. When executed, the instructions also cause the one or more processors to generate audio data that includes either a compressed representation or an uncompressed representation of the sound information based on the selected compression mode or bypass mode. When executed, the instructions also cause the one or more processors to transmit the audio data as streaming data to the playback device via wireless transmission.

[0011] According to another specific implementation of the technology disclosed herein, a device includes one or more processors configured to receive data associated with the attitude of a playback device via wireless transmission. The one or more processors are further configured to select a specific representation of a sound field from multiple representations of a sound field based on the data. Each corresponding representation of the sound field corresponds to a different sector in a set of sectors. A sector represents a range of values ​​associated with movement of the playback device. The one or more processors are further configured to generate audio data corresponding to the selected representation of the sound field. The one or more processors are further configured to transmit the audio data as streaming data to the playback device via wireless transmission.

[0012] According to another specific implementation of the technology disclosed herein, a method includes receiving data associated with the posture of a playback device from a playback device via wireless transmission at a device. The method also includes selecting a specific representation of a sound field from multiple representations of a sound field based on the data. Each corresponding representation of the sound field corresponds to a different sector in a set of sectors. A sector represents a range of values ​​associated with movement of the playback device. The method further includes generating audio data corresponding to the selected representation of the sound field at the device. The method also includes transmitting the audio data as streaming data from the device to the playback device via wireless transmission.

[0013] According to another specific implementation of the technology disclosed herein, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, cause one or more processors to receive data associated with the orientation of a playback device via wireless transmission. When executed by one or more processors, the instructions also cause one or more processors to select a specific representation of a sound field from multiple representations of a sound field based on the data. Each corresponding representation of the sound field corresponds to a different sector in a set of sectors. A sector represents a range of values ​​associated with movement of the playback device. When executed by one or more processors, the instructions also cause one or more processors to generate audio data corresponding to the selected representation of the sound field. When executed by one or more processors, the instructions also cause one or more processors to transmit the audio data as streaming data to the playback device via wireless transmission.

[0014] According to another specific implementation of the technology disclosed herein, an apparatus includes components for receiving data associated with the attitude of a playback device via wireless transmission. The apparatus also includes components for selecting a specific representation of a sound field from a plurality of representations of a sound field based on the data. Each corresponding representation of the sound field corresponds to a different sector in a set of sectors. A sector representation represents a range of values ​​associated with movement of the playback device. The apparatus further includes components for generating audio data corresponding to the selected representation of the sound field. The apparatus also includes components for transmitting the audio data as streaming data to the playback device via wireless transmission.

[0015] According to another specific implementation of the technology disclosed herein, the device includes one or more processors configured to receive encoded ambisonic audio data representing a sound field from a streaming device via wireless transmission. The one or more processors are also configured to perform decoding of the ambisonic audio data to generate decoded ambisonic audio data. Decoding of the ambisonic audio data includes base layer decoding of the base layer of the encoded ambisonic audio data, and selectively includes enhancement layer decoding in response to the amount of movement of the device. The one or more processors are also configured to adjust the decoded ambisonic audio data to alter the sound field based on data associated with at least one of translation or orientation related to the movement of the device. The one or more processors are also configured to output the adjusted decoded ambisonic audio data to two or more speakers for playback.

[0016] According to another specific implementation of the technology disclosed herein, a method includes receiving, at a device, encoded panoramic audio data representing a sound field from a streaming device via wireless transmission. The method also includes performing decoding of the panoramic audio data at the device to generate decoded panoramic audio data. Decoding of the panoramic audio data includes base layer decoding of the base layer of the encoded panoramic audio data, and selectively includes enhancement layer decoding in response to an amount of movement of the device. The method further includes adjusting the decoded panoramic audio data at the device based on data associated with at least one of translation or orientation related to movement of the device to alter the sound field. The method also includes outputting the adjusted decoded panoramic audio data from the device to two or more speakers for playback.

[0017] According to another specific implementation of the technology disclosed herein, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, cause one or more processors to receive encoded panoramic audio data representing a sound field from a streaming device via wireless transmission. When executed by one or more processors, the instructions also cause one or more processors to perform decoding of the panoramic audio data to generate decoded panoramic audio data. Decoding of the panoramic audio data includes base layer decoding of the base layer of the encoded panoramic audio data, and selectively includes enhancement layer decoding in response to a movement of the device. When executed by one or more processors, the instructions also cause one or more processors to adjust the decoded panoramic audio data to alter the sound field based on data associated with at least one of translation or orientation related to movement of the device. When executed by one or more processors, the instructions also cause one or more processors to output the adjusted decoded panoramic audio data to two or more speakers for playback.

[0018] An apparatus includes components for receiving encoded panoramic audio data representing a sound field from a streaming device via wireless transmission. The apparatus also includes components for performing decoding of the panoramic audio data to generate decoded panoramic audio data. Decoding of the panoramic audio data includes base layer decoding of the encoded panoramic audio data and selectively includes enhancement layer decoding in response to a movement of the device. The apparatus further includes components for adjusting the decoded panoramic audio data to alter the sound field based on data associated with at least one of translation or orientation related to movement of the device. The apparatus also includes components for outputting the adjusted decoded panoramic audio data to two or more speakers for playback.

[0019] According to a specific implementation of the technology disclosed herein, a device includes one or more processors configured to receive compressed audio data representing a sound field via wireless transmission and to decompress the compressed audio data. The one or more processors are further configured to adjust the decompressed audio data to alter the sound field based on data associated with at least one of translation or orientation related to movement of the device, render the adjusted decompressed audio data into the gains of two or more speakers to drive two or more speakers, and output the adjusted decompressed audio data to two or more speakers for playback.

[0020] According to another specific implementation of the technology disclosed herein, a method includes receiving compressed audio data representing a sound field via wireless transmission at one or more processors, and decompressing the compressed audio data. The method also includes adjusting the decompressed audio data to alter the sound field based on data associated with at least one of translation or orientation related to movement of the device, rendering the adjusted decompressed audio data into the gains of two or more speakers to drive two or more speakers, and outputting the adjusted decompressed audio data to two or more speakers for playback.

[0021] According to another specific implementation of the technology disclosed herein, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to receive compressed audio data representing a sound field via wireless transmission and to decompress the compressed audio data. When executed by the one or more processors, the instructions also cause the one or more processors to adjust the decompressed audio data to alter the sound field based on data associated with at least one of translation or orientation related to movement of the device, render the adjusted decompressed audio data into the gains of two or more speakers to drive two or more speakers, and output the adjusted decompressed audio data to two or more speakers for playback.

[0022] According to another specific implementation of the technology disclosed herein, an apparatus includes means for receiving compressed audio data representing a sound field via wireless transmission at one or more processors, and means for decompressing the compressed audio data. The apparatus also includes means for adjusting the decompressed audio data to alter the sound field based on data associated with at least one of translation or orientation related to movement of the device, means for rendering the adjusted decompressed audio data into the gains of two or more speakers to drive two or more speakers, and means for outputting the adjusted decompressed audio data to two or more speakers for playback.

[0023] According to a specific implementation of the technology disclosed herein, a device includes one or more processors configured to receive sound information from an audio source and panning data from a playback device. The panning data corresponds to a panning associated with the playback device. The one or more processors are also configured to convert the sound information into audio data representing a sound field based on the panning, and to transmit the audio data as streaming data to the playback device via wireless transmission.

[0024] According to another specific implementation of the technology disclosed herein, a method includes receiving sound information from an audio source and receiving translation data from a playback device. The translation data corresponds to a translation associated with the playback device. The method also includes converting the sound information into audio data representing a sound field based on the translation, and transmitting the audio data as streaming data to the playback device via wireless transmission.

[0025] According to another specific implementation of the technology disclosed herein, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to receive sound information from an audio source and to receive translation data from a playback device. The translation data corresponds to a translation associated with the playback device. When executed by the one or more processors, the instructions also cause the one or more processors to convert the sound information into audio data representing a sound field based on the translation, and to transmit the audio data as streaming data to the playback device via wireless transmission.

[0026] According to another specific implementation of the technology disclosed herein, an apparatus includes components for receiving sound information from an audio source and components for receiving translation data from a playback device. The translation data corresponds to a translation associated with the playback device. The apparatus also includes components for converting the sound information into audio data representing a sound field based on the translation, and components for transmitting the audio data as streaming data to the playback device via wireless transmission.

[0027] According to a specific implementation of the technology disclosed herein, a device includes one or more processors configured to acquire data associated with location and orientation related to movement of the device at multiple time instances, and to transmit the data to a remote device via wireless transmission. The one or more processors are also configured to receive compressed audio data representing a sound field from the remote device via wireless transmission, and to decompress the compressed audio data representing the sound field. The one or more processors are further configured to adjust the decompressed audio data to alter the sound field based on the orientation associated with the device, and to output the adjusted decompressed audio data to two or more speakers.

[0028] According to another specific implementation of the technology disclosed herein, a method includes acquiring data associated with location and orientation at multiple time instances related to tracking movement of a device, and transmitting the data to a remote device via wireless transmission. The method also includes receiving compressed audio data representing a sound field from the remote device via wireless transmission, and decompressing the compressed audio data representing the sound field. The method further includes adjusting the decompressed audio data to alter the sound field based on the orientation associated with the device, and outputting the adjusted decompressed audio data to two or more speakers.

[0029] According to another specific implementation of the technology disclosed herein, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to acquire data associated with position and orientation related to tracking movement of a device at multiple time instances, and to transmit the data to a remote device via wireless transmission. When executed by the one or more processors, the instructions also cause the one or more processors to receive compressed audio data representing a sound field from the remote device via wireless transmission and to decompress the compressed audio data representing the sound field. When executed by the one or more processors, the instructions also cause the one or more processors to adjust the decompressed audio data to alter the sound field based on the orientation associated with the device, and to output the adjusted decompressed audio data to two or more speakers.

[0030] According to another specific implementation of the technology disclosed herein, an apparatus includes components for acquiring data associated with position and orientation at multiple time instances and for tracking movement of the apparatus, and components for transmitting the data to a remote device via wireless transmission. The apparatus also includes components for receiving compressed audio data representing a sound field from the remote device via wireless transmission, and components for decompressing the compressed audio data representing the sound field. The apparatus further includes components for adjusting the decompressed audio data to alter the sound field based on the orientation associated with the apparatus, and components for outputting the adjusted decompressed audio data to two or more loudspeakers.

[0031] According to a specific implementation of the technology disclosed herein, a device includes one or more processors configured to receive sound information from an audio source and to receive data corresponding to positions associated with the playback device at multiple time instances from a playback device. The one or more processors are further configured to convert the sound information into audio data representing a sound field based on the data corresponding to the positions associated with the playback device, and to transmit the audio data as streaming data to one or both of the playback device or a second device coupled to the playback device via wireless transmission.

[0032] According to another specific implementation of the technology disclosed herein, a method includes receiving sound information from an audio source and receiving data from a playback device corresponding to positions associated with the playback device at multiple time instances. The method includes converting the sound information into audio data representing a sound field based on the data corresponding to the positions associated with the playback device, and transmitting the audio data as streaming data to one or both of the playback device or a second device coupled to the playback device via wireless transmission.

[0033] According to another specific implementation of the technology disclosed herein, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to receive sound information from an audio source and to receive data corresponding to positions associated with the playback device at multiple time instances. When executed by the one or more processors, the instructions also cause the one or more processors to convert the sound information into audio data representing a sound field based on the data corresponding to the positions associated with the playback device, and to transmit the audio data as streaming data to one or both of the playback device or a second device coupled to the playback device via wireless transmission.

[0034] According to another specific implementation of the technology disclosed herein, an apparatus includes components for receiving sound information from an audio source, and components for receiving data from a playback device corresponding to positions associated with the playback device at multiple time instances. The apparatus also includes components for converting the sound information into audio data representing a sound field based on the data corresponding to the positions associated with the playback device. The apparatus further includes components for transmitting the audio data as streaming data to one or both of the playback device or a second device coupled to the playback device via wireless transmission.

[0035] According to a specific implementation of the technology disclosed herein, a device includes one or more processors configured to receive compressed audio data representing a sound field from a streaming device. The one or more processors are also configured to receive data corresponding to positions associated with the playback device at multiple time instances from a playback device, and to generate a predicted position of the device based on the data corresponding to the positions associated with the playback device. The predicted position indicates a prediction of where the playback device will be when the audio data is played on the playback device. The one or more processors are further configured to decompress the compressed audio data, adjust the decompressed audio data based on the predicted position to translate the sound field, compress the adjusted audio data, and transmit the compressed adjusted audio data as streaming data to the playback device via wireless transmission.

[0036] According to another specific implementation of the technology disclosed herein, a method includes receiving compressed audio data representing a sound field from a streaming device. The method also includes receiving data from a playback device corresponding to positions associated with the playback device at multiple time instances, and generating a predicted position of the device based on the data corresponding to the positions associated with the playback device. The predicted position indicates a prediction of where the playback device will be when the audio data is played on the playback device. The method further includes decompressing the compressed audio data, adjusting the decompressed audio data based on the predicted position to translate the sound field, compressing the adjusted audio data, and transmitting the compressed, adjusted audio data as streaming data to the playback device via wireless transmission.

[0037] According to another specific implementation of the technology disclosed herein, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause one or more processors to receive compressed audio data representing a sound field from a streaming device. When executed by one or more processors, the instructions also cause one or more processors to receive data corresponding to positions associated with the playback device at multiple time instances from a playback device, and to generate a predicted position of the playback device based on the data corresponding to the positions associated with the playback device. The predicted position indicates a prediction of where the playback device will be when the audio data is played on the playback device. When executed by one or more processors, the instructions also cause one or more processors to decompress the compressed audio data, adjust the decompressed audio data based on the predicted position to translate the sound field, compress the adjusted audio data, and transmit the compressed adjusted audio data as streaming data to the playback device via wireless transmission.

[0038] According to another specific implementation of the technology disclosed herein, an apparatus includes means for receiving compressed audio data representing a sound field from a streaming device. The apparatus also includes means for receiving data from a playback device corresponding to positions associated with the playback device at multiple time instances, and means for generating a predicted position of the device based on the data corresponding to the positions associated with the playback device. The predicted position indicates a prediction of where the playback device will be when the audio data is played on the playback device. The apparatus also includes means for decompressing the compressed audio data, means for adjusting the decompressed audio data based on the predicted position to translate the sound field, means for compressing the adjusted audio data, and means for transmitting the compressed adjusted audio data as streaming data to the playback device via wireless transmission.

[0039] According to a specific implementation of the technology disclosed herein, a device includes one or more processors configured to receive compressed audio data corresponding to a first representation of a sound field from a streaming device via wireless transmission. The first representation corresponds to a first viewport field of view associated with a first orientation of the device. The one or more processors are further configured to decompress the compressed audio data and output the decompressed audio data to two or more speakers. The one or more processors are further configured to send data associated with a second orientation of the device to the streaming device and receive compressed, updated audio data from the streaming device. The compressed, updated audio data corresponds to a second representation of the sound field. The second representation corresponds to a second viewport field of view that partially overlaps with the first viewport field of view and is associated with the second orientation. The one or more processors are further configured to decompress the compressed, updated audio data and output the decompressed, updated audio data to two or more speakers.

[0040] According to another specific implementation of the technology disclosed herein, a method includes receiving compressed audio data corresponding to a first representation of a sound field from a streaming device at one or more processors of the device and via wireless transmission. The first representation corresponds to a first viewport field of view associated with a first orientation of the device. The method also includes decompressing the compressed audio data and outputting the decompressed audio data to two or more speakers. The method further includes sending data associated with a second orientation of the device to the streaming device and receiving compressed, updated audio data from the streaming device. The compressed, updated audio data corresponds to a second representation of the sound field. The second representation corresponds to a second viewport field of view that partially overlaps with the first viewport field of view and is associated with the second orientation. The method also includes decompressing the compressed, updated audio data and outputting the decompressed, updated audio data to two or more speakers.

[0041] According to another specific implementation of the technology disclosed herein, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause one or more processors to receive compressed audio data corresponding to a first representation of a sound field from a streaming device via wireless transmission. The first representation corresponds to a first viewport field of view associated with a first orientation of the device. When executed by one or more processors, the instructions also cause one or more processors to decompress the compressed audio data and output the decompressed audio data to two or more speakers. When executed by one or more processors, the instructions also cause one or more processors to send data associated with a second orientation of the device to the streaming device and receive compressed, updated audio data from the streaming device. The compressed, updated audio data corresponds to a second representation of the sound field. The second representation corresponds to a second viewport field of view that partially overlaps with the first viewport field of view and is associated with the second orientation. When executed by one or more processors, the instructions also cause one or more processors to decompress the compressed, updated audio data and output the decompressed, updated audio data to two or more speakers.

[0042] According to another specific implementation of the technology disclosed herein, an apparatus includes means for receiving compressed audio data corresponding to a first representation of a sound field from a streaming device via wireless transmission, the first representation corresponding to a first viewport field of view associated with a first orientation of the device. The apparatus also includes means for decompressing the compressed audio data and means for outputting the decompressed audio data to two or more speakers. The apparatus further includes means for transmitting data associated with a second orientation of the device to the streaming device and means for receiving compressed, updated audio data from the streaming device. The compressed, updated audio data corresponds to a second representation of the sound field. The second representation corresponds to a second viewport field of view that partially overlaps with the first viewport field of view and is associated with the second orientation. The apparatus also includes means for decompressing the compressed, updated audio data and means for outputting the decompressed, updated audio data to two or more speakers.

[0043] According to a specific implementation of the technology disclosed herein, the device includes one or more processors configured to receive, via wireless transmission, data associated with the attitude of the playback device, and to select a specific representation of the sound field from multiple representations of the sound field based on the data. Each corresponding representation of the sound field corresponds to a different viewport field in a set of multiple overlapping viewport fields. The one or more processors are also configured to generate compressed audio data corresponding to the selected representation of the sound field, and to transmit the compressed audio data as streaming data to the playback device via wireless transmission.

[0044] According to another specific implementation of the technology disclosed herein, a method includes receiving data associated with the attitude of a playback device at one or more processors of a streaming device and via wireless transmission from a playback device. The method also includes selecting a specific representation of a sound field from multiple representations of a sound field based on the data. Each corresponding representation of the sound field corresponds to a different viewport field in a set of multiple overlapping viewport fields. The method further includes generating compressed audio data corresponding to the selected representation of the sound field, and transmitting the compressed audio data as streaming data to the playback device via wireless transmission.

[0045] According to another specific implementation of the technology disclosed herein, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to receive data associated with the attitude of the playback device via wireless transmission from a playback device. When executed by the one or more processors, the instructions also cause the one or more processors to select a specific representation of a sound field from a plurality of representations of a sound field based on the data. Each corresponding representation of the sound field corresponds to a different viewport field in a plurality of overlapping viewport field-of-view sets. When executed by the one or more processors, the instructions also cause the one or more processors to generate compressed audio data corresponding to the selected representation of the sound field, and to transmit the compressed audio data as streaming data to the playback device via wireless transmission.

[0046] According to another specific implementation of the technology disclosed herein, an apparatus includes components for receiving data associated with the attitude of a playback device via wireless transmission. The apparatus also includes components for selecting a specific representation of a sound field from a plurality of representations of a sound field based on the data. Each corresponding representation of the sound field corresponds to a different viewport field in a plurality of overlapping viewport field sets. The apparatus further includes components for generating compressed audio data corresponding to the selected representation of the sound field, and components for transmitting the compressed audio data as streaming data to the playback device via wireless transmission.

[0047] According to a specific implementation of the technology disclosed herein, a device includes one or more processors configured to receive encoded panoramic audio data representing a sound field from a streaming device via wireless transmission. The one or more processors are further configured to perform decoding of the encoded panoramic audio data to generate decoded panoramic audio data. Decoding of the encoded panoramic audio data includes base layer decoding of a base layer of the encoded panoramic audio data, and selectively including enhancement layer decoding in response to detecting that the encoded panoramic audio data includes at least one encoded enhancement layer. The one or more processors are further configured to adjust the decoded panoramic audio data to alter the sound field based on data associated with at least one of translation or orientation related to movement of the device, and to output the adjusted decoded panoramic audio data to two or more speakers for playback.

[0048] According to another specific implementation of the technology disclosed herein, a method includes receiving encoded panoramic audio data representing a sound field from a streaming device at one or more processors of the device and via wireless transmission. The method also includes performing decoding of the encoded panoramic audio data to generate decoded panoramic audio data. Decoding of the encoded panoramic audio data includes performing base layer decoding of the base layer of the encoded panoramic audio data, and selectively performing enhancement layer decoding based on the detection that the encoded panoramic audio data includes at least one encoded enhancement layer. The method further includes adjusting the decoded panoramic audio data to alter the sound field based on data associated with at least one of translation or orientation related to movement of the device, and outputting the adjusted decoded panoramic audio data to two or more speakers for playback.

[0049] According to another specific implementation of the technology disclosed herein, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to receive, via wireless transmission, encoded panoramic audio data representing a sound field from a streaming device. When executed by the one or more processors, the instructions also cause the one or more processors to perform decoding of the encoded panoramic audio data to generate decoded panoramic audio data. Decoding of the encoded panoramic audio data includes base layer decoding of the base layer of the encoded panoramic audio data, and selectively includes enhancement layer decoding based on detection that the encoded panoramic audio data includes at least one encoded enhancement layer. When executed by the one or more processors, the instructions also cause the one or more processors to adjust the decoded panoramic audio data to change the sound field based on data associated with at least one of translation or orientation related to movement of the device, and to output the adjusted decoded panoramic audio data to two or more speakers for playback.

[0050] According to another specific implementation of the technology disclosed herein, an apparatus includes means for receiving encoded panoramic audio data representing a sound field from a streaming device via wireless transmission. The apparatus also includes means for performing base layer decoding of the base layer of the encoded panoramic audio data, and means for selectively performing enhancement layer decoding based on detection that the encoded panoramic audio data includes at least one encoded enhancement layer. The apparatus further includes means for adjusting the decoded panoramic audio data to alter the sound field based on data associated with at least one of translation or orientation related to movement of the device. The apparatus also includes means for outputting the adjusted decoded panoramic audio data to two or more speakers for playback.

[0051] According to another specific implementation of the technology disclosed herein, a device includes one or more processors configured to receive, via wireless transmission, first data associated with a first pose of the playback device, the first pose being associated with a first number of sound sources in a sound scene, to generate a first frame of encoded panoramic audio data corresponding to a base layer encoding of the sound scene, and to transmit the first frame to the playback device. The one or more processors are further configured to receive, via wireless transmission, second data associated with a second pose of the playback device, the second pose being associated with a second number of sound sources in the sound scene, the second number being greater than the first number, to generate a second frame of encoded panoramic audio data corresponding to an enhancement layer encoding of the sound scene, and to transmit the second frame to the playback device.

[0052] According to another specific implementation of the technology disclosed herein, a method includes receiving, via wireless transmission, first data associated with a first pose of the playback device, the first pose being associated with a first number of sound sources in a sound scene; generating a first frame of encoded panoramic audio data corresponding to a base layer encoding of the sound scene; and transmitting the first frame to the playback device. The method further includes receiving, via wireless transmission, second data associated with a second pose of the playback device, the second pose being associated with a second number of sound sources in the sound scene, the second number being greater than the first number; generating a second frame of encoded panoramic audio data corresponding to an enhancement layer encoding of the sound scene; and transmitting the second frame to the playback device.

[0053] According to another specific implementation of the technology disclosed herein, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to receive, via wireless transmission, first data associated with a first pose of the playback device, the first pose being associated with a first number of sound sources in a sound scene, to generate a first frame of encoded panoramic audio data corresponding to a base layer encoding of the sound scene, and to transmit the first frame to the playback device. The instructions, when executed by one or more processors, cause the one or more processors to receive, via wireless transmission, second data associated with a second pose of the playback device, the second pose being associated with a second number of sound sources in the sound scene, the second number being greater than the first number, to generate a second frame of encoded panoramic audio data corresponding to an enhancement layer encoding of the sound scene, and to transmit the second frame to the playback device.

[0054] According to another specific implementation of the technology disclosed herein, an apparatus includes components for receiving, via wireless transmission, first data associated with a first pose of the playback device, the first pose being associated with a first number of sound sources in a sound scene. The apparatus includes components for generating a first frame of encoded panoramic audio data corresponding to a base layer encoding of the sound scene, and components for transmitting the first frame to the playback device. The apparatus includes components for receiving, via wireless transmission, second data associated with a second pose of the playback device, the second pose being associated with a second number of sound sources in the sound scene, the second number being greater than the first number. The apparatus also includes components for generating a second frame of encoded panoramic audio data corresponding to an enhancement layer encoding of the sound scene, and components for transmitting the second frame to the playback device.

[0055] Other implementations, advantages, and features of this disclosure will become apparent after reviewing the entire application, which includes the following parts: the description of the drawings, the detailed description, and the claims. Attached Figure Description

[0056] Figure 1 This is a schematic diagram showing spherical harmonic basis functions of various orders and sub-orders.

[0057] Figure 2 This is a block diagram illustrating an example implementation of a system for adjusting the sound field.

[0058] Figure 3A This is a block diagram illustrating the components and operation of a system for adjusting the sound field.

[0059] Figure 3B This is a block diagram illustrating a second implementation of the components and operation of a system for adjusting the sound field.

[0060] Figure 4A This is a block diagram illustrating the components and operation of a system for adjusting the sound field.

[0061] Figure 4B This is a block diagram illustrating the components and operation of a system for adjusting the sound field.

[0062] Figure 5A This is a block diagram illustrating the components and operation of a system for adjusting the sound field.

[0063] Figure 5B This is a block diagram illustrating the components and operation of a system for adjusting the sound field.

[0064] Figure 6 This is a block diagram illustrating the components and operation of a system for adjusting the sound field.

[0065] Figure 7This is a block diagram illustrating the components and operation of a system for adjusting the sound field.

[0066] Figure 8A This is a block diagram illustrating the components and operation of a system for adjusting the sound field.

[0067] Figure 8B This is a block diagram illustrating the components and operation of a system for adjusting the sound field.

[0068] Figure 8C This is a block diagram illustrating the components and operation of the system used to adjust the sound field.

[0069] Figure 9 This is a block diagram illustrating the components and operation of a system for adjusting the sound field.

[0070] Figure 10A This is a block diagram illustrating the components and operation of a system for adjusting the sound field.

[0071] Figure 10B This is a block diagram illustrating the components and operation of a system for adjusting the sound field.

[0072] Figure 10C It shows that it can be generated by Figure 10A or Figure 10B A schematic diagram illustrating an example of audio data frames generated by the system.

[0073] Figure 11A This is a schematic diagram illustrating a first implementation of streaming audio data and a decoder layer for decoding streaming audio.

[0074] Figure 11B This is a schematic diagram illustrating a second implementation of streaming audio data and a decoder layer for decoding streaming audio.

[0075] Figure 12 This is a schematic diagram illustrating a third implementation of streaming audio data and a decoder layer used to decode the streaming audio.

[0076] Figure 13 This is a block diagram illustrating the fourth implementation of streaming audio data and the decoder layer used to decode the streaming audio.

[0077] Figure 14 This is a block diagram illustrating a first implementation of an integrated circuit for adjusting the sound field.

[0078] Figure 15 This is a block diagram illustrating a second implementation of an integrated circuit for adjusting the sound field.

[0079] Figure 16This is a block diagram illustrating an illustrative implementation of a system for adjusting the sound field and including an external loudspeaker.

[0080] Figure 17 This is a schematic diagram illustrating the implementation of a portable electronic device for adjusting the sound field.

[0081] Figure 18 This is a schematic diagram of the first implementation of a vehicle configured to adjust the sound field.

[0082] Figure 19 This is a schematic diagram of a second implementation of a vehicle configured to adjust the sound field.

[0083] Figure 20 A first example of a method for adjusting the sound field is shown.

[0084] Figure 21 A second example of a method for adjusting the sound field is shown.

[0085] Figure 22 A third example of a method for adjusting the sound field is shown.

[0086] Figure 23 A fourth example of a method for adjusting the sound field is shown.

[0087] Figure 24 A fifth example of a method for adjusting the sound field is shown.

[0088] Figure 25 A sixth example of a method for adjusting the sound field is shown.

[0089] Figure 26 A seventh example of a method for adjusting the sound field is shown.

[0090] Figure 27 An eighth example of a method for adjusting the sound field is shown.

[0091] Figure 28 A ninth example of a method for adjusting the sound field is shown.

[0092] Figure 29 A tenth example of a method for adjusting the sound field is shown.

[0093] Figure 30 Eleventh example of a method for adjusting the sound field is shown.

[0094] Figure 31 The twelfth example of a method for adjusting the sound field is shown.

[0095] Figure 32 It is operable to execute the reference. Figures 1-31 A block diagram illustrating a specific example of a computing device for the described technology. Detailed Implementation

[0096] Systems and methods for performing sound field adjustment are described, wherein at least a portion of the sound field adjustment is performed by a streaming client device operating on audio data from a streaming source device. In conventional systems, the latency associated with the headset device sending head tracking information to a remote server, updating audio data based on the head tracking information at the remote server, and sending updated binaural audio data to the headset device results in unnaturally high motion-to-sound latency and degrades the user experience. By performing at least a portion of the sound field adjustment at the streaming client device (e.g., a headset device), the latency associated with sending head tracking data to a remote source device (e.g., a remote server) is reduced, and the user experience is improved.

[0097] In some examples, the split audio rendering system reduces rendering latency by incorporating head rotation adjustments (e.g., three-degree-of-freedom (3DOF) rotation adjustments) of the audio scene at the wearable device (e.g., a streaming client device such as headphones) instead of at the streaming source device. In some examples, in addition to 3DOF rotation adjustments, limited translation adjustments (e.g., 3DOF+) are also performed at the wearable device. The streaming source device converts the audio content into a sound field, such as based on first-order atmos (FOA), higher-order atmos (HOA), or mixed-order atmos (MOA). The streaming source device compresses the resulting audio data (e.g., atmos coefficients) and wirelessly transmits the compressed audio data to the wearable device.

[0098] The wearable device decodes compressed audio data and also acquires head tracker information, such as measurements from one or more sensors on the wearable device. Based on this head tracker information, the wearable device compensates for head rotation via sound field rotation, and in some examples, it also processes a low-latency 3DOF+ effect with finite translation to adjust the sound field. The wearable device binauralizes the adjusted sound field using a head-related transfer function (HRTF) or binaural impulse response (BRIR), with or without a headphone compensation filter associated with the wearable device. The wearable device can combine one or more of the decoding, sound field adjustment, and binauralization operations to reduce the overall processing resources used for audio adjustment and playback at the wearable device.

[0099] By performing 3DOF or 3DOF+ rendering after transmission, wireless devices enable wireless immersive audio systems to operate with low rendering latency compared to traditional systems.

[0100] In some examples, wearable devices can send position or motion information to a streaming audio source. The streaming audio source generates audio data by rendering audio content onto a sound field and adjusts the audio data using the received position or motion information to account for changes in the user's position. The streaming audio source compresses the audio data adjusted for the user's position changes and sends it to the wearable device, which then performs 3DOF or 3DOF+, binauralization, and playback of the resulting audio based on the user's head rotation.

[0101] In other examples, instead of sending location or movement data to the streaming audio source, the wearable device compensates for head rotation by rotating the sound field and also modifies the sound field based on changes in the user's position (e.g., a 6DOF effect).

[0102] In some implementations, the streaming audio source selects between a compression mode and a bypass mode. In compression mode, the audio data (e.g., Atmos coefficients) is compressed, while in bypass mode, the audio data is not compressed for transmission to the playback device. The selection between compression and bypass modes can be based on latency criteria associated with the playback of the audio data. For example, as an illustrative, non-limiting example, the latency associated with performing audio data compression at the streaming audio source and decompression at the playback device may exceed the permissible latency associated with low-latency audio applications such as extended reality applications, telephone calls, teleconferencing, or videophones. In some cases, such as for low-latency audio applications, where the wireless link between the streaming audio source and the playback device does not have sufficient bandwidth to support the transmission of the full original Atmos coefficient set, the streaming audio source may truncate higher-resolution audio data (e.g., discard higher-order Atmos coefficients) to reduce the amount of audio data to be transmitted without performing compression.

[0103] Some examples combine pose prediction and interactive sound rendering. For instance, in some implementations, the wearable device sends timestamped location information to a streaming audio source, which predicts the user's future location and adjusts the sound field to account for the predicted location changes. Predicting future user movement reduces the effective latency perceived by the wearable device user in relation to translating the sound field.

[0104] In some implementations, the streaming audio source renders head-locked stereo audio streams, such as background music or narrator voices during video games, which do not rotate and translate with the sound field based on the user's movement.

[0105] In some implementations, sound effects, such as user interaction sounds, can be sent and pre-buffered within the wearable device. As a result, the latency associated with playback of sound effects in response to user actions is reduced compared to waiting for the sound effects to be sent from the streaming audio source. In some examples, memory usage on the wearable device is reduced by decoding and pre-buffering the initial audio frames of the sound effect, and by delaying the decoding of the remaining frames of the sound effect until the effect sound is triggered.

[0106] In some implementations, companion devices, such as smartwatches or smartphones, are used to reduce latency by moving one or more operations from a remote streaming audio source to a location closer to the wearable device. In one example, the companion device receives location information from the wearable device and compressed audio data from the streaming audio source, performs a low-latency sound field translation operation based on the location information, and wirelessly transmits the translated sound field back to the wearable device.

[0107] In some examples, wearable devices can use different representations of the sound field based on the wearable device's pose. For example, different representations can correspond to overlapping viewport fields of view and can be encoded such that each representation provides higher resolution for audio sources within the viewport field of view corresponding to that representation. The viewport field of view can correspond to both translation and rotation and can be used to provide relatively large translations to different locations in the sound field, such as jumps to distant sound sources in video games, or to provide enhanced audio to distant sound sources (e.g., audio augmented reality).

[0108] In other implementations, such as audio-only implementations, wearable devices can use different representations of the sound field based on the wearable device's posture, and these different representations can correspond to overlapping sectors of the sound field. Sectors can correspond to both translation and rotation, and can be used to provide relatively large translations to different locations within the sound field, such as providing enhanced audio from distant sound sources (e.g., audio augmented reality). In some examples, the different representations can include different audio formats, such as a panoramic representation or a pre-rendered stereo representation. The choice of a particular audio format can be based on the amount of movement of the playback device.

[0109] In some examples, scalable audio coding is used to provide a base layer with coarse-resolution audio (e.g., zero-order immersive (ZOA), FOA, or both) and one or more enhancement layers with finer-resolution audio (e.g., second-order immersive, third-order immersive, or higher). To illustrate, when a sound scene transitions from having a relatively small number of sound sources to a relatively large number of different types of sounds, such as when an orchestra begins to play, the coding for the corresponding sound field can transition from the base layer to one or more enhancement layers. In such examples, wearable devices include a base layer decoder and one or more enhancement layer decoders selectively applied based on the coding level.

[0110] In some examples, the amount of movement (e.g., orientation, translation, or both) of the playback device can be used to determine the amount of audio resolution used for playback, since large rates of movement can cause audio "blurring," where high spatial audio resolution becomes imperceptible due to the movement. Audio resolution can be reduced at the playback device by not decompressing and / or decoding the received high-order immersive audio data during large rates of movement. Alternatively or additionally, audio resolution reduction can be achieved by the streaming audio source discarding the high-order immersive audio data before transmission to the playback device.

[0111] Specific aspects of this disclosure are described below with reference to the accompanying drawings. In the specification, common features are indicated by common reference numerals. As used herein, various terms are used only for the purpose of describing a particular implementation and are not intended to limit the implementation. For example, the singular forms “a,” “an,” and “described” are intended to also include the plural forms, unless the context explicitly indicates otherwise. It can be further understood that the terms “comprising,” “including,” and “containing” are interchangeable with “containing,” “having,” or “having.” Additionally, it should be understood that the term “wherein” is interchangeable with “this document.” As used herein, “exemplary” may indicate an example, implementation, and / or aspect and should not be construed as limiting or indicating a preferred or preferred implementation. As used herein, ordinal terms used to modify elements such as structures, components, operations, etc. (e.g., “first,” “second,” “third,” etc.) do not themselves indicate any priority or order of that element relative to another element, but merely distinguish that element from another element having the same name (but for the use of ordinal terms). As used herein, the term “set” refers to one or more specific elements, and the term “multiple” refers to multiple (e.g., two or more) specific elements.

[0112] As used herein, “coupled” can include “communicationally coupled,” “electrically coupled,” or “physically coupled,” and may also (or alternatively) include any combination thereof. Two devices (or components) may be directly or indirectly coupled (e.g., communicationally coupled, electrically coupled, or physically coupled) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof). Electrically coupled two devices (or components) may be in the same or different devices and may be connected via electronics, one or more connectors, or inductive coupling, as illustrative and non-limiting examples. In some implementations, such as communicationally coupled devices (or components) in electrical communication, two devices may directly or indirectly transmit and receive signals (e.g., digital or analog signals) via one or more wires, buses, networks, etc. As used herein, “direct coupling” can include two devices coupled (e.g., communicationally coupled, electrically coupled, or physically coupled) without intermediate components.

[0113] In this disclosure, terms such as “determine,” “calculate,” “estimate,” “shift,” and “adjust” can be used to describe how one or more operations are performed. It should be noted that these terms should not be construed as restrictive, and similar operations can be performed using other techniques. Additionally, as described herein, “generate,” “calculate,” “estimate,” “use,” “select,” “access,” and “determine” are used interchangeably. For example, “generating,” “calculating,” “estimate,” or “determining” a parameter (or signal) can refer to actively generating, estimating, calculating, or determining a parameter (or signal), or it can refer to using, selecting, or accessing a parameter (or signal) that has already been generated (such as one generated by another component or device).

[0114] Generally, this describes techniques for decoding 3D sound data, such as Atmos audio data. Atmos audio data can include Atmos coefficients of different orders, such as first-order, second-order, and higher (which can be referred to as high-order Atmos (HOA) coefficients corresponding to spherical harmonic basis functions of order greater than one). Atmos audio data can also include mixed-order Atmos (MOA). Therefore, Atmos audio data can include at least one Atmos coefficient corresponding to a harmonic basis function.

[0115] The development of surround sound has provided entertainment with a multitude of audio output formats. Most examples of these consumer surround sound formats are based on "channels," as they implicitly specify the feed to the speakers in certain geometric coordinates. Consumer surround sound formats include the popular 5.1 format (which includes six channels: front left (FL), front right (FR), center or front center, rear left or left surround, rear right or right surround, and low-frequency effects (LFE)), the evolving 7.1 format, and various formats that include tall speakers (such as the 7.1.4 and 22.2 formats (e.g., used for Ultra HD TV standards)). Non-consumer formats can span any number of speakers (e.g., symmetrical and asymmetrical geometries), often referred to as "surround arrays." One example of such a sound array includes 32 speakers located at coordinates on the corners of a truncated icosahedron.

[0116] The input to an encoder (such as a Moving Picture Experts Group (MPEG) encoder) may optionally be one of three possible formats: (i) conventional channel-based audio (as described above), intended to be played through a speaker at a pre-specified location; (ii) object-based audio, which involves discrete pulse code modulation (PCM) data for a single audio object having associated metadata containing its location coordinates (and other information); or (iii) scene-based audio, which involves representing the sound field using coefficients of spherical harmonic basis functions (also known as "spherical harmonic coefficients" or SHC, "high-order immersive sound" or HOA, and "HOA coefficients"). This encoder is described in more detail in the document entitled "Call for Proposals for 3D Audio" published in Geneva, Switzerland in January 2013 by the International Organization for Standardization / International Electrotechnical Commission (ISO) / (IEC) JTC1 / SC29 / WG11 / N13411, which is available at: http: / / mpeg.chiariglione.org / sites / default / files / files / standards / parts / docs / w13411.zip.

[0117] Currently, there are various channel-based "surround sound" formats. For example, formats range from 5.1 home theater systems (which have been most successful in expanding the living room market beyond stereo) to the 22.2 system developed by NHK (Japan Broadcasting Corporation). Content creators (e.g., Hollywood studios) want to create a soundtrack for a film once, rather than spending effort remixing it for each speaker configuration. Recently, standards development organizations have been considering providing the encoding into a standardized bitstream, along with subsequent decoding that is adaptable and independent of the speaker geometry (and number) and acoustic conditions at the playback location (involving the renderer).

[0118] To provide this flexibility to content creators, a hierarchical set of elements can be used to represent the sound field. A hierarchical set of elements can refer to a set in which elements are ordered, such that a basic set of lower-order elements provides a complete representation of the modeled sound field. As the set is expanded to include higher-order elements, the representation becomes more detailed, thus increasing resolution.

[0119] An example of a hierarchical set of elements is the set of spherical harmonic coefficients (SHCs). The following expression demonstrates a description or representation of a sound field using SHCs:

[0120]

[0121] This expression shows that at time t, at any point in the sound field... pressure SHC can be used. Uniquely indicated. Here, c is the speed of sound (approximately 343 m / s). It is a reference point (or observation point). It is an nth-order spherical Bessel function, and These are spherical harmonic basis functions of order n and sub-order m. It can be recognized that the terms within the square brackets are the frequency domain representation of the signal (i.e.,...). This can be approximated by various time-frequency transforms, such as the Discrete Fourier Transform (DFT), Discrete Cosine Transform (DCT), or Wavelet Transform. Other examples of hierarchical sets include wavelet transform coefficient sets and other sets of coefficients for multi-resolution basis functions.

[0122] Figure 1 Figure 100 shows the spherical harmonic basis functions from order zero (n=0) to order four (n=4). It can be seen that for each order, there exists an extension of sub-order m, and these sub-orders m in... Figure 1 The examples are shown but not explicitly stated for ease of illustration. The number of spherical harmonic basis functions of a particular order can be determined as: #basis function = (n+1)^2. For example, the tenth order (n=10) will correspond to 121 spherical harmonic basis functions (e.g., (10+1)^2).

[0123] SHC It can be physically acquired (e.g., recorded) through various microphone array configurations, or alternatively, it can be derived from a channel-based or object-based description of the sound field. SHC stands for Scene-Based Audio, where an SHC can be input to an audio encoder to obtain an encoded SHC that can facilitate more efficient transmission or storage. For example, it can use a (4+1) approach. 2 A fourth-order representation of 25 (hence the fourth-order) coefficients.

[0124] As mentioned above, SHC can be derived from microphone recordings using microphone arrays. Various examples of how to derive SHC from microphone arrays are described in Poletti, M., “Three-Dimensional Surround Sound Systems Based on Spherical Harmonics”, J. Audio Eng. Soc., Vol. 53, No. 11, pp. 1004-1025, November 2005.

[0125] To illustrate how to derive SHC from an object-based description, consider the following equation: Coefficients corresponding to the sound field of a single audio object. It can be represented as:

[0126]

[0127] Where i is , It is an n-order spherical Hankel function (of the second kind), and It refers to the location of the object. Knowing the source energy of the object as a frequency function. (For example, using time-frequency analysis techniques, such as performing a Fast Fourier Transform on a PCM stream) it is possible to convert each PCM object and its corresponding location into an SHC. Furthermore, it can be shown (since the above is a linear and orthogonal decomposition) that each object... The coefficients are added together. In this way, multiple PCM objects can be generated by... The coefficients are represented (e.g., the sum of coefficient vectors for individual objects). Essentially, the coefficients contain information about the sound field (as a function of 3D coordinates), as shown above at the observation point. The transformation from individual objects to the overall sound field representation in the vicinity.

[0128] Reference Figure 2System 200 includes a first device 102 coupled to a second device 202 via a network 170. Network 170 may include one or more of a fifth-generation (5G) cellular digital network, Bluetooth® (a registered trademark of BLUETOOTH SIG, INC., Washington), an IEEE 802.11 network (e.g., WiFi), one or more other wireless networks, or any combination thereof. The first device 102 is configured to generate audio data representing a sound field and transmit that audio data to the second device 202 via network 170. The second device 202 is configured to perform one or more adjustments to the sound field based on movement of the second device 202 before playing the obtained audio, thereby reducing the latency associated with sending movement information from the second device 202 to the first device 102 to adjust the sound field at the first device 102.

[0129] The first device 102 includes a memory 110, one or more processors 120, and a transceiver 130. The memory 110 includes instructions 112 executable by the one or more processors 120. The memory 110 also includes one or more media files 114. The processors 120 can access the one or more media files 114 as a source of audio information, as further described below. In some examples, the one or more processors 120 are integrated into a portable electronic device, such as a smartphone, tablet, laptop, or other electronic device. In other examples, the one or more processors 120 are integrated into a server, such as an edge server.

[0130] Transceiver 130 is coupled to one or more processors 120 and configured to enable communication with second device 202 via network 170. Transceiver 130 includes transmitter 132 and receiver 134. Although first device 102 is shown as including transceiver 130, in other implementations, first device 102 does not include transceiver 130, but instead includes transmitter 132 and receiver 134 as separate components.

[0131] One or more processors 120 are configured to execute instructions 112 to perform operations associated with audio processing. For illustration, one or more processors 120 are configured to receive sound information 123 from an audio source 122. For example, the audio source 122 may correspond to a portion of one or more media files 114, a game engine, one or more other sound information sources, or a combination thereof.

[0132] One or more processors 120 are configured to adjust the sound field 126 associated with the sound information 123 via the operation of a sound field representation generator 124. The sound field representation generator 124 is configured to output audio data 127 to an encoder 128. In the example, the audio data 127 includes panoramic sound data and corresponds to at least one of 2D audio data representing a two-dimensional (2D) sound field or 3D audio data representing a three-dimensional (3D) sound field. In some implementations, the sound field representation generator 124 may acquire one or more representations of the sound field from a content creator or some other device or source outside the first device 102 (e.g., loaded from a webpage or stored in a file and then loaded), process them, and stream them to the second device 202.

[0133] Encoder 128 is configured to perform atmos encoding or transcoding (e.g., spatially encoding audio data 127 into atmos coefficients) to generate encoded audio data 129. In some implementations, encoder 128 is configured to compress the encoded audio data 129 (e.g., psychoacoustic compression encoding), and in other implementations, encoder 128 does not compress the encoded audio data 129, as described in further detail with reference to FIG3. For example, encoder 128 may be configured by one or more processors 120 to operate in a compression mode in which compression is performed to reduce the size of the encoded audio data 129, or in a bypass mode in which the encoded audio data 129 is not compressed (e.g., the original atmos coefficients), such as based on a latency metric associated with playback at the second device 202.

[0134] Encoded audio data 129 is output by one or more processors 120 to transceiver 130 for transmission to second device 202. For example, audio data 127 corresponding to sound field 126 may be transmitted as streaming data via one or more first audio packets 162. In some implementations, audio source 122 corresponds to a portion of a media file (e.g., a portion of one or more media files 114), and the streaming data is associated with a virtual reality experience streamed to second device 202 (e.g., a playback device) via at least one of a 5G cellular digital network or a Bluetooth® network.

[0135] In some implementations, one or more processors 120 are also configured to receive translation data from the playback device, such as data 166 received from the second device 202. The translation data corresponds to translations associated with the second device 202, such as movement of the second device 202 (e.g., movement of the wearer of the second device 202, implemented as an earphone device). As used herein, “movement” includes rotation (e.g., a change in orientation without a change in position, such as a change in rolling, tilting, or yaw), translation (e.g., non-rotational movement), or combinations thereof.

[0136] One or more processors 120 are configured to convert sound information 123 into audio data representing a sound field based on a translation associated with the second device 202. For illustration, a sound field representation generator 124 adjusts the sound field 126 to generate updated audio data 127 representing the translated sound field. For example, in some implementations, the sound field representation generator 124 performs a translation on the object before conversion to immersive sound, and in some implementations, the sound field representation generator 124 performs a translation operation to apply the translation to immersive sound representing the existing sound field. One or more processors 120 are configured to transmit the updated audio data as streaming data to the second device 202 via wireless transmission, such as via a second audio packet 164.

[0137] The first device 102 is configured to receive subsequent translation data, such as data 168 received after data 166, and can perform further adjustments to the sound field 126 to account for the translation of the second device 202. Therefore, the first device 102 can receive a translation information stream indicating a change in the position of the second device 202, and update the streaming audio data sent to the second device 202 to represent an adjusted version of the sound field 126 corresponding to the changed position of the second device 202. However, in some implementations, the first device 102 does not perform a rotation of the sound field 126 in response to a change in the orientation of the second device 202, but instead performs a rotation of the sound field at the second device 202.

[0138] The second device 202 includes one or more processors 220 coupled to memory 210, a transceiver 230, one or more sensors 244, a first speaker 240, and a second speaker 242. In this illustrative example, the second device 202 corresponds to a wearable device. For illustration, one or more processors 220, memory 210, transceiver 230, one or more sensors 244, and speakers 240, 242 may be integrated into an earphone device, wherein the first speaker 240 is configured to be placed near the user's first ear when the user wears the earphone device, and the second speaker 242 is configured to be placed near the user's second ear when the user wears the earphone device.

[0139] Memory 210 is configured to store instructions 212 executable by one or more processors 220. One or more sensors 244 are configured to generate sensor data 246 indicating the movement of the second device 202, the attitude of the second device 202, or a combination thereof. As used herein, the “attitude” of the second device 202 indicates the position and orientation of the second device 202. The one or more sensors 244 include one or more inertial sensors, such as accelerometers, gyroscopes, compasses, positioning sensors (e.g., Global Positioning System (GPS) receivers), magnetometers, inclinometers, optical sensors, and one or more other sensors for detecting the acceleration, position, velocity, angular azimuth, angular velocity, angular acceleration, or any combination thereof of the second device 202. In one example, the one or more sensors 244 include GPS, electronic maps, and electronic compasses that use inertial and magnetic sensor techniques to determine orientation, such as a 3-axis magnetometer for measuring the Earth’s magnetic field and a 3-axis accelerometer for providing a horizontal reference to the Earth’s magnetic field vector based on the direction of gravity. In some examples, one or more sensors 244 include one or more optical sensors (e.g., cameras) to track movement individually or in combination with one or more other sensors (e.g., inertial sensors).

[0140] Transceiver 230 includes a wireless receiver 234 and a wireless transmitter 232. The wireless receiver 234 is configured to receive encoded audio data 129 from the first device 102 via wireless transmission and output corresponding encoded audio data 229 to one or more processors 220. In some implementations, the encoded audio data 229 matches the encoded audio data 129, while in other implementations, the encoded audio data 229 may differ from the encoded audio data 129 due to the loss of one or more audio packets during transmission, one or more bit errors in the received audio packets, or one or more other reasons for data loss. Any such data loss can be corrected (e.g., via forward error correction coding or redundant information transmission) or compensated (e.g., by estimating the audio data of the lost packets via interpolation between received packets). Although the second device 202 is shown as including transceiver 230, in other implementations, the second device 202 may omit transceiver 230 and may include receiver 234 and transmitter 232 as separate components.

[0141] One or more processors 220 are configured to receive encoded audio data 229 representing sound field 126 via wireless transmission. In some implementations, one or more processors 220 are configured to receive the encoded audio data 229 as streaming data from a streaming device (e.g., via a first audio packet 162 from the first device 102).

[0142] One or more processors 220 are configured to decode the encoded audio data 229. For example, a decoder 228 is configured to process the encoded audio data 229 (e.g., decompress it if the encoded audio data 229 is compressed) to generate audio data 227 that corresponds to audio data 127 at the first device 102 and indicates a sound field 226 corresponding to a sound field 126 at the first device 102. In some implementations, the audio data 227 includes panoramic sound data and corresponds to at least one of two-dimensional (2D) audio data or three-dimensional (3D) audio data.

[0143] One or more processors 220 are configured to adjust audio data 227 to alter the sound field 226 based on data associated with at least one of translation or orientation related to movement of the second device 202 (such as data indicated by sensor data 246). For illustration, a sound field adjuster 224 is configured to adjust audio data 227 to alter the sound field 226 based on sensor data 246 indicating a change in orientation or translation of the second device 202. In one example, one or more processors 220 are configured to adjust audio data 227 to rotate the sound field 226 in response to sensor data 246 indicating a change in orientation. In another example, one or more processors 220 are configured to translate and rotate the sound field 226 in response to movement of the second device 202, and not send translation data associated with movement of the second device 202 to the streaming device (e.g., not send data 166, 168 to the first device 102). In one example, one or more processors 120 are configured to perform one of the translations or rotations of the sound field 126 based on translation data (e.g., data 166, 168) received from the second device 202, and processor 220 is configured to perform the other of the translations or rotations of the sound field 226 based on sensor data 246.

[0144] One or more processors 220 are configured to render the adjusted decompressed audio data 223 to two or more speaker gains to drive two or more speakers. For example, a first speaker gain 219 is generated to drive a first speaker 240, and a second speaker gain 221 is generated to drive a second speaker 242. For illustration, one or more processors 220 are configured to perform binauralization of the adjusted decompressed audio data 223, such as using one or more HRTFs or BRIRs to generate speaker gains 219, 221, and output the adjusted decompressed audio data as attitude-adjusted binaural audio signals 239, 241 to speakers 240, 242 for playback.

[0145] First device 102 and second device 202 can each perform operations corresponding to the split audio rendering operation when combined. First device 102 processes sound information 123 from audio source 122 and generates audio data 127 (such as 2D or 3D panoramic sound data) representing sound field 126. In some implementations, first device 102 also performs translation on sound field 126 before sending encoded audio data 129 to second device 202. In some implementations, second device 202 adjusts audio data 227 based on the orientation of second device 202 to change sound field 226 and renders the resulting adjusted audio data 223 for playback. In some implementations, second device 202 also performs translation on sound field 226. (Refer to Figure 3-) Figure 13 Examples of various operations that can be performed by the first device 102 and the second device 202 are described in more detail.

[0146] Therefore, the first device 102 can operate as a streaming source device, and the second device 202 can operate as a streaming client device. By performing the operation of the rotating sound field 226 at the second device 202, the latency associated with sending rotation tracking data from the second device 202 to the first device 102 is avoided, and the user experience is improved.

[0147] Although the second device 202 is described as a headphone device for illustrative purposes, in other implementations, the second device 202 is implemented as a different type of device. For example, in some implementations, one or more processors 220 are integrated into the vehicle, and data 166, 168 indicate the translation and orientation of the vehicle, such as reference... Figure 18 and Figure 19 Further described. In some implementations, one or more processors 220 are integrated into the speaker array device and are also configured to perform beamguiding operations to direct binaural signals to a location associated with the user, such as a reference. Figure 16 Further described. In some implementations, one or more processors 220 are integrated in at least one of a mobile phone, tablet computer device, wearable electronic device, camera device, or extended reality headset (such as a virtual reality headset, mixed reality headset, or augmented reality headset).

[0148] Figure 3A This is a block diagram illustrating the components and operation of a system for adjusting a sound field. System 300 includes a streaming device 302 coupled to a wearable device 304.

[0149] Streaming device 302 includes an audio source 310 configured to provide atmos data 312 representing first audio content, non-atmos audio data 314 representing second audio content, or a combination thereof. Streaming device 302 is configured to perform a rendering / conversion to atmos operation 316 to convert the streaming non-atmos audio data 314 to an atmos sound field (e.g., FOA, HOA, mixed-level atmos) to generate atmos data 318. As used herein, "atmos data" includes a set of one or more atmos coefficients representing a sound field.

[0150] Streaming device 302 is configured to perform immersive audio encoding or transcoding operation 320 to compress immersive coefficients of immersive data 312, immersive data 318, or a combination thereof to generate compression coefficients 322, which are then wirelessly transmitted to wearable device 304 via wireless transmission 350 (e.g., via Bluetooth®, 5G, or WiFi, as an illustrative non-limiting example). In this example, as an illustrative non-limiting example, a low-latency decoder, such as Audio Processing Technology-X (AptX), Advanced Audio Low Latency Decoding (AAC-LD), or Enhanced Voice Services (EVS), is used to perform immersive audio encoding or transcoding operation 320.

[0151] In some implementations, streaming device 302 corresponds to Figure 2 The first device 102, audio source 310 corresponds to audio source 122, rendering / conversion to immersive sound operation 316 is performed at sound field representation generator 124, immersive sound audio encoding or transcoding operation 320 is performed at encoder 128, and compression factor 322 corresponds to encoded audio data 129.

[0152] Wearable device 304 (e.g., headphone device) is configured to receive compression factor 322 and perform panoramic audio decoding operation 360 to generate panoramic data 362. Wearable device 304 is also configured to generate head tracker data 372 based on the detection of rotation 366 and translation 368 of wearable device 304 by one or more sensors 344. Figure 390 shows an example representation of wearable device 304 implemented as headphone device 370 to demonstrate an example of rotation 366 and translation 368.

[0153] The 3DOF / 3DOF+ rotation and binauralization operation 364 of the immersive sound field at wearable device 304 performs compensation for head rotation via sound field rotation based on head tracker data 372 measured on wearable device 304 (and optionally also processes low-latency 3DOF+ effects with limited translation). For example, the 3DOF+ effect may be limited to translations in the forward, backward, left, and right directions (relative to the forward-facing direction of headphone device 370). The 3DOF / 3DOF+ rotation and binauralization operation 364 of the immersive sound field at wearable device 304 also performs binauralization of the compensated immersive sound field using HRTF or BRIR with or without the headphone compensation filter associated with wearable device 304, to output posture-adjusted binaural audio via a first output signal 374 to the first speaker 340 and a second output signal 376 to the second speaker 342. In some implementations, the immersive audio decoding operation 360 and the immersive sound field 3DOF / 3DOF+ rotation and binauralization operation 364 can be combined into a single operation to reduce the computational resource usage at the wearable device 304.

[0154] In some implementations, wearable device 304 corresponds to Figure 2 The second device 202 performs panoramic audio decoding operation 360 at decoder 228, performs panoramic sound field 3DOF / 3DOF+ rotation and binauralization operation 364 at sound field adjuster 224 and renderer 222, one or more sensors 344 correspond to one or more sensors 244, and head tracker data 372 corresponds to sensor data 246.

[0155] Therefore, System 300 can achieve low-latency wireless immersive audio by rendering with 3DOF or 3DOF+ after transmission.

[0156] Figure 3B An implementation of system 300 is depicted, wherein streaming device 302 is configured to selectively perform bypass operation 326 to circumvent compression coding 324 of immersive sound data 312, immersive sound data 318, or both. For illustration, streaming device 302 is configured to perform encoding operation 380, which may include compression coding 324 (e.g., psychoacoustic compression coding for compressing immersive sound coefficients) or bypass operation 326. In a particular implementation, audio data 382 output from encoding operation 380 may include compressed immersive sound coefficients from compression coding 324 or uncompressed immersive sound coefficients from bypass operation 326. Audio data 382 is wirelessly transmitted to wearable device 304 (also referred to as playback device 304) via wireless transmission 350.

[0157] In a particular implementation, the streaming device 302 may include one or more processors, such as Figure 2 One or more processors 120 are configured to acquire sound information from audio source 310 and perform mode selection 328. Mode selection 328 includes selecting either compression mode 330 or bypass mode 329 based on a latency criterion 331 associated with playback device 304. In compression mode 330, the representation of sound information (e.g., a set of immersive sound coefficients) is compressed before being transmitted to playback device 304; in bypass mode 329, the representation of sound information is not compressed before being transmitted to playback device 304. Audio data 382 is obtained, comprising either a compressed representation of the sound information or an uncompressed representation of the sound information based on the selected compression mode 330 or bypass mode 329. For illustration, in response to the selection of compression mode 330, one or more switches or other data flow control devices may be configured such that the input immersive sound coefficients are processed by compression coding 324 to generate compressed immersive sound coefficients as audio data 382. If bypass mode 329 is selected, one or more switches or data flow control devices can be configured to process the input Atmos coefficients using bypass operation 326, which can output at least a portion of the input (uncompressed) Atmos coefficients as output audio data 382. Atmos audio decoding 360 at playback device 304 is configured to receive audio data 382, ​​determine whether the audio data 382 includes compressed or uncompressed Atmos coefficients, and selectively decompress compressed Atmos coefficients to generate Atmos data 362. In some implementations where Atmos coefficients have already been discarded at streaming device 302, such as via truncation operation 327 under bypass mode 329 as explained further below, Atmos data 362 may contain fewer Atmos coefficients than Atmos data 312 or 318.

[0158] In some implementations, latency criterion 331 is based on whether the playback latency associated with streaming data exceeds latency threshold 332. For example, some applications such as extended reality applications, telephone calls, teleconferencing, or videophones may have one or more low-latency criteria for audio playback to provide a positive user experience. The latency associated with compression encoding 324 at streaming device 302 and the latency associated with decompression of compressed audio data during immersive audio decoding 360 at playback device 304 may cause the latency associated with playback of audio data 382 at playback device 304 to exceed latency threshold 332. In response to the determination that the playback latency exceeds latency threshold 332, bypass mode 329 is selected such that the transmitted audio data 382 includes uncompressed immersive coefficients and reduces latency. In some cases, the streaming device 302 receives an indication 333 from the playback device 304 that the playback delay associated with the streaming data exceeds a delay threshold 332, and the streaming device 302 selects a bypass mode 329 based on the received indication 333.

[0159] In some implementations, the latency criterion 331 is based at least in part on the bandwidth of the wireless link from the streaming device 302 to the playback device 304. When the amount of audio data 382 to be transmitted exceeds the available bandwidth of the wireless transmission 350, transmission delays that interfere with the timely playback of the audio data 382 at the playback device 304 occur. In some implementations, the streaming device 302 determines whether the wireless link to the playback device 304 corresponds to a "higher bandwidth" wireless link or a "lower bandwidth" wireless link, and selects a bypass mode 329 based on whether the wireless link corresponds to a "higher bandwidth" wireless link.

[0160] For example, in response to a wireless transmission occurring on a fifth-generation (5G) cellular digital network or a WiFi-type network, streaming device 302 can determine that wireless transmission 350 uses a "higher bandwidth" wireless link that provides sufficient bandwidth to transmit uncompressed Atmos coefficients. As another example, in response to a wireless transmission occurring on a Bluetooth-type wireless network, streaming device 302 can determine that wireless transmission 350 uses a "lower bandwidth" wireless link that does not provide sufficient bandwidth to transmit uncompressed Atmos coefficients. Streaming device 302 can select a compression mode 330 based on the wireless link corresponding to a "lower bandwidth" wireless link.

[0161] In the example above, "higher bandwidth" and "lower bandwidth" are relative terms used to classify wireless links based on wireless network type. While 5G, WiFi, and Bluetooth® are given as illustrative and non-limiting examples, it should be understood that the classification of 5G, WiFi, and Bluetooth® as "lower bandwidth" or "higher bandwidth" may change over time as the capabilities of these technologies evolve. In addition to classifying wireless links based on network type, or alternatively, streaming device 302 may estimate the wireless link's ability to transmit uncompressed immersive audio coefficients based on measured link parameters (transmit power level, receive power level, etc.), and select either compression mode 330 or bypass mode 329 based on the estimated wireless link's ability to transmit audio data 382 in uncompressed format.

[0162] In some implementations, streaming device 302 receives a request from playback device 304 for compressed or uncompressed audio data, and selects either bypass mode 329 or compression mode 330 based on the request. For example, playback device 304 may request to receive uncompressed audio data to reduce latency associated with decompressing the audio data, to reduce power consumption associated with decompressing the audio data, for one or more other reasons, or any combination thereof. As another example, playback device 304 may request to receive compressed audio data in response to network conditions that cause latency or packet loss during transmission on wireless transmission 350, to reduce the amount of memory used to store audio data 382 locally on playback device 304, for one or more other reasons, or any combination thereof.

[0163] In some cases, such as when audio data 382 for low-latency audio applications is transmitted over a wireless link with insufficient bandwidth to support the transmission of the entire set of uncompressed Atmos coefficients, but latency standard 331 prevents the use of compressed Atmos coefficients, the streaming device 302 can perform a truncation operation 327 in bypass operation 326. Truncation operation 327 truncates higher-resolution audio data (e.g., discarding some Atmos coefficients of one or more higher Atmos orders corresponding to the full-order uncompressed representation of the audio data frame) to reduce the size of the audio data 382 without performing compression. For example, in bypass mode 329, the streaming device 302 can discard the high-resolution portion of the uncompressed Atmos coefficients based on the bandwidth of the wireless link from the streaming device 302 to the playback device 304. For illustration, the high-resolution portion of the uncompressed representation may correspond to a subset of the Atmos coefficients. As an illustrative, non-limiting example, streaming device 302 can select the truncated atmos coefficients by starting with the highest-order atmos coefficient in the full-order uncompressed representation and progressively selecting lower-order atmos coefficients to be truncated until the combined size of the remaining atmos coefficients is small enough to be transmitted to playback device 304. In other non-limiting examples, truncation operation 327 discards all atmos coefficients except for the zero-order coefficient, discards all atmos coefficients except for the zero-order and first-order coefficients, or discards all atmos coefficients except for the zero-order, first-order, and second-order coefficients.

[0164] In some implementations, order selection 334 is used during encoding operation 380 to determine which immersive audio orders (e.g., zeroth, first, second, etc.) to transmit and which to discard (e.g., first, second, third, etc.). For example, as described above, order selection 334 can be generated by streaming device 302 to determine which immersive audio coefficients to truncate for transmission over a lower bandwidth wireless link for audio data 382 used in low-latency applications. As another example, as further described with reference to FIG10, order selection 334 can be determined by streaming device 302, by playback device 304, or by both, to reduce or enhance the resolution of audio data 382 based on actual or predicted motion of wearable device 304. When operating in bypass mode 329, order selection 334 can be used to control truncation operation 327. Alternatively or additionally, when operating in compression mode 330, order selection 334 can be used in conjunction with compression encoding 324. For example, the Atmos coefficients associated with one or more unselected Atmos orders can be discarded before, during, or after compression encoding 324 is performed.

[0165] Figure 4A This is a block diagram illustrating the components and operation of a system for adjusting the sound field. System 400 includes a streaming device 402 coupled to a wearable device 404 (also referred to as playback device 404).

[0166] Streaming device 402 includes an audio source 310 and receives translation metadata 478, including location information, from wearable device 404 via wireless transmission 480 (e.g., via Bluetooth®, 5G, or WiFi, as an illustrative and non-limiting example). Streaming device 402 performs a rendering / conversion to immersive sound operation 416 on streaming audio content 414 to render the streaming audio content 414 to an immersive sound field (e.g., FOA, HOA, mixed-level immersive sound), and uses motion information from the received translation metadata 478 to account for changes in the user's position on wearable device 404.

[0167] The rendering / conversion to Atmos operation 416 generates Atmos data 418 including Atmos coefficients. The streaming device 402 is configured to perform Atmos audio encoding or transcoding operation 320 to compress the Atmos coefficients of the Atmos data 418 to generate compression coefficients 422, and wirelessly transmit the compression coefficients 422 to the wearable device 404 via wireless transmission 350.

[0168] In some implementations, streaming device 402 corresponds to Figure 2The first device 102, audio source 310 corresponds to audio source 122, rendering / conversion to immersive sound operation 416 is performed at sound field representation generator 124, immersive sound audio encoding or transcoding operation 320 is performed at encoder 128, and translation metadata 478 corresponds to data 166.

[0169] Wearable device 404 (e.g., headphone device) is configured to receive compression coefficient 422 and perform panoramic audio decoding operation 360 to generate panoramic data 462. Wearable device 404 is also configured to generate rotation head tracker data 472 based on rotation 366 of wearable device 404, and to generate translation metadata 478 based on translation 368 of wearable device 404. Figure 490 shows an example representation of wearable device 404 implemented as headphone device 370 to demonstrate examples of rotation 366 and translation 368.

[0170] The 3DOF / 3DOF+ rotation and binauralization operation 364 of the immersive sound field at wearable device 404 performs compensation for head rotation via sound field rotation based on head rotation tracker data 472 measured on wearable device 404 (and optionally also based on translation 368 to process a low-latency 3DOF+ effect with limited translation). For example, the 3DOF+ effect may be limited to translations in the forward, backward, left, and right directions (relative to the forward-facing direction of headphone device 370). The 3DOF / 3DOF+ rotation and binauralization operation 364 of the immersive sound field at wearable device 404 also performs binauralization of the compensated immersive sound field using HRTF or BRIR with or without the headphone compensation filter associated with wearable device 404, to output posture-adjusted binaural audio via a first output signal 374 to the first speaker 340 and a second output signal 376 to the second speaker 342. In some implementations, the immersive audio decoding operation 360 and the immersive sound field 3DOF / 3DOF+ rotation and binauralization operation 364 can be combined into a single operation to reduce the computational resource usage at wearable device 404.

[0171] In some implementations, wearable device 404 corresponds to Figure 2 The second device 202 performs panoramic audio decoding operation 360 at decoder 228, and performs panoramic sound field 3DOF / 3DOF+ rotation and binauralization operation 364 at sound field adjuster 224 and renderer 222. One or more sensors 344 correspond to one or more sensors 244, and rotation head tracker data 472 and translation metadata 478 together correspond to sensor data 246.

[0172] Therefore, wearable device 404 is configured to send translation data (e.g., translation metadata 478) to streaming device 402, which is associated with the movement of wearable device 404. In response to sending the translation data, wearable device 404 is also configured to receive compressed, updated audio data (e.g., compression factor 422) from streaming device 402 representing a sound field translated based on the translation data. Wearable device 404 is configured to decompress the compressed, updated audio data to generate updated audio data (e.g., immersive data 462) and adjust the updated audio data to rotate the sound field based on the orientation associated with wearable device 404 (e.g., rotation head tracker data 472). In some implementations, wearable device 404 is also configured to adjust the updated audio data to translate the sound field based on changes in the translation of wearable device 404 (e.g., via a 3DOF+ effect).

[0173] Therefore, system 400 uses translation processing before transmission and 3DOF or 3DOF+ rendering after transmission to achieve low-latency wireless immersive audio. For illustration, the first latency associated with sending translation metadata 478 to streaming device 402 and receiving compressed, updated audio data (e.g., compression factor 422) from streaming device 402 is greater than the second latency associated with adjusting the updated audio data to rotate the sound field based on the orientation associated with the movement of wearable device 404 at the 3DOF / 3DOF+ rotation and binauralization operation 364 of the immersive sound field. Therefore, adjusting the sound field at wearable device 404 in response to the user's head rotation is performed faster after head rotation is detected, compared to adjusting the sound field at streaming device 402 after translation 368 is detected.

[0174] Figure 4B This is a block diagram illustrating another implementation of system 400, in which panoramic sound data 418 is transmitted through... Figure 3B The audio data 482 output from the encoding operation 380 (e.g., bypass operation 326 or compression encoding 324) is processed by an encoding operation 380. The audio data 482 may include compressed Atmos coefficients from compression encoding 324 or uncompressed Atmos coefficients from bypass operation 326. Additionally, the audio data 482 may include a subset (e.g., less than all) of the Atmos coefficients of the Atmos data 418, such as due to operations such as order selection 334, truncation operation 327, or combinations thereof.

[0175] In some implementations, playback device 404 transmits motion data 476 to streaming device 402 via wireless transmission 480. For example, motion data 476 may include rotation head tracker data 472, translation metadata 478, or a combination thereof. In some implementations, motion data 476 is used in conjunction with encoding operation 380 to determine the actual or predicted amount of motion of playback device 404, which may be used to determine the order selection 334 at playback device 404. In some implementations, streaming device 402 processes motion data 476 to determine or predict the amount of motion of playback device 404. Alternatively or additionally, motion data 476 includes an indication of the amount of motion or a predicted amount of motion generated by playback device 404 and transmitted to streaming device 402. Alternatively or additionally, motion data 476 may include an indication of the order selection 334 from playback device 404.

[0176] When operating in bypass mode 329, order selection 334 can be used to control truncation operation 327. Alternatively or additionally, when operating in compression mode 330, order selection 334 can be used in conjunction with compression coding 324. For example, Atmos coefficients associated with one or more unselected Atmos orders can be discarded before, during, or after compression coding 324 is performed.

[0177] Figure 5A This is a block diagram illustrating another implementation of the components and operation of a system for adjusting a sound field. System 500 includes a streaming device 502 coupled to a wearable device 504 (also referred to as playback device 504). Streaming device 502 includes an audio source 310 and performs operations as described in the reference... Figure 3A One or more operations described in the streaming device 302 are used to provide a compression factor 322 to the wearable device 504 via wireless transmission 350.

[0178] Wearable device 504 (e.g., headphone device) is configured to receive compression factor 322 and perform panoramic audio decoding operation 360 to generate panoramic data 362. See reference... Figure 4A As described in the wearable device 404, the wearable device 504 is also configured to generate rotation head tracker data 472 based on rotation 366 of the wearable device 504, and to generate translation metadata 478 based on translation 368 of the wearable device 504. Figure 590 shows an example representation of the wearable device 504 implemented as a headphone device 370 to demonstrate an example of rotation 366 and translation 368.

[0179] The immersive sound field 6DOF scene displacement and binauralization operation 564 at wearable device 504 performs compensation for head rotation via sound field rotation based on head rotation tracker data 472 measured on wearable device 504, and also modifies the sound field based on changes in the user's position based on translation metadata 478 (e.g., 6DOF effect). In the illustrative example, the audio processing for sound field rotation is relatively simple and can be performed in the time domain with negligible latency (e.g., effectively instantaneously) compared to the update rate of one or more sensors 344 (e.g., 100 updates per second); however, in some implementations, the immersive sound rotation matrix is ​​updated for each new audio frame. The audio processing for 6DOF translation at wearable device 504 can include processing in different domains (such as the short-time Fourier transform (STFT) domain), which may result in increased processing latency compared to sound field rotation. However, the latency associated with translation processing at wearable device 504 can be comparable to or less than the latency associated with sending translation data to streaming device 502 and receiving updated audio data from streaming device 502 based on the translation.

[0180] The 6DOF scene shift and binauralization operation 564 at wearable device 504 also performs binauralization of the compensated immersive sound field using HRTF or BRIR, with or without utilizing an earphone compensation filter associated with wearable device 504, to output attitude-adjusted binaural audio via a first output signal 374 to the first speaker 340 and a second output signal 376 to the second speaker 342. In some implementations, the immersive audio decoding operation 360 and the 6DOF scene shift and binauralization operation 564 can be combined into a single operation to reduce computational resource usage at wearable device 504.

[0181] In some implementations, wearable device 504 corresponds to Figure 2 The second device 202 performs panoramic audio decoding operation 360 at decoder 228, performs panoramic sound field 6DOF scene displacement and binauralization operation 564 at sound field adjuster 224 and renderer 222, and rotate head tracker data 472 and translation metadata 478 together correspond to sensor data 246.

[0182] Therefore, System 500 is able to achieve low-latency wireless immersive audio by utilizing rotation and translation processing after transmission.

[0183] Figure 5B This is a block diagram illustrating another implementation of system 500, in which panoramic sound data 312 or 318 is transmitted through... Figure 3BThe audio data 382 output from the encoding operation 380 (e.g., bypass operation 326 or compression encoding 324) is processed by an encoding operation 380. The audio data 382 may include compressed Atmos coefficients from compression encoding 324 or uncompressed Atmos coefficients from bypass operation 326. Additionally, the audio data 382 may include a subset (e.g., less than all) of the Atmos coefficients from Atmos data 312 or 318, such as due to operations such as order selection 334, truncation operation 327, or combinations thereof.

[0184] For reference Figure 4B As described, in some implementations, playback device 504 sends motion data 476 to streaming device 502 via wireless transmission 480. In some implementations, motion data 476 is used in conjunction with encoding operation 380 to obtain order selection 334 and reduce the number of immersive coefficients (uncompressed or compressed) to be sent to playback device 504.

[0185] Figure 6 This is a block diagram illustrating another implementation of the components and operation of a system 600 for adjusting a sound field. System 600 includes a streaming device 602 and a wearable device 604.

[0186] The streaming device 602 includes a game audio engine 610 that can correspond to the audio source 122. The game audio engine 610 outputs audio data including a head-tracking audio section 614, a head-locking audio section 628, and user interaction (UI) sound effects (FX) 634 (also referred to as "user interaction sound data 634"). For illustration, the head-tracking audio section 614 is updated to react to how a person's head turns when hearing sounds from the audio scene, while the head-locking audio section 628 is not updated to react to how a person's head turns.

[0187] Streaming device 602 is configured to receive timestamped location information 656 from wearable device 604 via wireless transmission 653. Streaming device 602 is configured to render streaming audio content (e.g., head-tracking audio portion 614) to an immersive sound field (e.g., FOA, HOA, or mixed-level immersive sound) and use the received timestamped location information 656 to account for changes in the user's location on wearable device 604. The timestamping enables prediction of future user movement, shown as future location prediction 658, which allows system 600 to reduce latency in translation processing as perceived at wearable device 604.

[0188] The streaming device 602 is also configured to selectively reduce the atmos order of the output atmos data via a HOA order truncation operation 624, which is based on a request 654 for a specific atmos order received from the wearable device 604. For example, the request 654 for a specific atmos order may request the reception of FOA data (e.g., to reduce the processing load at the wearable device 604 or to accommodate reduced available network bandwidth of the wireless transmission 650), and the HOA order truncation operation 624 may remove second-order and higher-order atmos data generated by the rendering / conversion to HOA operation 616 (e.g., data corresponding to...). Figure 1 (Data n>1) is used to generate output panoramic sound data 626. In another example, such as reference... Figure 4B As described, the HOA order truncation operation 624 is based on the actual or predicted amount of motion of the wearable device 604.

[0189] The streaming device 602 is also configured to render a head-locked stereo headphone audio stream 632 based on the head-locked audio portion 628 via a render-to-stereo audio mixing operation 630. Further described below, the streaming device 602 is also configured to send user interaction audio data 634 to a wearable device 604, enabling the wearable device 604 to pre-buffer the user interaction audio data 634 to reduce latency in playing the user interaction audio data 634.

[0190] Streaming device 602 performs the encoding portion of audio decoding operation 640. In some implementations, audio decoding operation 640 includes compressing the immersive sound coefficients (e.g., output immersive sound data 626) and the head-locked stereo audio stream 632, such as using a low-latency decoder (e.g., based on AptX, AAC-LD, or EVS), and streaming device 602 wirelessly transmits the compressed audio data along with user interaction sound data 634 to wearable device 604 within a configured payload. In other implementations, audio decoding operation 640 does not include compressing the immersive sound coefficients, such as referring to... Figure 3B The bypass operation described in 326.

[0191] In some implementations, streaming device 602 corresponds to Figure 2 The first device 102, the game audio engine 610 corresponding to the audio source 122, the rendering / conversion to HOA operation 616 is performed at the sound field representation generator 124, the HOA order truncation operation 624 and the rendering to two-channel audio mixing operation 630 are performed by one or more processors 120, and the encoding part of the audio decoding operation 640 is performed at the encoder 128.

[0192] Wearable device 604 decodes immersive sound coefficients (e.g., output immersive sound data 626) and head-locked audio (e.g., head-locked stereo headphone audio stream 632) via the decoding portion of audio decoding operation 640. Wearable device 604 also decodes user interaction sound data 634 and buffers the decoded user interaction sound data 634 in memory as pre-buffered user interaction sound data 643.

[0193] Wearable device 604 generates posture-adjusted binaural audio data 636 based on head tracker data 648 measured via one or more sensors 344 on wearable device 604, which compensates for head rotation via sound field rotation (and optionally also processes low-latency 3DOF+ effects with limited translation). Wearable device 604 also generates metadata 652 for transmission to streaming device 602. Metadata 652 includes a request for a specific immersive sound order 654 and timestamped location information 656 (e.g., indicating the user's location (e.g., using (x, y, z) coordinates) and a timestamp associated with the user's location).

[0194] The posture-adjusted binaural audio 636 and the head-locked audio portion 632 are combined at a combiner 638 (e.g., a mixer) and output to a speaker 690 (e.g., speaker 340, 342) of the wearable device 604. Additionally, user interaction sound 635 can be triggered by a user interaction 646 detected at the wearable device 604, and user interaction sound 635 can be provided to the combiner 638 for playback at the speaker. For example, in response to the detection of user interaction 646 (e.g., by detecting translation data 368 indicating the user of the wearable device 604 is positioned within a virtual object in the game environment, oriented towards the virtual object, or a combination thereof), audio data 642 corresponding to a specific user interaction is retrieved from pre-buffered user interaction sound data 643 stored in memory on the wearable device 604. Audio data 642 is rendered at an audio effects renderer 644, which may also consider head tracker data 648 to generate user interaction sound 635. Therefore, the pre-buffered user interaction sounds are triggered and rendered with low latency at point 604 on the wearable device.

[0195] To reduce memory usage on wearable device 604, in some implementations, one or more initial audio frames of each user interaction sound represented in user interaction sound data 634 are decoded, and the decoded frames are pre-buffered at wearable device 604, with the remaining encoded frames stored in memory on wearable device 604. Once a sound effect is triggered, the previously decoded and pre-buffered initial one or more frames are played with low latency, while the remaining frames are decoded so that they are available for playback after the initial one or more frames have been played.

[0196] In some implementations, wearable device 604 corresponds to Figure 2 The second device 202 performs the decoding part of the audio decoding operation 640 at the decoder 228, performs the panoramic sound field 3DOF / 3DOF+ rotation and binaural operation 364 at the sound field adjuster 224 and the renderer 222, and the head tracker data 648 corresponds to the sensor data 246.

[0197] Although System 600 demonstrates processing head-locked audio, pre-buffered user interaction sounds, and HOA order truncation, in other implementations, the functionality associated with head-locked audio processing, user interaction sounds, HOA order reduction, or any combination thereof may be omitted.

[0198] Therefore, System 600 achieves low-latency wireless immersive audio by further merging pose prediction and interactive sound rendering.

[0199] Figure 7 This is a block diagram illustrating another implementation of the components and operation of a system 700 for adjusting a sound field. System 700 includes a wearable accessory 706 coupled to a streaming device 602 and a wearable device 604. The streaming device 602 and wearable device 604 are shown in reference to... Figure 6 Operate in a similar manner as described.

[0200] Wearable accessory 706 is associated with wearable device 604 and receives immersive sound coefficients (e.g., output immersive sound data 626) from streaming device 602. Based on location information received from wearable device 604 via wireless transmission 780 (e.g., user location and timestamp data 766), it performs a low-latency immersive sound field translation operation 768 and wirelessly transmits the translated sound field to wearable device 604. For example, the immersive sound field translation operation 768 outputs adjusted audio data 770, which is encoded via the encoding portion of audio decoding operation 740 and transmitted to wearable device 604 via wireless transmission 750. User interaction audio data 634 can be retained as encoded (as encoded audio data 734) at wearable accessory 706 and can be retransmitted to wearable device 604 via wireless transmission 750. The head-locked stereo headphone audio stream 632 can be kept encoded at the wearable accessory 706 (as encoded head-locked stereo headphone audio stream 732) and can be retransmitted to the wearable device 604 via wireless transmission 750.

[0201] The immersive sound field translation operation 768 can adjust the sound field based on the actual detected user position or based on a prediction of the future position of the wearable device 604 based on the time when the currently adjusted audio will play at the wearable device 604. For example, the future position prediction can be performed by estimating the direction and speed of movement of the wearable device 604, indicated by changes between two or more of its nearest positions, and extrapolating the position the wearable device 604 will be at a specific future time based on that direction and speed. The future time can be based at least in part on the transmission delay associated with the transmission path to the wearable device 604, such that the longer it takes for the audio data to reach the wearable device 604, the further into the future the position prediction is made. In some implementations, the future position prediction is performed at the streaming device 602, the wearable companion device 706, or both. In some implementations, the latency associated with generating and transmitting the adjusted audio data 770 at the wearable accessory 706 to the wearable device 604 is small enough to enable the sound field to be shifted using the actual user location (i.e., not a future prediction) without causing a user-perceptible delay between the user's movement and the corresponding shift of the sound field during playback.

[0202] Wearable accessory 706 includes one or more processors 760 coupled to one or more transceivers 764 and a memory 762. The memory 762 stores instructions executable by the one or more processors 760. The one or more processors 760 are configured to receive compressed audio data representing a sound field (e.g., a compressed version of output immersive sound data 626 generated during the encoding portion of audio decoding operation 640) from streaming device 602. The one or more processors 760 are configured to receive data (e.g., user location and timestamp data 766) corresponding to the location associated with the playback device at multiple time instances from a playback device (e.g., wearable device 604). In some implementations, the one or more processors 760 are configured to generate a predicted location of the device based on the data corresponding to the location associated with the playback device. The predicted location indicates a prediction of where the playback device will be when the audio data is played on the playback device.

[0203] One or more processors 760 are configured to decompress compressed audio data, such as via a decoding section of audio decoding operation 640, and to adjust the decompressed audio data to translate the sound field based on predicted position, such as via a panoramic sound field translation operation 768. One or more processors 760 are configured to compress the adjusted audio data 770 (e.g., in the encoding section of audio decoding operation 740) and transmit the compressed adjusted audio data as streaming data to a playback device via wireless transmission.

[0204] Wearable companion device 706 enables computationally expensive operations (such as sound field translation) to be offloaded to a device (such as a smartphone, smartwatch, or one or more other electronic devices) that is closer to wearable device 604 than streaming device 602 and can have increased computing and power resources compared to wearable device 604. Sound field translation performed at wearable companion device 706 (based on actual or predicted user location) can be more accurate than sound field translation performed by streaming device 602 (based on actual or predicted user location) due to the reduced distance to and from wearable device 604, thus reducing transmission latency. As a result, the user experience can be improved.

[0205] Figure 6 and Figure 7The system shown thus illustrates several operations that can be performed by streaming device 602, wearable device 604, and optionally wearable companion device 706. In some implementations, streaming device 602 includes one or more processors (e.g., one or more processors 120) configured to receive sound information from an audio source (e.g., a game audio engine 610). The one or more processors of streaming device 602 are also configured to receive data (e.g., timestamped user location data 656) corresponding to the location associated with the playback device at multiple time instances from a playback device (e.g., wearable device 604).

[0206] One or more processors of streaming device 602 are also configured to convert sound information into audio data representing a sound field based on data corresponding to a location associated with a playback device (e.g., via rendering / conversion to HOA operation 616). One or more processors of streaming device 602 are also configured to transmit the audio data as streaming data to one or both of a playback device (e.g., wearable device 604) or a second device coupled to the playback device (e.g., wearable companion device 706) via wireless transmission.

[0207] In some implementations, one or more processors of streaming device 602 are configured to generate a predicted location of the playback device (e.g., a predicted future location 658) based on data corresponding to the location associated with the playback device. The predicted location indicates a prediction of where the playback device will be when audio data is played on the playback device (e.g., wearable device 604). One or more processors of streaming device 602 are configured to convert sound information into audio data representing a sound field (e.g., head-tracking audio section 614) based on the predicted location.

[0208] In some implementations, one or more processors of streaming device 602 are configured to send sound effect data (e.g., user interaction sound data 634) from an audio source to one or both of a playback device (e.g., wearable device 604) or a second device (e.g., wearable companion device 706), the sound effect data being buffered and accessible by the playback device for future playback, and at least a portion of the sound effect data being sent independently of any scheduled playback of that portion of the sound effect data.

[0209] In some implementations, one or more processors of the streaming device 602 are configured to receive head-locked audio portions (e.g., head-locked audio portion 628) from an audio source and generate head-locked audio data corresponding to posture-independent binaural audio based on the head-locked audio portions (e.g., head-locked binaural headphone audio stream 632).

[0210] In some implementations, one or more processors of the streaming device 602 are configured to wirelessly transmit head-locked audio data to one or both of a playback device or a second device for playback at the playback device.

[0211] In some implementations, the audio data corresponds to Atmos data, and one or more processors of the streaming device 602 are also configured to receive an indication of the Atmos order from the playback device (e.g., a request for a specific Atmos order 654) and adjust the audio data to have that Atmos order (e.g., via HOA order truncation operation 624).

[0212] In some implementations, one or more processors of the streaming device 602 are configured to, after receiving data corresponding to a position associated with the playback device, receive additional data corresponding to the position associated with the playback device; generate updated audio data based on the additional data; and send the updated audio data to the playback device.

[0213] In some implementations, Figure 6 and Figure 7 The wearable device 604 includes one or more processors (e.g., one or more processors 220) configured to receive sound information from an audio source (e.g., a game audio engine 610).

[0214] One or more processors of wearable device 604 are configured to acquire data associated with location and orientation at multiple time instances related to tracking the movement of wearable device 604, such as head tracker data 648, metadata 652, timestamped user location data 656, user location and timestamped data 766, or any combination thereof. One or more processors of wearable device 604 are also configured to transmit this data wirelessly to a remote device, such as streaming device 602 or wearable companion device 706.

[0215] One or more processors of the wearable device 604 are also configured to receive compressed (or uncompressed) audio data representing a sound field from a remote device via wireless transmission, decompress the compressed audio data representing the sound field, adjust the decompressed audio data based on the orientation associated with the wearable device 604 (e.g., outputting immersive sound data 626) to change the sound field, and output the adjusted decompressed audio data (e.g., posture-adjusted binaural audio data 636) to two or more speakers (e.g., via a combiner 638).

[0216] In some implementations, wearable device 604 includes a memory configured to store decompressed audio data, such as Figure 2The memory 210. One or more processors of the wearable device 604 may be configured to adjust decompressed audio data based on data associated with the position and orientation of the wearable device 604, such as via 3DOF / 3DOF+ rotation and binaural operation 364 of the immersive sound field. The decompressed audio data may include immersive sound data, which corresponds to at least one of two-dimensional (2D) audio data representing a 2D sound field or three-dimensional (3D) audio data representing a 3D sound field.

[0217] In some implementations, one or more processors of the wearable device 604 are configured to further adjust the decompressed audio data to translate the sound field based on the difference between the position of the wearable device 604 and the position associated with the sound field. This adjustment of the decompressed audio data based on the difference is limited to a translation of the sound field forward, backward, left, or right. For example, during the execution of immersive sound field 3DOF / 3DOF+ rotation and binauralization operation 364, the adjustment of the decompressed audio data based on the difference can be performed as a 3DOF+ effect.

[0218] In some implementations, one or more processors of the wearable device 604 are also configured to receive head-locked audio data (such as head-locked binaural headphone audio stream 632) via wireless transmission and combine the head-locked audio data with adjusted decompressed audio data (such as at combiner 638) for output to two or more speakers. In one example, the adjusted decompressed audio data (e.g., posture-adjusted binaural audio data 636) corresponds to posture-adjusted binaural audio, and the head-locked audio data (e.g., head-locked binaural headphone audio stream 632) corresponds to posture-independent binaural audio.

[0219] In some implementations, wearable device 604 includes one or more processor-accessible buffers, such as memory 210, a dedicated portion of memory 210, one or more other storage devices or buffers, or combinations thereof. One or more processors of wearable device 604 may also be configured to receive sound effects data in advance via wireless transmission and pre-buffer the sound effects data, such as pre-buffered user interaction sound data 643, in a buffer.

[0220] In some implementations, one or more processors of the wearable device 604 are also configured to, in response to an instruction received of an interaction between a user and a virtual object associated with sound effects data, retrieve a portion of pre-buffered sound effects data corresponding to the virtual object from a buffer, and combine that portion of the pre-buffered sound effects data (e.g., rendered as user interaction sound 635) with adjusted decompressed audio data (e.g., posture-adjusted binaural audio data 636) for output to two or more speakers.

[0221] In some implementations, one or more processors of the wearable device 604 are configured to send an indication of the immersive sound level (such as a request for a specific immersive sound level 654) to a remote device and, in response to the sending indication, receive updated audio data having that immersive sound level via wireless transmission.

[0222] Figure 8A This is a block diagram illustrating another implementation of the components and operation of a system 800 for adjusting a sound field. System 800 includes a source device 802 coupled to a device 804 (also referred to as playback device 804). Source device 802 is configured to provide device 804 with an audio stream 816 based on a panoramic sound representation selected from multiple panoramic sound representations of a sound scene. As an illustrative and non-limiting example, source device 802 may correspond to a portable electronic device (e.g., a telephone), a vehicle (e.g., a car), or a server (e.g., a cloud server).

[0223] Source device 802 includes one or more processors 832 and memory 830. Memory 830 is coupled to one or more processors 832 and configured to store multiple representations of a sound field. As shown, memory 830 includes multiple panoramic representations 822-828 of the sound field, corresponding to different viewport fields of view of device 804. In some implementations, the Nth panoramic representation VN828 (where N is a positive integer) corresponds to a panoramic representation of the sound field that is not specific to any particular viewport field of view. In some implementations, source device 802 corresponds to first device 102, one or more processors 832 corresponds to one or more processors 120, and memory 830 corresponds to memory 110. In some implementations, source device 802 corresponds to... Figures 3A-7 One or more streaming devices.

[0224] One or more processors 832 are configured to provide a manifest 818 to device 804, indicating atmos representations 822-828 available at memory 830. The one or more processors 832 are also configured to receive an audio stream request 820 from device 804 indicating one of the selected atmos representations 822-828, and to update the audio stream 816 based on the selected atmos representation in response to the audio stream request.

[0225] Device 804 includes a streaming client 806, an audio decoder and binaural units 808, a head motion sensor 810, and multiple speakers 834. It can be accessed via one or more processors that execute instructions from memory (such as executing instructions 212 stored in memory 210). Figure 2One or more processors 220 of the second device 202 implement a streaming client 806, an audio decoder, and a binauralizer 808 at device 804. In some implementations, device 804 corresponds to the second device 202, the audio decoder and binauralizer 808 corresponds to decoder 228, sound field adjuster 224, and renderer 222, head motion sensor 810 corresponds to one or more sensors 244, and speaker 834 corresponds to speakers 240, 242. In some implementations, device 804 corresponds to... Figures 3A-7 One or more wearable devices.

[0226] Streaming client 806 is configured to receive audio stream 816 and provide it to an audio decoder and binauralizer 808, which outputs attitude-adjusted binaural audio signals to speaker 834. Head motion sensor 810 determines the attitude of device 804 by detecting the position and orientation of device 804. For illustration, head motion sensor 810 detects the current steering direction 812 of device 804 (which may correspond to the viewport field of view of device 804) and outputs head tracker data 811 (e.g., sensor data 246). Head tracker data 811 is provided to audio decoder and binauralizer 808 for sound field rotation (e.g., 3DOF), rotation and limited translation (e.g., 3DOF+), or rotation and translation (e.g., 6DOF). Head tracker data 811 is also provided to audio stream selector 814 of streaming client 806.

[0227] The audio stream selector 814 selects one of the immersive sound representations 822-828 based on the position of the device 804, the current turning direction 812, or other rotational information, or a combination thereof. The audio stream selector 814 issues an audio stream request 820 when it determines that the selected immersive sound representation is different from the previously selected immersive sound representation corresponding to audio stream 816.

[0228] Figure 840 illustrates eight overlapping viewport fields of view (VFOV 1-VFOV 8), where alternating dashed and solid lines are used to represent the viewport fields of view for clarity. Each viewport field of view corresponds to a 45-degree rotation of the wearer's head on device 804 when the wearer is positioned at the center of Figure 840. In this example, Figure 840 may correspond to eight possible viewports in a VR device in a 2D plane. Alternatively, Figure 840 may correspond to a 2D cross-section of eight overlapping spherical viewports in a VR device in 3D space.

[0229] Each overlapping viewport field of view corresponds to a corresponding immersive sound representation in memory 830. For example, the first viewport field of view 841 corresponds to the first immersive sound representation 822, the second viewport field of view 842 corresponds to the second immersive sound representation 824, and the third viewport field of view 843 corresponds to the third immersive sound representation 826. Although eight viewport fields of view are shown in a rotationally symmetric arrangement, in other implementations, fewer or more than eight viewport fields of view may be used, arranged in different ways, or in combinations thereof.

[0230] During operation, the wearer of device 804 (e.g., a headphone device) may face a first viewport field of view 841 (e.g., a first pose of device 804). In an implementation where source device 802 is configured to select a immersive representation for device 804 based on the pose of device 804, device 804 sends data associated with the first pose of device 804, such as an audio stream request 820 for a first immersive representation 822, or orientation information, translation information, or both.

[0231] Source device 802 receives data associated with a first pose of playback device 804 from device 804 (e.g., playback device) via wireless transmission. Based on this data, source device 802 selects a specific representation of the sound field (e.g., a first immersive sound representation 822) from a plurality of representations 822-828 of the sound field stored in memory 830. Source device 802 generates audio data corresponding to the selected first immersive sound representation 822 and transmits the audio data as streaming data (e.g., audio stream 816) to device 804 via wireless transmission. (See reference...) Figure 3B As described, audio data can be sent as compressed or uncompressed immersive sound coefficients, based on factors such as latency standards, available bandwidth, or both.

[0232] Device 804 receives audio data from source device 802 via wireless transmission, corresponding to a first immersive sound representation 822 of the sound field, which corresponds to a first viewport field of view 841 associated with a first pose of device 804. If the audio data is compressed, device 804 decompresses the compressed audio data and outputs the resulting audio to speaker 834 (e.g., two or more speakers).

[0233] In response to the wearer of device 804 rotating the user’s head toward the second viewport field of view 842 (e.g., the second pose of device 804), device 804 sends data associated with the second pose to source device 802 (e.g., an audio stream request 820 for the second immersive representation 824, or orientation information, translation information, or both).

[0234] Source device 802 receives second data associated with a second posture and selects a second panoramic sound representation 824 corresponding to the second viewport field of view from multiple representations of the sound field based on the second data. Source device 802 generates second audio data corresponding to the second representation of the sound field and transmits the second audio data as streaming data (e.g., audio stream 816) to device 804 via wireless transmission.

[0235] Device 804 receives updated audio data from source device 802 corresponding to a second panoramic sound representation 824, which corresponds to a second viewport field of view 842 that partially overlaps with the first viewport field of view 841 and is associated with a second pose. Device 804 outputs the updated audio data to speaker 834.

[0236] In some implementations, such as when immersive sound representations 822-828 correspond to the use of mixed-level immersive sound, the first immersive sound representation 822 provides a higher resolution for audio sources in the first viewport field of view 841 than for audio sources outside the first viewport field of view 841, and the second immersive sound representation 824 provides a higher resolution for audio sources in the second viewport field of view 842 than for audio sources outside the second viewport field of view 842. By altering the sound field as the device 804 changes its orientation, higher resolution can be provided for sounds of interest to the wearer, while bandwidth and processing resources can be saved by reducing the resolution of sounds that the wearer may not be of interest.

[0237] While in some implementations, the immersive sound representations 822-828 correspond to the use of mixed-order immersive sound, in others, they correspond to the use of the full immersive sound order. In other examples, the source device 802 may provide one or more of the following: an object-based representation of the sound field, a high-order immersive sound representation of the sound field, a mixed-order immersive sound representation of the sound field, a combination of an object-based representation of the sound field and a high-order immersive sound representation of the sound field, a combination of an object-based representation of the sound field and a mixed-order immersive sound representation of the sound field, or a combination of a mixed-order immersive sound representation of the sound field and a high-order immersive sound representation of the sound field. By varying between representations corresponding to overlapping viewport fields of view, abrupt transitions in immersive audio caused by switching between non-overlapping representations can be reduced or avoided, thereby improving the user experience.

[0238] In addition to the panoramic representation of the audio field based on rotation, in some implementations the panoramic representation of the audio field is selected based on the translation of the device 804 (such as as the wearer of the device 804 walks from the position of the wearer at the center of the graphics 840).

[0239] The second graphic 850 shows a portion of the viewport field of view set, which, in addition to VFOVs 1-8 of the first graphic 840, may include a fourth viewport field of view 851 (VFOV 31) and a fifth viewport field of view 852 (VFOV 32). A wearer 860 of device 804 is shown in a first posture at position "A" and facing the first viewport field of view 841. The wearer 860 can move to a second posture, in which device 804 translates to position "B" within a third field of view 843 and rotates to face the fourth viewport field of view 851. During the transition between the first and second postures, device 804 can receive streaming audio encoded based on a first immersive sound representation 822 (corresponding to VFOV 1), then encoded based on a second immersive sound representation 824 (corresponding to VFOV 2), and finally encoded based on a third immersive sound representation 826 (corresponding to VFOV 3). Once the second attitude is obtained at position B, device 804 receives streaming audio based on a representation of the sound field corresponding to the fourth viewport field of view 851.

[0240] Device 804 moves further from a second pose at position B to a third pose, wherein the wearer of device 804 is located at position "C" and facing the fifth viewport field of view 852, causing device 804 to receive streaming audio based on a representation of the sound field corresponding to the fifth viewport field of view 852. Therefore, the representation of the sound field selected from memory 830 can be selected based on rotation (e.g., as described with respect to figure 840), based on translation (e.g., moving from the second pose at position B to the third pose at position C), or based on both rotation and translation (e.g., moving from the first pose at position A to the second pose at position C).

[0241] In some implementations, device 804, source device 802, or both are configured to select a sound field representation corresponding to a translation of the sound field, and this translation exceeds the translation of device 804 between a first and a second pose. For example, even if wearer 860 does not move from position A, device 804 may select a sound field representation associated with fifth viewport field of view 852. For example, wearer 860 may be in a position that allows wearer 860 to “jump” to a distant position (e.g., to position C), or device 804 may include a camera capable of associating audio with a distant source, and device 804 may transition directly from one representation of the sound field (e.g., corresponding to third viewport field of view 843) to a second representation of the sound field (e.g., corresponding to fifth viewport field of view 852) without transitioning through a sound field representation corresponding to an intermediate viewport field of view (e.g., not using the sound field representation corresponding to fourth viewport field of view 851).

[0242] Therefore, the panoramic sound representation can be streamed based on rotation, translation, or both, and an appropriate panoramic sound representation of the sound field can be sent to device 804.

[0243] Figure 8B A block diagram of another implementation of system 800 is included, in which source device 802 stores representations of multiple sectors of a sound scene. The representations of the sound scene sectors are stored in various formats, shown as immersive representations 862-868 corresponding to sectors 1-N and pre-rendered stereo representations 872-878 corresponding to sectors 1-M, where M is a positive integer. Although in some implementations, the sectors of a sound scene can be similar to... Figure 8A The viewport field of view shown, and / or may be related to Figure 8A The viewport fields shown coincide, but Figure 8B The system 800 can be used independently of any visual reference, as illustrative and non-limiting examples, such as in an audio-only implementation, an extended reality (XR) implementation, or an augmented reality (AR) implementation without any viewport field of view.

[0244] In some implementations, each corresponding immersive sound representation 862-868 of the sound field corresponds to a different sector in a set of sectors, such as the first sector 881, the second sector 882, and the third sector 883 shown in Figure 880. Each sector represents a range of values ​​associated with movement of the playback device 804. Each of the immersive sound representations 862-868 includes immersive sound data (e.g., immersive sound coefficients corresponding to zero-order immersive, first-order immersive, higher-order immersive, mixed-order immersive, or combinations thereof). In some implementations, immersive sound representations 862-868 include or correspond to a pre-rotated sound field. For illustration, immersive sound representations 862-868 may correspond to regular increments of rotation about an axis (such as 1-degree increments), and memory 830 may store at least 360 immersive sound representations (e.g., N=360). As another example, immersive sound representations 862-868 may correspond to a pre-rotated sound field with increments of five degrees (e.g., N=72) or any other increment size. Similarly, in some implementations, each corresponding stereo representation also includes pre-rendered stereo data corresponding to a pre-rotated sound field (such as in 45-degree increments (e.g., M=8) or any other increment size). While uniform increments (e.g., 45-degree increments) are used in some implementations, non-uniform increment sizes can be used in others.

[0245] One or more processors 832 of source device 802 are configured to receive, via wireless transmission, attitude data 871 associated with the attitude of playback device 804. For example, attitude data 871 may be included in audio stream request 820 and may include orientation information, translation information, or both. Orientation information may indicate the detected orientation, detected rotation (e.g., change in orientation), or both of playback device 804. Translation information may indicate the detected position, detected translation (e.g., change in position), or both of playback device 804.

[0246] One or more processors 832 are configured to select a specific representation of the sound field from multiple representations of the sound field (such as from immersive representations 862-868 or from stereo representations 872-878) based on attitude data 871. The one or more processors 832 are configured to generate audio data corresponding to the selected representation of the sound field, and to transmit the audio data as streaming data (e.g., audio stream 816) to a playback device 804 via wireless transmission.

[0247] In some implementations, source device 802 selects a specific representation based on a predicted pose 870 of playback device 804. In one example, one or more processors 832 are configured to determine the predicted pose 870 based on a time series of pose data 871 received from playback device 804, such as via a Kalman filter or another prediction technique, and to select the representation based on a prediction of what the pose of playback device 804 will be when audio data corresponding to the selected representation is played. In another example, source device 802 receives the predicted pose 870 from playback device 804.

[0248] In some implementations, source device 802 is configured to further select a specific representation of the sound field based on the reference orientation of playback device 804. For example, playback device 804 may correspond to a helmet-mounted device, and during helmet initialization, the helmet's orientation and position can be used as a reference orientation (e.g., the source device 802 can be used as a coordinate origin from which changes in orientation or translation are calculated). Source device 802 is operable to update the reference orientation based on the orientation of the playback device and in response to one or more events, such as receiving a user instruction to update the reference orientation. In an illustrative example, source device 802 may update the reference orientation based on receiving a reference reset instruction, such as being received wirelessly from playback device 804 in response to user input received at the user interface of playback device 804.

[0249] In some implementations, source device 802 is configured to select a specific representation of the sound field in an audio format having a different audio format than the previous representation of the sound field, based on a change in the orientation of playback device 804 exceeding a threshold. For illustration, when the amount of movement of playback device 804 (e.g., the speed of a user's head rotation) exceeds or is predicted to exceed a threshold, this movement may impair the user's ability to perceive the fine resolution of the sound field, and source device 802 may switch from streaming audio data in an immersive sound format to streaming audio data in a pre-rendered stereo format. When the amount of movement is below (or predicted to be below) the threshold, source device 802 can revert to sending audio data using the immersive sound format.

[0250] Examples of sectors corresponding to the range of values ​​associated with both rotation and translation of the playback device 804 are graphically depicted in figures 880 and 886, respectively. In such examples, the source device 802 can be referenced respectively. Figure 8A In a similar manner to that described in graphics 840 and 850, the appropriate representation of the sound field is selected based on the movement of the playback device 804.

[0251] Although two audio formats (Atmos and Stereo) are shown, it should be understood that in other implementations, source device 802 may operate using a single audio format or more than two audio formats to provide audio stream 816 to device 804. Although Atmos and Stereo formats are shown, in other implementations, one or more other audio formats may be used instead of Atmos, Stereo, or both, or one or more other audio formats may be used in addition to Atmos, Stereo, or both.

[0252] although Figure 8B An example of sector overlap is depicted, but in other implementations, source device 802 selects the representation of the sound field based on non-overlapping sectors representing a range of values ​​associated with the movement of playback device 804. For illustration, Figure 8C An example of system 800 is depicted, in which source device 802 selects a representation of the sound field based on non-overlapping sectors associated with movement of playback device 804. Figure 8C In Figure 890, eight non-overlapping sectors can be used to select a stereo representation of the sound field based on a coarser estimate of the orientation of the playback device 804 (e.g., based on a 45-degree rotation increment), and Figure 892 shows sixteen non-overlapping sectors that can be used to select an atmos representation of the sound field based on a finer estimate of the orientation of the playback device 804 (e.g., based on a 22.5-degree rotation increment). Although Figures 890 and 892 show eight and sixteen sectors, respectively, for the sake of simplicity, it should be understood that any other number of sectors can be used, and the sectors can be of uniform or non-uniform size.

[0253] Figure 9 This is a block diagram illustrating an implementation of system 900 corresponding to device 804, wherein one or more immersive sound representations 822-828 are stored in memory 930 at streaming client 906 (e.g., a local streaming client). In one example, streaming client 906 may use a list of locally available audio representations to determine the feasibility of one or more of the locally available representations 822-828, and then use the information provided in the list to select an appropriate sound field representation.

[0254] As a result, compared to requesting panoramic sound representations 822-828 from a remote streaming server, the streaming client 906 can switch between panoramic sound representations 822-828 based on changes in orientation, translation, or both detected by the head motion sensor 810 and with reduced latency. Although the streaming client 906 is shown as including panoramic sound representations 822-828, in other implementations, the streaming client may use any other set of representations of the sound field instead of panoramic sound representations 822-828, or any other set of representations of the sound field besides panoramic sound representations 822-828. For example, in some implementations, besides Figure 8B In addition to one or more of the stereo representations 872-878, the streaming client 906 also stores the immersive representations 822-828 (or immersive representations 862-868).

[0255] Figure 10A This is a block diagram illustrating another implementation of the components and operation of a system for adjusting a sound field. System 1000 includes a streaming device 1002 configured to generate encoded immersive audio data 1018 representing the sound field using scalable audio decoding (e.g., compressed or uncompressed immersive coefficients, such as reference...). Figure 3B The encoding operation 380 described herein is used to send at least a portion of the panoramic sound data 1010 to the wearable device 1004. The wearable device 1004 includes a scalable audio decoder configured to decode the encoded panoramic sound audio data 1018.

[0256] Encoded panoramic audio data 1018 is shown in Figure 1050 as being transmitted from streaming device 1002 to wearable device 1004 as a frame sequence 1051-1064 using scalable audio coding. Scalable audio coding comprises a base layer of audio data providing coarse audio information and one or more higher layers (referred to as "enhancement layers") providing audio data with finer resolution. As shown, the first four frames 1051-1054 are encoded using first-order panoramic audio (FOA), and the next three frames 1055-1057 are encoded using second-order panoramic audio (SOA), which provides a higher resolution than FOA. The next two frames 1058 and 1059 are encoded using third-order panoramic audio (TOA), which provides a higher resolution than SOA. Frames 1060 and 1061 are encoded using SOA, and frames 1062-1064 are encoded using FOA.

[0257] In the illustrative example, frames 1051-1064 correspond to the orchestral performance of the song, which begins with a relatively small number of instruments (e.g., a single instrument) and is encoded using FOA. As more instruments begin to play, providing more different types of sound in the sound scene, the encoding transitions to SOA, and then to TOA, to provide a progressively increasing resolution of the sound scene. As the number of instruments begins to decrease, the encoding reverts from TOA to SOA, and then back to FOA. In this example, each frame encodes approximately one second of the sound scene, but in other implementations, each frame could correspond to a longer or shorter time span.

[0258] In another illustrative example, one or more transitions between the encoding types of frames 1051-1064 are based on the movement of wearable device 1004. For example, wearable device 1004 may acquire head tracker data 1036 associated with the location and orientation of the movement of wearable device 1004 at multiple time instances, and transmit at least a portion of the head tracker data 1036 to streaming device 1002 via wireless transmission, such as... Figure 2 Data 166, 168. Streaming device 1002 may include one or more processors (e.g., Figure 1 One or more processors 120 are configured to receive head tracker data from a playback device via wireless transmission as first data associated with a first pose of the wearable device 1004 (e.g., the playback device). As further described below, the first pose may be associated with a first number of sound sources in a sound scene. The streaming device 1002 may generate a first frame (e.g., frame 1504) of encoded panoramic audio data corresponding to a base layer encoding of the sound scene and transmit the first frame to the wearable device 1004.

[0259] In the panoramic audio data, the transition from a frame encoded according to the base layer (e.g., frame 1054) to a subsequent frame encoded according to the enhancement layer (e.g., frame 1055) corresponds to the movement of the wearable device 1004. For example, the transition from the base layer encoding of frame 1054 to the enhancement layer encoding of frame 1055 corresponds to a transition from a first orientation of the wearable device 1004 associated with a first number of sound sources to a second orientation of the wearable device 1004 associated with a second number of sound sources, the second number being greater than the first number. For illustration, frames 1051-1054 may correspond to the wearable device 1004 on the user's head and facing... Figure 8A First viewport field of view 841 or Figure 8B The first sector 881 has a relatively small number of sound sources.

[0260] In response to a user's head movement changing the orientation of wearable device 1004 to another viewport field of view (e.g., a second viewport field of view 842) or towards another sector that includes a greater number of audio sources than the first viewport field of view 841 or the first sector 881, an enhancement layer is used to encode subsequent frame 1055 to achieve higher resolution, thereby accommodating the greater number of sound sources. For example, wearable device 1004 sends updated head tracker data indicating user head movement, which is received at streaming device 1002 as second data associated with a second pose of wearable device 1004, which is associated with a second number of sound sources. Streaming device 102 is configured to generate a second frame (e.g., frame 1055) of encoded panoramic audio data corresponding to the enhancement layer encoding of the sound scene, and sends the second frame to wearable device 1004.

[0261] Wearable device 1004 is configured to perform an immersive audio decoding operation 1020 to generate decoded immersive audio data 1022. Based on head tracker data 1036 from one or more sensors 1034, the decoded immersive audio data 1022 is processed via an immersive sound field 3DOF / DOF+ rotation and binauralization operation 1024 to provide posture-adjusted binaural audio signals 1026, 1028 to speakers 1030, 1032. In this illustrative implementation, wearable device 1004 corresponds to second device 202, performs the immersive audio decoding operation 1020 at decoder 228, performs the immersive sound field 3DOF / DOF+ rotation and binauralization operation 1024 at sound field adjuster 224 and renderer 222, one or more sensors 1034 correspond to one or more sensors 244, and speakers 1030, 1032 correspond to speakers 240, 242.

[0262] Wearable device 1004 uses a scalable decoder including a base layer decoder 1040, a first enhancement layer decoder 1042, and a second enhancement layer decoder 1044 to perform immersive audio decoding operation 1020. Although two enhancement layer decoders 1042, 1044 are depicted, in other implementations, wearable device 1004 includes a single enhancement layer decoder or three or more enhancement layer decoders.

[0263] The base layer decoder 1040 is configured to decode FOA-coded frames, the first enhancement layer decoder 1042 is configured to decode SOA-coded frames, and the second enhancement layer decoder 1044 is configured to decode TOA-coded frames. The immersive audio decoding operation 1020 can adjust, on a frame-by-frame basis, which of the decoders 1040, 1042, and 1044 is used to decode each of frames 1051-1064. In the illustrative example, the base layer decoder 1040 is activated to decode FOA frames 1051-1054, the first enhancement layer decoder 1042 is activated to decode SOA frames 1055-1057, and the second enhancement layer decoder 1044 is activated to decode TOA frames 1058 and 1059. The second enhancement layer decoder 1044 is deactivated after decoding TOA frame 1059, and the first enhancement layer decoder 1042 is deactivated after decoding SOA frame 1061.

[0264] although Figure 10A The description shows that each of the base layer decoder 1040 and enhancement layer decoders 1042, 1044 corresponds to a single corresponding Atmos order, but in other implementations, each layer (and associated decoder) may correspond to multiple Atmos orders or resolutions, such as... Figure 11A , Figure 11B , Figure 12 Hehe Figure 13 As depicted in the illustrative examples.

[0265] Figure 10B An implementation of system 1000 is depicted, wherein the order of the panoramic sound data 1018 sent to device 1004, decoded by device 1004, or both, is based on the amount of movement of device 1004. For example, when device 1004 corresponds to a head-mounted wearable device and the wearer's head moves by an amount exceeding a certain threshold (e.g., translation, orientation change, or both), the wearer may not be able to perceive the obtained audio at the resolution level provided by the full-order panoramic sound data 1010 representing the sound field. However, when the amount of movement of device 1004 is equal to or less than the threshold, a higher resolution or full-resolution representation of the audio scene can be provided. Therefore, latency, computational resource usage, power consumption, or any combination thereof can be controlled based on the amount of movement of device 1004 or predicted movement.

[0266] As shown in the figure, device 1004 is configured to perform motion-based resolution selection 1070 based on data 1037 from one or more sensors 1034 indicating a movement 1072 of device 1004. In some implementations, device 1004 compares the movement 1072 with one or more thresholds 1074 to determine an amount of audio resolution to be provided for playback at speakers 1030, 1032. For example, threshold 1074 may indicate a threshold amount of movement associated with a coding layer, such as a first threshold amount of movement exceeding which it intends to perform only base layer decoding, and a second threshold amount of movement exceeding which it intends to perform only base layer and first enhancement layer decoding. As another example, thresholds may indicate threshold amounts of movement associated with various immersive orders, such as a first threshold amount of movement exceeding which it intends to perform only zero-order immersive coefficient decoding, a second threshold amount of movement exceeding which it intends to perform only zero-order and first-order immersive coefficient decoding, and so on.

[0267] Based on the resolution selection of the motion, 1070 generates a set of one or more signals to control the decoding of the received panoramic sound data 1018. For example, a first signal 1080 may instruct the panoramic sound audio decoding operation 1020 which level of the panoramic sound data 1018 to decode, a second signal 1082 may control the operation of the base layer decoder 1040, a third signal 1084 may control the operation of the first enhancement layer decoder 1042, and a fourth signal 1086 may control the operation of the second enhancement layer decoder 1044.

[0268] For illustration, the second signal 1082 can configure the base layer decoder 1040 to decode only the immersive coefficients corresponding to zero-order immersive (e.g., to generate a non-directional audio signal), decode the immersive coefficients corresponding to first-order immersive, or both. The third signal 1084 can configure the first enhancement layer decoder 1042 to not decode any immersive coefficients, but only decode second-order immersive coefficients, or decode both second-order and third-order immersive coefficients. The fourth signal 1086 can configure the second enhancement layer decoder 1044 to not decode any immersive coefficients, but only decode fourth-order immersive coefficients, or decode both fourth-order immersive coefficients and immersive coefficients corresponding to immersive orders higher than fourth (not shown).

[0269] Movement 1072 may represent the measured movement of device 1004 based on data 1037, the predicted movement of device 1004 (e.g., using a Kalman filter or another prediction technique), or a combination thereof. For illustration, movement-based resolution selection 1070 may include determining the future predicted pose 1076 of device 1004, the predicted amount of movement of device 1004, or both. In some implementations, movement-based resolution selection 1070 includes determining: a prediction duration 1078 during which the amount of movement of device 1004 will exceed one or more thresholds 1074 (and thus reduce the amount of immersive data decoding), a prediction duration 1078 during which the amount of movement of device 1004 will remain less than or equal to one or more thresholds 1074 (and thus increase the amount of immersive data decoding), or a combination thereof.

[0270] During operation, device 1004 may receive encoded panoramic audio data 1018 representing the sound field from streaming device 1002 via wireless transmission 1006. Device 1004 may perform decoding of the panoramic audio data 1018 to generate decoded panoramic audio data 1022. Decoding of the panoramic audio data 1022 may include base layer decoding of the base layer of the encoded panoramic audio data 1022, and may selectively include enhancement layer decoding in response to a movement amount 1072 of device 1004. In some implementations, device 1004 adjusts the decoded panoramic audio data 1022 to alter the sound field based on at least one associated head tracker data 1036 of translation or orientation associated with the movement 1072 of device 1004, and outputs the adjusted decoded panoramic audio data to two or more speakers for playback.

[0271] In some implementations, device 1004 is configured to perform enhancement layer decoding based on a movement amount 1072 being less than a threshold amount (e.g., not exceeding one or more thresholds 1074), and to avoid performing enhancement layer decoding based on a movement amount not being less than a threshold amount. Device 1004 can determine a threshold atmos order based on the movement amount to select whether to perform enhancement layer decoding in response to the movement amount of device 1004, such as selecting second order in response to a relatively large movement amount or selecting a higher order (e.g., fourth order) in response to a relatively small movement amount. Device 1004 can decode enhancement layers corresponding to atmos orders less than the selected threshold atmos order, and can avoid decoding enhancement layers corresponding to atmos orders greater than or equal to the selected threshold atmos order.

[0272] In some implementations, device 1004 is configured to send a message to streaming device 1002 based on motion amount 1072 to avoid transmitting enhancement layer audio data. For example, motion-based resolution selection 1070 can generate a signal 1088 to be transmitted to streaming device 1002 via wireless transmission 1090. Signal 1088 may include an indication of the highest order of the immersive audio data to be transmitted, an indication of one or more orders of the immersive audio data 1010 to be transmitted, an indication of one or more enhancement layers to be transmitted, or a combination thereof. In an illustrative example, signal 1088 indicates order selection 1092, which can be used to adjust the order of the encoded immersive audio data 1018 during encoding at streaming device 1002, such as by referencing motion-based resolution selection 1092. Figure 3B , Figure 4B or Figure 5B The order is chosen using 334 encoding, as shown in the reference. Figures 6-7 The request described is for a specific Atmos order of 654, or a combination thereof.

[0273] In some implementations, signal 1088 includes a message for streaming device 1002 to avoid transmitting enhancement layer audio data for a specific duration. For example, motion-based resolution selection 1070 may determine that the amount of motion 1072 exceeds a threshold amount (e.g., exceeds one or more thresholds 1074), and determine a predicted duration 1078 based on a prediction of when the amount of motion of device 1004 will be less than the threshold amount. Device 1004 may send signal 1088 to streaming device 1002 to avoid transmitting enhancement layer audio data (e.g., only transmitting immersive coefficients corresponding to the base layer, such as zero-order and first-order coefficients) until a future time (e.g., coinciding with it), which future time is based on when the amount of motion of device 1004 1072 is predicted to be sufficiently reduced to perceive a higher resolution representation of the audio scene.

[0274] In some implementations, streaming device 1002 calculates the actual or predicted amount of movement 1072 of device 1004 and determines the amount of augmented layer audio data to be sent to wearable device 1004 based on the calculated amount of movement 1072. For example, device 1004 may acquire data 1037 associated with the location and orientation of tracking the movement 1072 of device 1004 at multiple time instances and may send data 1037 to streaming device 1002 via wireless transmission 1090. For illustration, data 1037 may correspond to signal 1088 or be included in signal 1088. In such an implementation, the functionality described for motion-based resolution selection 1070 may be performed at streaming device 1002 instead of at device 1004 or appended to device 1004. In this implementation, the amount of enhancement layer audio data in the encoded panoramic audio data 1018 received by device 1004 from streaming device 1002 is based on the amount of movement of device 1004, such as in combination with... Figure 4B The encoding operation performed on the mobile data 476 is similar to that described in 380.

[0275] Figure 10C A graph 1098 is shown illustrating the implementation where encoded panoramic audio data 1018 is transmitted from streaming device 1002 to device 1004 as a frame sequence 1051-1064. The base layer includes zero-order panoramic (ZOA) data and no panoramic data of higher orders than zero-order. The first enhancement layer includes FOA and SOA data, and the second enhancement layer includes TOA data. As shown, the first four frames 1051-1054 correspond to the base layer and include ZOA data, and the next three frames 1055-1057 correspond to the first enhancement layer and include FOA data (frames 1055, 1056) and SOA data (frame 1057). The next two frames 1058, 1059 correspond to the second enhancement layer and include TOA data. Frames 1060-1062 correspond to the first enhancement layer and include SOA data (frame 1060) and FOA data (frames 1061, 1062), while frames 1063-1064 correspond to the base layer and include ZOA data. For example, based on determining or predicting that device 1004 experiences relatively large motion (e.g., ZOA data in frames 1051-1054), which gradually decreases until device 1004 becomes stationary (e.g., TOA data in frames 1059-1059), and then increases to relatively large motion of device 1004 (e.g., ZOA data in frames 1063-1064), different orders of panoramic sound data in the frame 1051-1064 sequence can be transmitted, decoded, or both.

[0276] exist Figure 11A and Figure 11BThe image shows a sequence of panoramic audio frames 1111-1124, including frames 1111-1114 corresponding to FOA, frames 1115-1117 corresponding to SOA, frames 1118 and 1119 corresponding to fourth-order panoramic sound (fourth OA), frames 1120 and 1121 corresponding to SOA, and frames 1122-1124 corresponding to FOA.

[0277] exist Figure 11A In this structure, the base layer 1102 corresponds to the Forward Atmosphere (FOA), the first enhancement layer 1104 corresponds to the Solo Atmosphere (SOA) and the Tolo Atmosphere (TOA), and the second enhancement layer 1106 corresponds to the fourth OOA and higher-order immersive sound. Therefore, FOA frames 1111-1114 and 1122-1124 correspond to the base layer 1102 and can be decoded by the base layer decoder 1040. SOA frames 1115-1117, 1120, and 1121 correspond to the first enhancement layer 1104 and can be decoded by the first enhancement layer decoder 1042. Fourth OOA frames 1118 and 1119 correspond to the second enhancement layer 1106 and can be decoded by the second enhancement layer decoder 1044.

[0278] exist Figure 11B In this structure, the base layer 1102 corresponds to the Forward Object Frame (FOA), and the first enhancement layer 1108 corresponds to the Sort of Objects (SOA), the Tort of Objects (TOA), and the fourth Outer Object (OA). Therefore, FOA frames 1111-1114 and 1122-1124 correspond to the base layer 1102 and can be decoded by the base layer decoder 1040. SOA frames 1115-1117, 1120, and 1121 and fourth OA frames 1118 and 1119 correspond to the first enhancement layer 1108 and can be decoded using the first enhancement layer decoder 1042.

[0279] exist Figure 12The diagram shows a sequence of panoramic audio frames 1211-1224, including frames 1211-1214 corresponding to ZOA, frame 1215 corresponding to FOA, frames 1216-1217 corresponding to SOA, frames 1218 and 1219 corresponding to the fourth OA, frames 1220 and 1221 corresponding to TOA, and frames 1222-1224 corresponding to ZOA. The base layer 1202 corresponds to ZOA, the first enhancement layer 1204 corresponds to FOA and SOA, and the second enhancement layer 1206 corresponds to TOA, the fourth OA, and higher-order panoramic sound. Therefore, ZOA frames 1211-1214 and 1222-1224 correspond to base layer 1202 and can be decoded by base layer decoder 1040. FOA and SOA frames 1215-1217 correspond to the first enhancement layer 1204 and can be decoded using first enhancement layer decoder 1042. The fourth OA frames 1218 / 1219 and TOA frames 1220, 1221 correspond to the second enhancement layer 1206 and can be decoded using the second enhancement layer decoder 1044.

[0280] exist Figure 13 The image shows a sequence of panoramic sound audio frames 1311-1322, including frames 1311-1314 corresponding to the Mixed Order of the Panoramic Sound (MOA), frames 1315-1317 corresponding to the Total Order of the Sound (TOA), frames 1318 and 1319 corresponding to the Fourth OA, and frames 1320-1322 corresponding to the MOA. For example, the MOA representation can provide accuracy for some areas of the sound field, but lower accuracy for others. In one example, the MOA representation of the sound field may include eight coefficients (e.g., one coefficient for n=0, three coefficients for n=1, two coefficients for n=2, etc.). Figure 1 The two outermost ones depicted in the middle), and the two coefficients with n=3 ( Figure 1 (The outermost two are depicted in the image). Conversely, the TOA representation of the same sound field can include sixteen coefficients. Therefore, compared to the corresponding TOA representation of the same sound field, the MOA representation of the sound field can be less storage-intensive and less bandwidth-intensive, and can provide a lower resolution sound field representation.

[0281] The base layer 1302 corresponds to the MOA, the first enhancement layer 1304 corresponds to the TOA, and the second enhancement layer 1306 corresponds to the fourth OA and higher-order immersive sound. Therefore, MOA frames 1311-1314 and 1320-1322 correspond to the base layer 1302 and can be decoded by the base layer decoder 1040. TOA frames 1315-1317 correspond to the first enhancement layer 1304 and can be decoded using the first enhancement layer decoder 1042. Fourth OA frames 1318 and 1319 correspond to the second enhancement layer 1306 and can be decoded using the second enhancement layer decoder 1044.

[0282] Therefore, wearable device 1004 can be implemented to include one or more processors, such as one or more processors 220, configured to receive encoded panoramic sound audio data 1018 representing a sound field from streaming device 1002 via wireless transmission. For example, wearable device 1004 receives encoded panoramic sound audio data 1018 from streaming device 1002 via wireless transmission 1006.

[0283] One or more processors of wearable device 1004 may be configured to perform decoding of encoded panoramic sound audio data 1018 to generate decoded panoramic sound audio data 1022. Decoding of the encoded panoramic sound audio data 1018 includes base layer decoding of the base layer of the encoded panoramic sound audio data 1018 (e.g., FOA), and selectively includes enhancement layer decoding in response to detection that the encoded panoramic sound audio data 1018 includes at least one encoded enhancement layer (e.g., SOA). As an example, base layer decoding is performed using a base layer decoder 1040, and enhancement layer decoding is performed using at least a first enhancement layer decoder 1042 corresponding to a first enhancement layer of the encoded panoramic sound audio data 1018.

[0284] One or more processors of the wearable device 1004 may be configured to adjust decoded immersive audio data to alter the sound field, such as via immersive sound field 3DOF / 3DOF+ rotation and binaural operation 1024, based on data associated with at least one of translation or orientation related to the movement of the device.

[0285] One or more processors of the wearable device 1004 may be configured to output tuned and decoded panoramic audio data to two or more speakers for playback, such as attitude-tuned binaural audio signals 1026 and 1028 provided to speakers 1030 and 1032 respectively.

[0286] In some implementations, the encoded panoramic audio data 1018 includes first-order panoramic data in the base layer and higher-order panoramic data in the first enhancement layer, such as... Figure 11A and Figure 11B As shown, the first enhancement layer decoder 1042 is configured to decode high-order panoramic sound data.

[0287] In some implementations, the encoded panoramic audio data 1018 includes first-order panoramic data in the base layer, one or more higher-order panoramic data in the first enhancement layer, and one or more additional higher-order panoramic data in the second enhancement layer, such as... Figure 11AAs shown. In such an implementation, one or more processors of the wearable device 1004 are also configured to perform enhancement layer decoding using a second enhancement layer decoder 1044, which is configured to decode additional high-order immersive audio data.

[0288] In some implementations, the encoded immersive audio data 1018 includes mixed-order immersive data (which contains a partial set of coefficients for the immersive orders in the base layer) and additional immersive data in the enhancement layer, which includes one or more coefficients for the immersive orders omitted from the base layer, such as... Figure 13 As shown in the diagram. In such an implementation, the hybrid-order immersive sound can be decoded by the base layer decoder 1040, the additional immersive sound data (e.g., TOA) can be decoded by the first enhancement layer decoder 1042, and the additional higher-order immersive sound data (e.g., fourth OA) can be decoded by the second enhancement layer decoder 1044.

[0289] In conjunction with the aforementioned devices and systems, as a non-limiting example, streaming audio data can be tailored based on one or more criteria such as the latency requirements of the playback device, bandwidth, or motion. Although references... Figure 8B and Figure 8C An example of switching between immersive and pre-rendered stereo audio formats is described, but in other implementations, one or more other formats, such as pulse code modulation (PCM) audio or object audio formats, may be used in addition to or instead of immersive or pre-rendered stereo.

[0290] In one example, if the playback device is stationary or has relatively little motion, the streaming source (e.g., source device 802 or streaming device 1002) can generate binauralized pre-rendered stereo from full-order immersive sound and send this pre-rendered stereo to the playback device. However, in response to detecting or predicting motion of the playback device, the streaming source can switch from sending pre-rendered stereo to sending low-order immersive sound data (e.g., FOA or SOA) to rotate locally at the playback device. In response to detecting or predicting that the playback device has stopped moving, the streaming source can switch back to sending pre-rendered stereo.

[0291] In some implementations, streaming sources can combine transitions between enhancement layers to switch between formats. For example, streaming device 1002 can switch between sending a representation of an audio scene encoded using base layer immersive sound and sending a pre-rendered stream encoded with both base and enhancement layers. This can provide enhanced resolution for the audio scene with reduced bandwidth compared to sending enhancement layer immersive sound coefficients.

[0292] In some implementations, the source device can switch between encoded formats and / or layers (e.g., mono, stereo, base layer, base layer, and enhancement layer, etc.) based on one or more other conditions. For example, streaming audio can be adjusted in response to the detection or prediction of an event in which a richer audio experience or a reduced audio resolution would be appropriate. For instance, in applications where a user wearing a playback device moves within a virtual reality or mixed reality setting or immersive audio scene, and an event such as the initiation of a voice call or local or streaming audio playback separate from the immersive audio scene is detected or predicted, the source device can downmix the audio to mono or stereo PCM based on the prediction of which sound source the user will focus on. Taking a voice call as an example, the audio resolution of the scene can be reduced to a lower level of mono to better allow the user to focus on the voice call. Similarly, as an illustrative rather than limiting example, for concurrent stereo audio playback, the immersive audio scene can be reduced to a base layer presented in stereo.

[0293] In some implementations, split rendering can be performed via a virtual assistant (e.g., running on a mobile phone, cloud server, other electronic device, or a combination thereof), and the helmet wearer can operate the helmet in pass-through mode, where audio received via one or more external microphones of the helmet is played to the wearer via the helmet's earpiece. To illustrate, in an example where the helmet wearer is listening to music sent to the helmet as audio data encoded using a base layer and optionally one or more enhancement layers in a coffee shop, sounds captured by the helmet's external microphones (such as the barista's voice) can be mixed with the base layer of the music to be played to the wearer.

[0294] Figure 14 This is a block diagram illustrating an implementation 1400 of a first device 102 as an integrated circuit 1402 for adjusting a sound field. The integrated circuit 1402 includes one or more processors 120. The one or more processors 120 include a sound field representation generator 124, an encoder 128, or both. The integrated circuit 1402 also includes signal inputs 1404 and 1406 (such as a bus interface) to enable receiving sound information 1430 (e.g., sound information 123) from an audio source and translation data 1440 (e.g., data 166) from a playback device. The integrated circuit 1402 also includes a signal output 1412 (such as a bus interface) to enable transmitting audio data 1450 (e.g., encoded audio data 129) after adjusting the sound field based on the translation data 1440. The integrated circuit 1402 enables sound field adjustment as a component in a system including an audio source and a wireless transceiver, such as... Figures 3A-8C Or 10A- Figure 10B The streaming device shown.

[0295] Figure 15 This is a block diagram illustrating an implementation 1500 of a second device 202 as an integrated circuit 1502 for adjusting a sound field. The integrated circuit 1502 includes one or more processors 220. The one or more processors 220 include a decoder 228, a sound field adjuster 224, or both. The integrated circuit 1502 also includes signal inputs 1504 and 1506 (such as a bus interface) to enable receiving compressed audio data 1530 (e.g., encoded audio data 229) and head tracking data 1540 (e.g., sensor data 246) from one or more sensors. The integrated circuit 1502 also includes a signal output 1512 (such as one or more bus interfaces) to enable transmitting posture-adjusted binaural signals 1550 (e.g., signals 239, 241) after adjusting the sound field based on the head tracking data 1540. The integrated circuit 1502 is capable of implementing sound field adjustment as a component in a system including a wireless receiver and two or more speakers, such as... Figures 3A-10B Wearable devices (such as headphones), as shown Figure 16 The speaker array shown, such as Figure 17 The virtual reality, mixed reality, or augmented reality headsets shown are, for example... Figure 18 or Figure 19 The vehicle shown or such Figure 32 The wireless communication device shown.

[0296] Figure 16 This is a block diagram illustrating an implementation of a system 1600 for adjusting the sound field, wherein a second device 202 corresponds to or is integrated into a speaker array (such as a soundbar device 1602). The soundbar device 1602 is configured to perform beamguiding operations to direct binaural signals to a location associated with a user. The soundbar device 1602 may receive immersive sound audio data 1608 from a remote streaming server via a wireless network 1606. The soundbar device 1602 may include one or more processors 220 (e.g., including a decoder 228, a sound field adjuster 224, or both), configured to adjust the sound field represented by the immersive sound audio data 1608 and perform beamguiding operations to direct binaural signals to a location associated with a listener 1620.

[0297] The soundbar device 1602 includes or is coupled to one or more sensors (e.g., a camera, structured light sensor, ultrasound, lidar, etc.) to enable the detection of the listener's posture 1620 and the generation of head tracker data for the listener 1620. For example, the soundbar device 1602 can detect the listener 1620's posture at a first position 1622 (e.g., at a first angle to a reference 1624), adjust the sound field based on the listener 1620's posture, and perform beamguiding operations such that the emitted sound 1604 is perceived by the listener 1620 as a posture-adjusted binaural signal. In one example, the beamguiding operation is based on the listener 1620's first position 1622 and a first orientation (e.g., facing the soundbar 1602). In response to a change in the posture of the listener 1620, such as the listener 1620 moving to the second position 1632, the soundbar 1602 adjusts the sound field (e.g., according to 3DOF / 3DOF+ or 6DOF operation) and performs beam steering operation so that the resulting emitted sound 1604 is perceived by the listener 1620 as a posture-adjusted binaural signal at the second position 1632.

[0298] Figure 17 An implementation 1700 is depicted, wherein the second device 202 is a portable electronic device corresponding to a virtual reality, augmented reality, or mixed reality headset 1702. One or more processors 220 (e.g., including decoder 228, sound field adjuster 224, or both), speakers 240, 242, memory 210, one or more sensors 244, transceiver 230, or combinations thereof, are integrated into the headset 1702. Adjustments to the sound field corresponding to audio data received from a remote streaming server can be performed based on head tracker data generated by one or more sensors 244, such as reference... Figures 2-13 As described.

[0299] Figure 18An implementation 1800 is depicted, wherein a first device 102 corresponds to or is integrated within a vehicle 1802, shown as a car. In some implementations, one or more processors 220 (e.g., including a decoder 228, a sound field adjuster 224, or both) are integrated into the vehicle 1802, and data from one or more sensors 244 indicates the translation and orientation of the vehicle 1802. In some implementations, the data indicating the translation and orientation of the vehicle 1802 is sent to a remote server (such as the first device 102). Audio data (e.g., navigation data) from the remote server can be received at the vehicle 1802, and the sound field associated with the received audio data can be adjusted based on translation, orientation, or both before being played at one or more speakers in the vehicle 1802. For example, the playback of navigation data (e.g., verbal driving directions to a destination) can be adjusted so that, to the occupants of vehicle 1802, the verbal directions appear to originate from the location or direction of the navigation destination, and thus can provide the driver of vehicle 1802 with additional information encoded as a perception of the distance and direction of the source of the verbal navigation directions.

[0300] Figure 19 Another implementation 1900 is depicted, in which the second device 202 corresponds to or is integrated within the vehicle 1902, which is shown as a manned or unmanned aerial device (e.g., a package delivery drone). In some implementations, one or more processors 220 (e.g., including decoder 228, sound field adjuster 224, or both) are integrated into the vehicle 1902, and data from one or more sensors 244 indicate the translation and orientation of the vehicle 1902. In some implementations, the vehicle 1902 is manned (e.g., carrying a pilot, one or more passengers, or both) and the sound field of the received audio data can be adjusted in a manner similar to that described for reference vehicle 1802. In another implementation where the vehicle 1902 is unmanned and the speakers 240, 242 are located on the outer surface of the vehicle 1902, the vehicle can be... Figure 16 A similar manner to that described for a loudspeaker array works to adapt beamforming based on changes in the listener’s posture during audio playback to one or more listeners (e.g., as used as a hovering loudspeaker array). Alternatively or additionally, the vehicle 1902 may move while playing audio (e.g., around an outdoor audience during a concert), and one or more processors 220 (e.g., including decoder 228, sound field adjuster 224, or both) may perform operations to adjust the sound field based on the translation and rotation of the vehicle 1902.

[0301] Figure 20A first example of a method 2000 for adjusting a sound field is shown. As an illustrative and non-limiting example, method 2000 may be performed by an electronic device such as second device 202, wearable device 304, wearable device 404, or wearable device 504.

[0302] Method 2000 includes receiving compressed audio data representing a sound field via wireless transmission at one or more processors at point 2002. For example, in Figure 2 In this process, one or more processors 220 of the second device 202 receive encoded audio data 229 representing the sound field 126. The encoded audio data 229 may be compressed and received as streaming data from a streaming device (e.g., the first device 102), and the streaming device may correspond to at least one of a portable electronic device or a server.

[0303] Method 2000 includes decompressing the compressed audio data at point 2004. For example, Figure 2 The decoder 228 can decompress the encoded audio data 229 to generate audio data 227 (e.g., decompressed audio data). In some examples, the decompressed audio data (e.g., audio data 227) includes immersive sound data.

[0304] Method 2000 includes, at 2006, adjusting the decompressed audio data to alter the sound field based on data associated with at least one of translation or orientation related to movement of the device. For example, Figure 2 The sound field modulator 224 adjusts the sound field 226 based on sensor data 246.

[0305] Method 2000 includes rendering the adjusted decompressed audio data to two or more speaker gains at point 2008 to drive two or more speakers. For example, Figure 2 The renderer 222 renders the adjusted audio data 223 to generate speaker gains 219, 221.

[0306] Method 2000 includes outputting the adjusted decompressed audio data to two or more speakers at point 2010 for playback. For example, Figure 2 One or more processors 220 drive speakers 240 and 242 with attitude-adjusted binaural audio signals 239 and 241 based on the output of speaker gains 219 and 221.

[0307] In some implementations, method 2000 includes performing binauralization of the adjusted decompressed audio data to generate two or more speaker gains, such as using HRTF or BRIR with or without headphone compensation filters associated with electronic devices.

[0308] In some implementations, method 2000 further includes sending translation data, such as data 166 or translation metadata 478, to a streaming device (e.g., first device 102). The translation data is associated with movement of the device (e.g., second device 202). In response to sending the translation data, compressed updated audio data (e.g., encoded audio data 229) is received from the streaming device (e.g., first device 102). The compressed updated audio data (e.g., encoded audio data 229) represents a sound field (e.g., sound field 126) translated based on the translation data (e.g., data 166 or translation metadata 478). The compressed updated audio data (e.g., encoded audio data 229) is decompressed to generate updated audio data (e.g., audio data 227), and the updated audio data is adjusted to rotate the sound field (e.g., sound field 226) based on azimuth. In some implementations, a first delay associated with sending translation data (e.g., data 166 or translation metadata 478) to a streaming device (e.g., first device 102) and receiving compressed updated audio data (e.g., encoded audio data 229) from the streaming device is greater than a second delay associated with adjusting the updated audio data (e.g., audio data 227) based on orientation to rotate the sound field (e.g., sound field 226).

[0309] In some implementations, updated audio data (e.g., audio data 227) is adjusted based on changes in the translation of a device (e.g., second device 202) to translate the sound field (e.g., sound field 226), and adjusting the updated audio data based on changes in translation is limited to translating the sound field forward, backward, left, or right, such as a reference. Figures 3A-4B , Figure 6 and Figure 7 The described 3DOF+ effect. In other implementations, the sound field (e.g., sound field 226) represented by decompressed audio data (e.g., audio data 227) is independent of the movement of the device (e.g., second device 202), and changing the sound field includes translating the sound field in response to data indicating a change in translation (e.g., sensor data 246), and rotating the sound field in response to data indicating a change in orientation (e.g., sensor data 246), such as reference reference. Figure 5A and 5B The 6DOF scene displacement is described.

[0310] By adjusting the decompressed audio data (e.g., audio data 227) to alter the sound field (e.g., sound field 226) based on the movement of the device (e.g., second device 202), the latency associated with sending head tracking data (e.g., data 166 or translation metadata 478) to a remote source device (e.g., streaming source device) is reduced. As a result, the user experience is improved.

[0311] Figure 21A second example of a method for adjusting a sound field is shown. As an illustrative and non-limiting example, method 2100 may be performed by an electronic device such as first device 102, streaming device 402, streaming device 602, or streaming device 702.

[0312] Method 2100 includes receiving sound information from an audio source at 2102. For example, the sound information may correspond to sound information 123, streaming audio content 414, or head-tracking audio portion 614.

[0313] Method 2100 includes receiving pan data from a playback device at 2104, the pan data corresponding to a pan associated with the playback device. For example, the pan data may correspond to data 166, pan metadata 478, pan data 656, or metadata 766.

[0314] Method 2100 includes converting sound information into audio data representing a sound field at 2106 based on translation. For example, Figure 2 The sound field representation generator 124 converts the sound information 123 into audio data 127 representing the sound field 126 based on the translation indicated by the data 166.

[0315] Method 2100 includes transmitting audio data as streaming data to a playback device via wireless transmission at point 2108. For example, Figure 2 The encoder 128 encodes the audio data 127 to generate encoded audio data 129, and the transceiver 130 transmits the encoded audio data 129 as streaming data.

[0316] Based on translation (e.g., indicated by data 166), sound information (e.g., sound information 123) is converted into audio data (e.g., audio data 127) representing a sound field (e.g., sound field 126) and this audio data is sent to the playback device (and not as binaural data). This offloads more intensive translation calculations from the playback device (e.g., the second device 202), enabling the playback device to reduce power consumption and cost, while enabling the playback device to perform less computationally intensive rotation processing and binauralization, thereby reducing the latency experienced by the user in response to changes in the orientation of the sound field (e.g., sound field 226) and improving the user experience.

[0317] Figure 22 A third example of a method 2200 for adjusting a sound field is shown. As an illustrative and non-limiting example, method 2200 may be performed by an electronic device such as second device 202 or wearable device 604.

[0318] Method 2200 includes acquiring, at 2202, data associated with the location and orientation of the tracking device at multiple time instances. For example, this data may correspond to data 166, sensor data 246, head tracker data 648, metadata 652, timestamped user location data 656, user location and timestamped data 766, or any combination thereof.

[0319] Method 2200 includes transmitting the data to a remote device via wireless transmission at 2204. For example, the data may be transmitted to a first device 102, a streaming device 602, or a wearable companion device 706.

[0320] Method 2200 includes receiving compressed audio data representing a sound field from a remote device via wireless transmission at 2206. In one example, the compressed audio data includes panoramic sound data. For example, the compressed audio data may correspond to encoded audio data 229.

[0321] Method 2200 includes decompressing compressed audio data representing a sound field at 2208, and adjusting the decompressed audio data at 2210 based on the orientation associated with the device to alter the sound field. For illustration, the adjustment of the decompressed audio data can be performed at a sound field adjuster 224 or at a 3DOF / 3DOF+ rotation and binauralization operation 364 in the immersive sound field. In some examples, the adjustment of the decompressed audio data is based on data associated with the position and orientation of the device, applied and tracked in relation to its movement. In some implementations, method 2200 also includes adjusting the decompressed audio data based on the difference between the device's position and the position associated with the sound field to translate the sound field, wherein the adjustment of the decompressed audio data based on this difference is limited to a translation of the sound field forward, backward, left, or right, such as a 3DOF+ effect.

[0322] Method 2200 includes outputting the adjusted decompressed audio data to two or more speakers at 2212.

[0323] In some implementations, method 2200 further includes receiving head-locked audio data from a remote device via wireless transmission, and combining the head-locked audio data with adjusted decompressed audio data for output to two or more speakers, such as those described in reference combiner 638. The adjusted decompressed audio data corresponds to posture-adjusted binaural audio (e.g., posture-adjusted binaural audio data 636), and the head-locked audio data (e.g., head-locked binaural headphone audio stream 632) corresponds to posture-independent binaural audio.

[0324] In some implementations, method 2200 includes receiving and pre-buffering sound effects data, such as pre-buffered user interaction sound data 643, via wireless transmission. In response to an instruction from the user to interact with a virtual object associated with the sound effects data, a portion of the pre-buffered sound effects data corresponding to that virtual object (e.g., rendered as user interaction sound 635) is retrieved and combined with adjusted, decompressed audio data (e.g., posture-adjusted binaural audio data 636) for output to two or more speakers.

[0325] In some implementations, method 2200 includes sending an indication of the immersive sound level to a remote device, and receiving updated audio data having the immersive sound level in response to sending the indication. For example, the indication of the immersive sound level may correspond to a request 654 for a specific immersive sound level.

[0326] Figure 23 A fourth example of a method for adjusting a sound field is shown. As an illustrative and non-limiting example, method 2300 may be performed by an electronic device such as first device 102, streaming device 402, or streaming device 602.

[0327] Method 2300 includes receiving sound information from an audio source at 2302. For example, the audio source may correspond to audio source 122 or game audio engine 610.

[0328] Method 2300 includes receiving, at 2304, data from a playback device (e.g., a second device 202, wearable device 404, or wearable device 604) corresponding to locations associated with the playback device at multiple time instances. For example, this data may correspond to data 166, sensor data 246, translation metadata 478, head tracker data 648, metadata 652, timestamped user location data 656, user location and timestamped data 766, or any combination thereof.

[0329] Method 2300 includes converting sound information into audio data representing a sound field at 2306 based on data corresponding to a location associated with the playback device. For example, the sound information may be converted via a sound field representation generator 124, a render / convert to atmos operation 416, or a render / convert to HOA operation 616.

[0330] Method 2300 includes transmitting audio data as streaming data at 2308 to one or both of a playback device (e.g., second device 202, wearable device 404, or wearable device 604) or a second device coupled to the playback device (e.g., wearable companion device 706) via wireless transmission.

[0331] Figure 24A fifth example of a method 2400 for adjusting a sound field is shown. Method 2400 can be performed by an electronic device such as wearable accessory 706.

[0332] Method 2400 includes receiving compressed audio data representing a sound field from a streaming device at 2402. For example, the compressed audio data may correspond to... Figure 1 The encoded audio data 129 or a compressed version of the output immersive sound data 626 generated during the encoding portion of the audio decoding operation 640.

[0333] Method 2400 includes receiving at 2404 data from a playback device (e.g., a second device 202 or a wearable device 604) corresponding to locations associated with the playback device at multiple time instances. For example, this data may correspond to user location and timestamp data 766.

[0334] Method 2400 may include at 2406 generating a predicted location of the device based on data corresponding to the location associated with the playback device. The predicted location indicates a prediction of where the playback device will be when audio data is played on the playback device (e.g., wearable device 604).

[0335] Method 2400 includes decompressing the compressed audio data at 2408. For example, the decompression of the audio data can be performed via the decoding portion of audio decoding operation 640.

[0336] Method 2400 includes adjusting the decompressed audio data at 2410 based on the predicted location to translate the sound field. For example, adjusting the decompressed audio data can be performed via a 360-degree surround sound field conversion operation 768.

[0337] Method 2400 includes compressing the adjusted audio data at 2412 (e.g., in the encoding portion of audio decoding operation 740) and transmitting the compressed adjusted audio data as streaming data to a playback device via wireless transmission at 2414. For example, wearable accessory 706 generates compressed adjusted audio data by compressing the adjusted audio data 770 and transmits the compressed adjusted audio data as streaming data to wearable device 604 via wireless transmission 750.

[0338] Figure 25 A sixth example of a method for adjusting a sound field is shown. As an illustrative and non-limiting example, method 2500 can be performed by an electronic device such as second device 202 or device 804.

[0339] Method 2500 includes receiving, at 2502, compressed audio data corresponding to a first representation of a sound field from a streaming device via wireless transmission, at one or more processors of the device, the first representation corresponding to a first viewport field of view associated with a first orientation of the device. For example, device 804 may receive an audio stream 816 corresponding to a first immersive sound representation 822, the first immersive sound representation 822 corresponding to a first viewport field of view 841 associated with a first orientation of device 804.

[0340] Method 2500 includes decompressing compressed audio data at 2504 and outputting the decompressed audio data to two or more speakers at 2506. For example, an audio decoder and binauralizer 808 may decompress a first portion of the audio stream 816 and output the audio data to speaker 834.

[0341] Method 2500 includes sending data associated with a second orientation of the device to the streaming device at 2508. For example, device 804 sends an audio stream request 820 to source device 802 (e.g., indicating the second orientation of device 804).

[0342] Method 2500 includes receiving compressed, updated audio data from a streaming device at 2510, the compressed, updated audio data corresponding to a second representation of a sound field, and the second representation corresponding to a second viewport field of view that partially overlaps with a first viewport field of view and is associated with a second pose. For example, device 804 may receive a second portion of an audio stream 816 corresponding to a second immersive sound representation 824, which corresponds to a second viewport field of view 842 associated with a second pose of device 804.

[0343] Method 2500 includes decompressing the compressed, updated audio data at 2512 and outputting the decompressed, updated audio data to two or more speakers at 2514. For example, an audio decoder and binauralizer 808 may decompress a second portion of the audio stream 816 and output the decompressed audio data to speaker 834.

[0344] Figure 26 A seventh example of a method for adjusting a sound field is shown. As an illustrative and non-limiting example, method 2600 can be performed by an electronic device such as first device 102 or source device 802.

[0345] Method 2600 includes receiving data associated with the attitude of the playback device from one or more processors of the streaming device at 2602 and via wireless transmission. For example, source device 802 receives audio stream request 820.

[0346] Method 2600 includes, at 2604, selecting a specific representation of a sound field from multiple representations of the sound field based on the data, each corresponding representation of the sound field corresponding to a different viewport field of view in a set of multiple overlapping viewport fields of view. For example, source device 802 selects a first panoramic sound representation 822 from panoramic sound representations 822-828 corresponding to overlapping viewport fields of view (e.g., VFOV 1-VFOV 8) based on an audio stream request 820 indicating an attitude corresponding to a first viewport field of view 841.

[0347] Method 2600 includes generating compressed audio data corresponding to a selected representation of a sound field at 2606, and transmitting the compressed audio data as streaming data to a playback device via wireless transmission at 2608. For example, source device 802 transmits an audio stream 816 corresponding to a first immersive sound representation 822 to device 804.

[0348] Figure 27 An eighth example of a method for adjusting a sound field is shown. As an illustrative and non-limiting example, method 2700 can be performed by an electronic device such as second device 202 or wearable device 1004.

[0349] Method 2700 includes receiving, at 2702, encoded panoramic audio data representing a sound field from a streaming device via wireless transmission at one or more processors of the device. For example, wearable device 1004 receives encoded panoramic audio data 1018 from streaming device 1002.

[0350] Method 2700 includes performing decoding of encoded panoramic audio data at 2704 to generate decoded panoramic audio data. Decoding of the encoded panoramic audio data includes performing basic layer decoding of the base layer of the encoded panoramic audio data and selectively performing enhancement layer decoding based on the detection that the encoded panoramic audio data includes at least one encoded enhancement layer. For example, when an SOA frame is received, wearable device 1004 uses a base layer decoder 1040 for FOA frames and selectively uses a first enhancement layer decoder 1042 to perform panoramic audio decoding operation 1020.

[0351] Method 2700 further includes, at 2706, adjusting the decoded immersive audio data to alter the sound field based on data associated with at least one of translation or orientation related to movement of the device, and at 2708, outputting the adjusted decoded immersive audio data to two or more speakers for playback. For example, wearable device 1004 performs immersive sound field 3DOF / 3DOF+ rotation and binauralization, and provides attitude-adjusted binaural audio signals 1026, 1028 to speakers 1030, 1032.

[0352] Figure 28 A ninth example of a method for adjusting a sound field is shown. As an illustrative and non-limiting example, method 2800 can be performed by an electronic device such as the first device 102 or the streaming device 1002.

[0353] Method 2800 includes receiving, at 2802, first data associated with a first posture of the playback device via wireless transmission, the first posture being associated with a first number of sound sources in the sound scene. For example, streaming device 1002 may receive at least a portion of head tracker data 1036.

[0354] Method 2800 includes generating a first frame of encoded panoramic audio data corresponding to a base layer encoding of a sound scene at 2804, and transmitting the first frame to a playback device at 2806. For example, streaming device 1002 generates a frame corresponding to base level encoding, such as a first-order panoramic frame 1054, and transmits the frame to wearable device 1004 via wireless transmission 1006.

[0355] Method 2800 includes receiving, at 2808, second data associated with a second orientation of the playback device via wireless transmission, the second orientation being associated with a second number of sound sources in the sound scene, and the second number being greater than a first number. Method 2800 includes generating, at 2810, a second frame of encoded panoramic audio data corresponding to an enhancement layer encoding of the sound scene, and transmitting the second frame to the playback device at 2812. For example, frames 1051-1054 may correspond to a wearable device 1004 on a user's head and be oriented towards a first viewport field of view 841 of FIG. 8 having a relatively small number of sound sources. In response to streaming device 1002 receiving data instructing user head movement to change the orientation of wearable device 1004 to another viewport field of view (e.g., a second viewport field of view 842) that includes a greater number of audio sources than the first viewport field of view 841, streaming device 1002 generates a subsequent frame 1055 corresponding to the enhancement layer encoding to achieve higher resolution, thereby accommodating the greater number of sound sources.

[0356] Figure 29 This is a flowchart illustrating a specific example of a method 2900 for processing audio data. Depending on a particular aspect, as an illustrative and non-limiting example, method 2900 may be initiated, executed, or controlled by an electronic device, such as... Figure 2 First equipment 102 Figure 3B Flow cytometer 302 Figure 4B Flow cytometer 402 or Figure 5B The flow cytometer 502.

[0357] Method 2900 includes acquiring sound information from an audio source at 2902. For example, streaming device 302 receives panoramic sound data 312 from audio source 310. Depending on a particular aspect, the sound information includes panoramic sound data and corresponds to at least one of 2D audio data representing a 2D sound field or 3D audio data representing a 3D sound field.

[0358] Method 2900 also includes selecting a compression mode or a bypass mode at 2904 based on a latency criterion associated with the playback device. In compression mode, the representation of the audio information is compressed before transmission to the playback device, and in bypass mode, the representation of the audio information is not compressed before transmission to the playback device. For example, encoding operation 380 includes selecting between compression mode 330 and bypass mode 329 based on latency criterion 331.

[0359] In some implementations, the latency criterion is based on whether the playback latency associated with the streaming data exceeds a latency threshold. In such an implementation, method 2900 may further include receiving an indication from the playback device that the playback latency associated with the streaming data exceeds the latency threshold, and selecting a bypass mode based on receiving the indication. For illustration, streaming device 302 receives an indication 333 from playback device 304 that the playback latency associated with the streaming data exceeds a latency threshold 332, and streaming device 302 selects a bypass mode 329 based on receiving the indication 333.

[0360] In a particular implementation, the latency standard is based on the bandwidth of the wireless link to the playback device, such as as described with reference to latency standard 331, which is at least in part based on the bandwidth of the wireless link associated with the wireless transmission 350 from the streaming device 302 to the playback device 304.

[0361] Method 2900 further includes generating audio data at 2906, which includes a compressed representation of the sound information or an uncompressed representation of the sound information based on a selected compression mode or bypass mode. For example, audio data 382 output from encoding operation 380 may include compressed immersive coefficients from compressed encoding 324 or uncompressed immersive coefficients from bypass operation 326.

[0362] In some implementations, in bypass mode, generating audio data includes discarding a high-resolution portion of the uncompressed representation based on the bandwidth of the wireless link to the playback device. In such implementations, for example, the uncompressed representation includes atmos coefficients, and the high-resolution portion of the uncompressed representation corresponds to a subset of the atmos coefficients, as described in reference truncation operation 327.

[0363] Method 2900 also includes transmitting the audio data as streaming data to the playback device via wireless transmission at 2908.

[0364] In some implementations, method 2900 includes determining whether the wireless link to the playback device corresponds to a higher bandwidth wireless link (e.g., a 5G cellular digital network or a WiFi-type network) or a lower bandwidth wireless link (e.g., a Bluetooth network). In such implementations, method 2900 may include selecting a bypass mode based on whether the wireless link corresponds to a higher bandwidth wireless link. Alternatively, in such implementations, method 2900 includes selecting a compression mode based on whether the wireless link corresponds to a lower bandwidth wireless link.

[0365] In some implementations, method 2900 includes receiving a request from the playback device for compressed or uncompressed audio data. In such implementations, method 2900 may also include selecting a bypass mode or a compression mode based on the request.

[0366] In some implementations, method 2900 also includes receiving translation data from the playback device. For example, the translation data may correspond to a translation associated with the playback device. In such an implementation, method 2900 may also include converting sound information into audio data representing a sound field based on the translation.

[0367] In some implementations, method 2900 includes receiving data from the playback device corresponding to a position and orientation associated with movement of the playback device. In such implementations, method 2900 also includes updating sound information based on the received data to alter the sound field. In some examples of this implementation, method 2900 further includes transmitting compressed audio data representing the sound field to the playback device via wireless transmission. The compressed audio data representing the sound field enables the playback device to decompress the compressed audio data representing the sound field, adjust the decompressed audio data based on the orientation associated with the device to alter the sound field, and output the adjusted decompressed audio data to two or more speakers. In other examples of this implementation, method 2900 includes transmitting uncompressed audio data representing the sound field to the playback device via wireless transmission. The uncompressed audio data representing the sound field enables the playback device to adjust the audio data based on the orientation associated with the device to alter the sound field, and output the adjusted audio data to two or more speakers.

[0368] Figure 30 This is a flowchart illustrating a specific example of a method 3000 for processing audio data. Depending on a particular aspect, as an illustrative and non-limiting example, method 3000 may be initiated, executed, or controlled by an electronic device, such as... Figure 2 The first device 102 or Figure 8B or Figure 8C The source device 802.

[0369] Method 3000 includes receiving, at 3002, data associated with the attitude of the playback device from the playback device via wireless transmission. For example, source device 802 may receive attitude data 871 via audio stream request 820.

[0370] Method 3000 also includes selecting a specific representation of the sound field from multiple representations of the sound field based on the data at 3004. For example, source device 802 selects one of immersive sound representations 862-868 or one of stereo representations 872-878 based on attitude data 871. Each corresponding representation of the sound field corresponds to a different sector in a set of sectors. The sector representation represents a range of values ​​associated with movement of the playback device.

[0371] Method 3000 also includes generating audio data corresponding to a selected representation of the sound field at 3006, and transmitting the audio data as streaming data to a playback device via wireless transmission at 3008. For example, source device 802 generates and transmits audio stream 816 based on the selected representation of the sound field.

[0372] Figure 31 This is a flowchart illustrating a specific example of a method 3100 for processing audio data. Depending on the specific aspect, as an illustrative and non-limiting example, method 3100 may be initiated, executed, or controlled by an electronic device, such as... Figure 2 The second device 202 or Figure 10B Wearable devices 1004.

[0373] Method 3100 includes receiving, at 3102, encoded panoramic audio data representing a sound field from a streaming device via wireless transmission. For example, wearable device 1004 receives encoded panoramic audio data 1018 from streaming device 1002.

[0374] The method 3100 further includes performing decoding of the panoramic sound audio data at 3104 to generate decoded panoramic sound audio data. For example, wearable device 1004 performs panoramic sound audio decoding operation 1020 to generate decoded panoramic sound audio data 1022. Decoding of the panoramic sound audio data includes base layer decoding of the encoded panoramic sound audio data base layer, and selectively includes enhancement layer decoding in response to the amount of movement of the device. For example, motion-based resolution selection 1070 generates signals 1080-1086 to control the operation of panoramic sound audio decoding operation 1020 and the operation of base layer decoder 1040, first enhancement layer decoder 1042 and second enhancement layer decoder 1044 based on the amount of movement 1072.

[0375] Method 3100 further includes, at 3106, adjusting the decoded panoramic audio data to alter the sound field based on data associated with at least one of translation or orientation related to movement of the device, and at 3108, outputting the adjusted decoded panoramic audio data to two or more speakers for playback. For example, wearable device 1004 performs panoramic sound field 3DOF / DOF+ rotation and binaural operation 1024 to provide posture-adjusted binaural audio signals 1026, 1028 to speakers 1030, 1032 based on head tracker data 1036 from one or more sensors 1034.

[0376] refer to Figure 32 A block diagram depicting a specific illustrative implementation of the device is shown, and is generally represented as 3200. In various implementations, device 3200 may have more than Figure 32 The number of components may be more or less. In an illustrative implementation, device 3200 may correspond to either first device 102 or second device 202. In an illustrative implementation, device 3200 may perform reference... Figures 2-31 One or more operations described.

[0377] In a particular implementation, device 3200 includes processor 3206 (e.g., central processing unit (CPU)). Device 3200 may include one or more additional processors 3210 (e.g., one or more DSPs). In a particular implementation, Figure 2 The processor 220 corresponds to processor 3206, processor 3210, or a combination thereof. For example, processor 3210 may include a speech and music encoder-decoder (CODEC) 3208, a renderer 222, a sound field adjuster 224, a decoder 228, or a combination thereof. In a particular implementation, Figure 2 The processor 120 corresponds to processor 3206, processor 3210, or a combination thereof. For example, processor 3210 may include a speech and music encoder-decoder (CODEC) 3208, a sound field representation generator 124, an encoder 128, or a combination thereof. Speech and music decoder 3208 may include a speech encoder (“vocoder”) encoder 3236, a vocoder decoder 3238, or both.

[0378] Device 3200 may include memory 3286 and decoder 3234. Memory 3286 may include instructions 3256, which may be executed by one or more additional processors 3210 (or processor 3206) to implement the functions described for reference renderer 222, sound field adjuster 224, decoder 228, or any combination thereof. Device 3200 may include modem 3240 coupled to antenna 3252 via transceiver 3250. Transceiver 3250 may correspond to... Figure 2The transceiver 230. In an implementation of device 3200 corresponding to a transmitting device, modem 3240 can be configured to modulate audio data for transmission to a playback device, and antenna 3252 can be configured to transmit modulated audio data to the playback device. In an implementation of device 3200 corresponding to a playback device, antenna 3252 can be configured to receive modulated transmission data representing encoded audio data, and modem 3240 can be configured to demodulate the received modulated transmission data to generate encoded audio data.

[0379] Device 3200 may include a display 3228 coupled to display controller 3226. Multiple speakers 3292 (e.g., speakers 240, 242) and one or more microphones (such as microphone 3294) may be coupled to decoder 3234. Decoder 3234 may include digital-to-analog converter (DAC) 3202 and analog-to-digital converter (ADC) 3204. In a particular implementation, decoder 3234 may receive analog signals from microphone 3294, convert the analog signals to digital signals using ADC 3204, and transmit the digital signals to voice and music decoder 3208. In a particular implementation, voice and music decoder 3208, renderer 222, or both may provide digital signals to decoder 3234. Decoder 3234 may use ADC 3202 to convert digital signals to analog signals and may provide analog signals to speakers 3292.

[0380] In a particular implementation, device 3200 may be included in a system-on-a-package or system-on-a-chip device 3222. In a particular implementation, memory 3286, processor 3206, processor 3210, display controller 3226, decoder 3234, and modem 3240 are included in the system-on-a-chip device 3222. In a particular implementation, input device 3230 (e.g., one or more sensors 244) and power supply 3244 are coupled to system-on-a-chip device 3222. Furthermore, in a particular implementation, such as... Figure 32 As shown, the display 3228, input device 3230, speaker 3292, microphone 3294, antenna 3252, and power supply 3244 are located external to the system-on-chip device 3222. In a particular implementation, each of the display 3228, input device 3230, speaker 3292, microphone 3294, antenna 3252, and power supply 3244 may be coupled to components of the system-on-chip device 3222 (such as an interface or controller).

[0381] Device 3200 may include smart speakers, speakers, mobile communication devices, smartphones, cellular phones, laptops, computers, tablets, personal digital assistants, display devices, televisions, game consoles, music players, radios, digital video players, digital video disc (DVD) players, tuners, cameras, navigation devices, vehicles, headsets, augmented reality headsets, mixed reality headsets, virtual reality headsets, aircraft, home automation systems, voice-activated devices, wireless speakers and voice-activated devices, portable electronic devices, automobiles, vehicles, computing devices, communication devices, Internet of Things (IoT) devices, virtual reality (VR) devices, base stations, mobile devices, or any combination thereof.

[0382] In conjunction with the described technology, the first device includes components for receiving compressed audio data representing a sound field via wireless transmission at one or more processors, such as transceiver 230, receiver 234, decoder 228, one or more processors 220, processor 3206, processor 3210, transceiver 3250, or combinations thereof.

[0383] The first device includes components for decompressing compressed audio data, such as a decoder 228, one or more processors 220, 3206, 3210, one or more processors or combinations thereof performing immersive audio decoding operation 360.

[0384] The first device includes components for adjusting decompressed audio data to change the sound field based on data associated with at least one of translation or orientation related to movement of the device, such as a sound field adjuster 224, one or more processors 220, 3206, 3210, one or more processors performing panoramic sound field 3DOF / 3DOF+ rotation and binaural operation 364, one or more processors performing panoramic sound field 6DOF scene displacement and binaural operation 564, or combinations thereof.

[0385] The first device includes components for rendering adjusted decompressed audio data to two or more speaker gains to drive two or more speakers, such as renderer 222, one or more processors 220, processor 3206, processor 3210, one or more processors performing 3DOF / 3DOF+ rotation and binauralization operations 364, one or more processors performing 6DOF scene displacement and binauralization operations 564, or combinations thereof.

[0386] The first device includes components for outputting adjusted, decompressed audio data to two or more speakers for playback, such as a renderer 222, one or more processors 220, processor 3206, processor 3210, one or more processors performing 3DOF / 3DOF+ rotation and binauralization operations 364, one or more processors performing 6DOF scene displacement and binauralization operations 564, or a combiner 638, or a combination thereof.

[0387] In conjunction with the described technology, the second device includes components for receiving sound information from an audio source, such as a sound field representation generator 124, one or more processors 220, 3206, 3210, one or more processors performing rendering / conversion to Atmos operation 416, one or more processors performing rendering / conversion to HOA operation 616, or a combination thereof.

[0388] The second device includes components for receiving translation data from a playback device, such as transceiver 130, receiver 134, one or more processors 120, one or more processors 220, processor 3206, processor 3210, transceiver 3250, or a combination thereof, the translation data corresponding to a translation associated with the playback device.

[0389] The second device includes components for converting sound information into audio data representing a sound field based on the translation, such as a sound field representation generator 124, one or more processors 220, 3206, 3210, one or more processors performing rendering / conversion to Atmos operation 416, one or more processors performing rendering / conversion to HOA operation 616, or a combination thereof.

[0390] The second device includes components for transmitting audio data as streaming data to a playback device via wireless transmission, such as transceiver 130, transmitter 132, one or more processors 120, one or more processors 220, processor 3206, processor 3210, one or more processors or devices performing wireless transmission 350, one or more processors or devices performing audio decoding 640 or wireless transmission 650, transceiver 3250, or a combination thereof.

[0391] In conjunction with the described technology, the third device includes components for acquiring data associated with the location and orientation of the tracking device at multiple time instances, such as one or more sensors 244, one or more processors 220, and one or more sensors 344. Figure 6 or Figure 7The wearable device 604 is configured to receive head tracker data 648 from one or more processors, processor 3206, processor 3210, or a combination thereof.

[0392] The third device includes components for transmitting the data to a remote device via wireless transmission, such as transceiver 230, transmitter 232, and one or more processors 220. Figure 6 The wearable device 604 is configured to perform wireless transmission via one or more processors 653. Figure 7 The wearable device 604 is configured to perform wireless transmissions by one or more processors 780, processor 3206, processor 3210, transceiver 3250, or a combination thereof.

[0393] The third device includes components for receiving compressed audio data representing a sound field from a remote device via wireless transmission, such as a transceiver 230, a receiver 234, and one or more processors 220. Figure 6 One or more processors of the wearable device 604 Figure 7 The wearable device 604 includes one or more processors, processor 3206, processor 3210, transceiver 3250, or combinations thereof.

[0394] The third device includes components for decompressing compressed audio data representing a sound field, such as one or more processors 220, decoders 228, etc. Figure 6 The wearable device 604 is configured to include one or more processors in the decoding section of the audio decoding operation 640. Figure 7 The wearable device 604 is configured to perform audio decoding operations 740 by one or more processors, processor 3206, processor 3210, or combinations thereof.

[0395] The third device includes components for adjusting decompressed audio data to change the sound field based on the orientation associated with the device, such as a sound field adjuster 224, one or more processors 220, etc. Figure 6 or Figure 7 The wearable device 604 is configured to perform panoramic sound field 3DOF / 3DOF+ rotation and binaural operation 364 by one or more processors, processor 3206, processor 3210, or a combination thereof.

[0396] The third device includes components for outputting adjusted decompressed audio data to two or more speakers, such as one or more processors 220, renderer 222, combiner 638, processor 3206, processor 3210, or combinations thereof.

[0397] In conjunction with the described technology, the fourth device includes components for receiving sound information from an audio source, such as a sound field representation generator 124, one or more processors 120, etc. Figure 6 One or more processors of the wearable device 604 Figure 7 The wearable device 604 may have one or more processors, one or more processors that perform rendering / conversion to HOA operations 616, processor 3206, processor 3210, transceiver 3250, or a combination thereof.

[0398] The fourth device includes components for receiving data from the playback device corresponding to positions associated with the playback device at multiple time instances, such as transceiver 130, receiver 134, and one or more processors 120. Figure 6 The wearable device 604 is configured to receive wireless transmissions from one or more processors of 653. Figure 7 The wearable device 604 is configured to receive wireless transmissions from one or more processors, processor 3206, processor 3210, transceiver 3250, or combinations thereof.

[0399] The fourth device includes components for converting sound information into audio data representing a sound field based on data corresponding to a location associated with the playback device, such as a sound field representation generator 124, one or more processors, processor 3206, processor 3210, or a combination thereof, performing rendering / conversion to HOA operation 616.

[0400] The fourth device includes components for transmitting audio data as streaming data to one or both of a playback device or a second device coupled to the playback device via wireless transmission, such as transceiver 130, transmitter 132, one or more processors 120, one or more processors or devices performing audio decoding operation 640 or wireless transmission 650, processor 3206, processor 3210, transceiver 3250, or a combination thereof.

[0401] In conjunction with the described technology, the fifth device includes components for receiving compressed audio data representing a sound field from a streaming device, such as one or more processors 760, 3206, 3210, or combinations thereof, performing a decoding portion of audio decoding operation 640 or a receiving operation of wireless transmission 650.

[0402] The fifth device includes components for receiving data from the playback device corresponding to the location associated with the playback device at multiple time instances, such as one or more processors 760, 3206, 3210, or combinations thereof that perform the receiving operation of the wireless transmission 780.

[0403] The fifth device includes components for generating a predicted location of the device based on data corresponding to the location associated with the playback device, such as one or more processors 760, 3206, 3210, or a combination thereof that perform a location prediction operation based on user location and timestamp data 766, the predicted location indicating a prediction of where the playback device will be when audio data is played at the playback device.

[0404] The fifth device includes components for decompressing compressed audio data, such as one or more processors 760, 3206, 3210, or combinations thereof, which perform the decoding portion of the audio decoding operation 640.

[0405] The fifth device includes components for adjusting decompressed audio data to translate the sound field based on the predicted position, such as one or more processors 760, 3206, 3210, or combinations thereof that perform the immersive sound field translation 768.

[0406] The fifth device includes components for compressing the adjusted audio data, such as one or more processors 760, 3206, 3210, or combinations thereof that perform the encoding portion of the audio decoding operation 740.

[0407] The fifth device includes components for transmitting compressed, adjusted audio data as streaming data to a playback device via wireless transmission, such as one or more processors 760, 3206, 3210, or combinations thereof that perform the transmission operation of wireless transmission 750.

[0408] In conjunction with the described technology, the sixth device includes components (e.g., transceiver 230, receiver 234, one or more processors 220, second device 202, device 804, streaming client 806, processor 3206, processor 3210, or a combination thereof) for receiving compressed audio data corresponding to a first representation of a sound field from a streaming device via wireless transmission, the first representation corresponding to a first viewport field of view associated with a first orientation of the device.

[0409] The sixth device includes components for decompressing compressed audio data, such as one or more processors 220, decoder 228, second device 202, streaming client 806, audio decoder and binauralizer 808, processor 3206, processor 3210, or combinations thereof.

[0410] The sixth device includes components for outputting decompressed audio data to two or more speakers, such as one or more processors 220, sound field adjuster 224, renderer 222, second device 202, audio decoder and binauralizer 808, processor 3206, processor 3210, or combinations thereof.

[0411] The sixth device includes components for transmitting data associated with a second attitude of the streaming device to the streaming device, such as one or more processors 220, transceiver 230, transmitter 232, second device 202, streaming client 806, processor 3206, processor 3210, or combinations thereof.

[0412] The sixth device includes components (e.g., transceiver 230, receiver 234, one or more processors 220, second device 202, device 804, streaming client 806, processor 3206, processor 3210, or a combination thereof) for receiving compressed, updated audio data from a streaming device, the compressed, updated audio data corresponding to a second representation of a sound field, and the second representation corresponding to a second viewport field of view that partially overlaps with the first viewport field of view and is associated with a second attitude.

[0413] The sixth device includes components for decompressing compressed, updated audio data, such as one or more processors 220, decoder 228, second device 202, streaming client 806, audio decoder and binauralizer 808, processor 3206, processor 3210, or combinations thereof.

[0414] The sixth device includes components for outputting decompressed, updated audio data to two or more speakers, such as one or more processors 220, sound field adjuster 224, renderer 222, second device 202, audio decoder and binauralizer 808, processor 3206, processor 3210, or combinations thereof.

[0415] In conjunction with the described technology, the seventh device includes components for receiving data associated with the attitude of the playback device via wireless transmission, such as a first device 102, a transceiver 130, a receiver 134, one or more processors 832, a source device 802, a processor 3206, a processor 3210, or a combination thereof.

[0416] The seventh device includes components (e.g., one or more processors 120, memory 110, first device 102, one or more processors 832, memory 830, processor 3206, processor 3210, or combinations thereof) for selecting a specific representation of the sound field from multiple representations of the sound field based on the data, each corresponding representation of the sound field corresponding to a different viewport field in a set of multiple overlapping viewport fields of view.

[0417] The seventh device includes components for generating compressed audio data corresponding to a selected representation of a sound field, such as one or more processors 120, a sound field representation generator 124, an encoder 128, a first device 102, one or more processors 832, processor 3206, processor 3210, or a combination thereof.

[0418] The seventh device includes components for transmitting compressed audio data as streaming data to a playback device via wireless transmission, such as one or more processors 120, transceiver 130, transmitter 132, first device 102, one or more processors 832, processor 3206, processor 3210, or combinations thereof.

[0419] In conjunction with the described techniques, the eighth device includes components for receiving encoded panoramic audio data representing a sound field from a streaming device via wireless transmission, such as transceiver 230, receiver 234, one or more processors 220, second device 202, device 1004, one or more processors of FIG. 10 configured to receive encoded panoramic audio data 1018, processor 3206, processor 3210, or a combination thereof.

[0420] The eighth device includes components for performing decoding of immersive audio data to generate decoded immersive audio data, such as one or more processors 220, decoder 228, second device 202 or base layer decoder 1040, first enhancement layer decoder 1042, processor 3206, processor 3210, or combinations thereof. Decoding of the encoded immersive audio data includes base layer decoding of the encoded immersive audio data, and selectively includes enhancement layer decoding in response to detection that the encoded immersive audio data includes at least one encoded enhancement layer.

[0421] The eighth device includes components for adjusting decoded immersive audio data to change the sound field based on data associated with at least one of translation or orientation related to movement of the device, such as one or more processors 220, sound field adjuster 224, second device 202, wearable device 1004 performing immersive sound field 3DOF / 3DOF+ rotation and binaural operation 1024, or combinations thereof.

[0422] The eighth device includes components for outputting adjusted, decoded immersive audio data to two or more speakers for playback, such as one or more processors 220, renderer 222, second device 202, wearable device 1004, one or more processors 1024, processor 3206, processor 3210, or combinations thereof for performing immersive sound field 3DOF / 3DOF+ rotation and binaural operation.

[0423] In conjunction with the described technology, the ninth device includes components for receiving, via wireless transmission, first data associated with a first attitude of the playback device, such as a first device 102, a sound field representation generator 124, a transceiver 130, a receiver 134, a streaming device 1002, a processor 3206, a processor 3210, an antenna 3252, a transceiver 3250, a modem 3240, or a combination thereof, the first attitude being associated with a first number of sound sources in a sound scene.

[0424] The ninth device includes components for generating a first frame of encoded panoramic audio data (which corresponds to the base layer encoding of the sound scene), such as one or more processors 120, sound field representation generator 124, memory 110, first device 102, streaming device 1002, processor 3206, processor 3210, or a combination thereof.

[0425] The ninth device includes components for transmitting the first frame to a playback device, such as a first device 102, a sound field representation generator 124, an encoder 128, a transceiver 130, a transmitter 132, a streaming device 1002, a processor 3206, a processor 3210, an antenna 3252, a transceiver 3250, a modem 3240, or a combination thereof.

[0426] The ninth device includes components for receiving second data associated with a second attitude of the playback device via wireless transmission, such as a first device 102, a transceiver 130, a receiver 134, a sound field representation generator 124, a streaming device 1002, a processor 3206, a processor 3210, an antenna 3252, a transceiver 3250, a modem 3240, or a combination thereof, wherein the second attitude is associated with a second number of sound sources in a sound scene, and the second number is greater than the first number.

[0427] The ninth device includes components for generating a second frame of encoded panoramic audio data (which corresponds to the enhancement layer encoding of the sound scene), such as one or more processors 120, memory 110, sound field representation generator 124, first device 102, streaming device 1002, processor 3206, processor 3210, or a combination thereof.

[0428] The ninth device includes components for transmitting the second frame to a playback device, such as a first device 102, a sound field representation generator 124, an encoder 128, a transceiver 130, a transmitter 132, a streaming device 1002, a processor 3206, a processor 3210, an antenna 3252, a transceiver 3250, a modem 3240, or a combination thereof.

[0429] In conjunction with the described technology, the tenth device includes components for acquiring sound information from an audio source, such as a sound field representation generator 124, one or more processors 120, 3206, 3210, one or more processors performing rendering / conversion to Atmos operation 316, one or more processors performing encoding operation 380, one or more processors performing rendering / conversion to HOA operation 616, or a combination thereof.

[0430] The tenth device includes components for selecting a compression mode or a bypass mode based on a latency metric associated with the playback device, such as one or more processors 120, one or more processors performing encoding operation 380, processor 3206, processor 3210, or a combination thereof, in which the representation of audio information is compressed before being transmitted to the playback device in compression mode, and in bypass mode the representation of audio information is not compressed before being transmitted to the playback device.

[0431] The tenth device includes components for generating audio data, which includes a compressed representation or an uncompressed representation of sound information based on a selected compression mode or bypass mode, such as one or more processors 120, one or more processors performing encoding operation 380, processor 3206, processor 3210, or a combination thereof.

[0432] The tenth device includes components for transmitting audio data as streaming data to a playback device via wireless transmission, such as first device 102, transceiver 130, transmitter 132, streaming device 302, streaming device 402, streaming device 502, streaming device 602, processor 3206, processor 3210, antenna 3252, transceiver 3250, modem 3240, or combinations thereof.

[0433] In conjunction with the described technology, the eleventh device includes components for receiving data associated with the attitude of the playback device via wireless transmission, such as one or more processors 120, transceiver 130, receiver 134, source device 802, processor 832, processor 3206, processor 3210, antenna 3252, transceiver 3250, modem 3240, or combinations thereof.

[0434] The eleventh device includes components for selecting a specific representation of the sound field from multiple representations of the sound field based on the data, such as one or more processors 120, one or more processors 832, processor 3206, processor 3210, or combinations thereof, wherein each corresponding representation of the sound field corresponds to a different sector in a set of sectors, and the sector representation is a range of values ​​associated with movement of the playback device.

[0435] The eleventh device includes components for generating audio data corresponding to a selected representation of the sound field, such as one or more processors 120, one or more processors 832, processor 3206, processor 3210, or combinations thereof.

[0436] The eleventh device includes components for transmitting audio data as streaming data to a playback device via wireless transmission, such as a first device 102, a transceiver 130, a transmitter 132, a source device 802, one or more processors 832, 3206, 3210, an antenna 3252, a transceiver 3250, a modem 3240, or a combination thereof.

[0437] In conjunction with the described technology, the twelfth device includes components for receiving encoded panoramic audio data representing a sound field from a streaming device via wireless transmission, such as one or more processors 220, transceiver 230, receiver 234, wearable device 1004, processor 3206, processor 3210, antenna 3252, transceiver 3250, modem 3240, or combinations thereof.

[0438] The twelfth device includes components for performing decoding of immersive audio data to generate decoded immersive audio data, such as one or more processors 220, one or more processors performing immersive audio decoding operation 1020, a base layer decoder 1040, a first enhancement layer decoder 1042, a second enhancement layer decoder 1044, processor 3206, processor 3210, or a combination thereof, wherein decoding of the immersive audio data includes base layer decoding of the encoded immersive audio data and selectively includes enhancement layer decoding in response to a certain amount of device movement.

[0439] The twelfth device includes components for adjusting decoded immersive audio data to change the sound field based on data associated with at least one of translation or orientation related to movement of the device, such as one or more processors 220, one or more processors performing immersive sound field 3DOF / DOF+ rotation and binauralization operation 1024, processor 3206, processor 3210, or a combination thereof.

[0440] The twelfth device includes components for outputting adjusted and decoded immersive audio data to two or more speakers for playback, such as one or more processors 220, renderer 222, one or more processors performing immersive sound field 3DOF / DOF+ rotation and binauralization operations 1024, processor 3206, processor 3210, decoder 3234, or combinations thereof.

[0441] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device such as memory 110, memory 210, memory 762, memory 830, memory 930, or memory 3286) includes instructions (e.g., instructions 112, 212, or 3256) that, when executed by one or more processors (e.g., one or more processors 120, 220, 832, 3206, or 3210), cause one or more processors to perform an action corresponding to a reference. Figures 1-19 or Figure 32 Any technology described Figures 20-31 Any method or any combination thereof, at least a portion of the operation.

[0442] This disclosure includes the following examples.

[0443] Example 1. A device comprising one or more processors configured to receive compressed audio data representing a sound field via wireless transmission; decompress the compressed audio data; adjust the decompressed audio data to alter the sound field based on data associated with at least one of translation or orientation related to movement of the device; render the adjusted decompressed audio data into the gains of two or more speakers to drive two or more speakers; and output the adjusted decompressed audio data to two or more speakers for playback.

[0444] Example 2. The device according to Example 1, wherein one or more processors are configured to perform binauralization of the adjusted decompressed audio data to generate two or more speaker gains.

[0445] Example 3. The device according to Example 1, wherein the decompressed audio data includes panoramic sound data and corresponds to at least one of two-dimensional (2D) audio data or three-dimensional (3D) audio data.

[0446] Example 4. The device according to Example 1, wherein one or more processors are further configured to receive compressed audio data as streaming data from a streaming device, the streaming device corresponding to at least one of a portable electronic device or a server.

[0447] Example 5. The device according to Example 4, wherein one or more processors are further configured to send translation data to the streaming device, the translation data being associated with movement of the device; in response to sending the translation data, receive compressed updated audio data from the streaming device, the compressed updated audio data representing a sound field translated based on the translation data; decompress the compressed updated audio data to generate updated audio data; and adjust the updated audio data based on orientation to rotate the sound field.

[0448] Example 6. According to the device of Example 4, one or more processors are further configured to send translation data to the streaming device, the translation data being associated with movement of the device; and in response to sending the translation data, to receive compressed updated audio data from the streaming device, the compressed updated audio data representing a sound field translated based on the translation data, and wherein a first delay associated with sending the translation data to the streaming device and receiving the compressed updated audio data from the streaming device is greater than a second delay associated with rotating the sound field based on azimuth-adjusted updated audio data.

[0449] Example 7. The device according to Example 4, wherein one or more processors are further configured to send translation data to the streaming device, the translation data being associated with movement of the device; and in response to sending the translation data, receive compressed updated audio data from the streaming device, the compressed updated audio data representing a sound field translated based on the translation data; decompress the compressed updated audio data to generate updated audio data; and adjust the updated audio data to translate the sound field based on a change in the device's translation, wherein the adjustment of the updated audio data based on the change in translation is limited to translating the sound field forward, backward, left, or right.

[0450] Example 8. The device according to Example 1, wherein the sound field represented by decompressed audio data is independent of the movement of the device, and wherein one or more processors are configured to adjust the audio data to translate the sound field in response to data indicating a change in translation.

[0451] Example 9. The device according to Example 1, wherein one or more processors are configured to adjust audio data to rotate the sound field in response to data indicating a change in orientation.

[0452] Example 10. The device according to Example 1, wherein one or more processors are configured to translate and rotate the sound field in response to movement of the device, and not to send translation data associated with the movement of the device to the streaming device.

[0453] Example 11. The device according to Example 1, wherein one or more processors are integrated in the headphone device.

[0454] Example 12. The device according to Example 11, wherein the headphone device further includes a memory configured to store instructions executable by one or more processors; one or more sensors configured to generate the data; a first speaker configured to be placed close to a user's first ear when the user wears the headphone device; a second speaker configured to be placed close to a user's second ear when the user wears the headphone device; and a wireless receiver configured to receive compressed audio data via wireless transmission.

[0455] Example 13. The device according to Example 1, wherein one or more processors are integrated into the vehicle, and wherein the data indicates the translation and orientation of the vehicle.

[0456] Example 14. The device according to Example 1, wherein one or more processors are integrated into the speaker array device and are also configured to perform beam steering operations to direct binaural signals to a location associated with the user.

[0457] Example 15. The device according to Example 1, wherein one or more processors are integrated in at least one of a mobile phone, a tablet computer device, a wearable electronic device, a camera device, a virtual reality headset, a mixed reality headset, or an augmented reality headset.

[0458] Example 16. A method comprising: receiving compressed audio data representing a sound field at one or more processors via wireless transmission; decompressing the compressed audio data; adjusting the decompressed audio data to alter the sound field based on data associated with at least one of translation or orientation related to movement of the device; rendering the adjusted decompressed audio data into the gains of two or more speakers to drive two or more speakers; and outputting the adjusted decompressed audio data to the two or more speakers for playback.

[0459] Example 17. The method according to Example 16 further includes: performing binauralization of the adjusted decompressed audio data to generate two or more speaker gains.

[0460] Example 18. Following the method of Example 16, wherein the decompressed audio data includes panoramic sound data.

[0461] Example 19. The method of Example 16, wherein compressed audio data is received as streaming data from a streaming device, the streaming device corresponding to at least one of a portable electronic device or a server.

[0462] Example 20. The method according to Example 19 further includes: sending translation data to a streaming device, the translation data being associated with movement of the device; receiving compressed updated audio data from the streaming device in response to sending the translation data, the compressed updated audio data representing a sound field translated based on the translation data; decompressing the compressed updated audio data to generate updated audio data; and adjusting the updated audio data based on orientation to rotate the sound field.

[0463] Example 21. According to the method of Example 20, wherein a first delay associated with sending translation data to the streaming device and receiving compressed updated audio data from the streaming device is greater than a second delay associated with adjusting the updated audio data based on orientation to rotate the sound field.

[0464] Example 22. The method according to Example 20 further includes adjusting the updated audio data based on changes in device translation to translate the sound field, wherein adjusting the updated audio data based on changes in translation is limited to translating the sound field forward, backward, left, or right.

[0465] Example 23. According to the method of Example 16, the sound field represented by the decompressed audio data is independent of the movement of the device, and wherein changing the sound field includes translating the sound field in response to data indicating a change in translation; and rotating the sound field in response to data indicating a change in orientation.

[0466] Example 24. A non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause one or more processors to: receive compressed audio data representing a sound field via wireless transmission; decompress the compressed audio data; adjust the decompressed audio data to change the sound field based on data associated with at least one of translation or orientation related to movement of the device; render the adjusted decompressed audio data into the gains of two or more speakers to drive two or more speakers; and output the adjusted decompressed audio data to two or more speakers for playback.

[0467] Example 25. According to the non-transitory computer-readable medium of Example 24, the instructions, when executed by one or more processors, also cause one or more processors to perform binauralization of the adjusted decompressed audio data to generate two or more speaker gains.

[0468] Example 26. A non-transitory computer-readable medium according to Example 24, wherein the instructions, when executed by one or more processors, also cause one or more processors to receive compressed audio data as streaming data from at least one of a portable electronic device or a server.

[0469] Example 27. An apparatus comprising: means for receiving compressed audio data representing a sound field via wireless transmission at one or more processors; means for decompressing the compressed audio data; means for adjusting the decompressed audio data to change the sound field based on data associated with at least one of translation or orientation related to movement of the device; means for rendering the adjusted decompressed audio data into the gains of two or more speakers to drive two or more speakers; and means for outputting the adjusted decompressed audio data to two or more speakers for playback.

[0470] Example 28. A device comprising one or more processors configured to receive sound information from an audio source; receive translation data from a playback device corresponding to a translation associated with the playback device; convert the sound information into audio data representing a sound field based on the translation; and transmit the audio data as streaming data to the playback device via wireless transmission.

[0471] Example 29. The device according to Example 28, wherein the audio data includes panoramic sound data and corresponds to at least one of two-dimensional (2D) audio data representing a 2D sound field or three-dimensional (3D) audio data representing a 3D sound field.

[0472] Example 30. The device according to Example 28, wherein streaming data is transmitted via a fifth-generation (5G) cellular digital network.

[0473] Example 31. The device according to Example 28, wherein the audio source corresponds to a portion of a media file, and wherein the streaming data is associated with a virtual reality experience that is streamed to a playback device via at least one of a fifth-generation (5G) cellular digital network or a Bluetooth network.

[0474] Example 32. A device according to Example 28, wherein one or more processors are integrated in a portable electronic device.

[0475] Example 33. The device according to Example 28, wherein one or more processors are integrated in a server.

[0476] Example 34. A method comprising: receiving sound information from an audio source; receiving translation data from a playback device, the translation data corresponding to a translation associated with the playback device; converting the sound information into audio data representing a sound field based on the translation; and transmitting the audio data as streaming data to the playback device via wireless transmission.

[0477] Example 35. A non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause one or more processors to: receive sound information from an audio source; receive translation data from a playback device, the translation data corresponding to a translation associated with the playback device; convert the sound information into audio data representing a sound field based on the translation; and transmit the audio data as streaming data to the playback device via wireless transmission.

[0478] Example 36. An apparatus includes: means for receiving sound information from an audio source; means for receiving translation data from a playback device, the translation data corresponding to a translation associated with the playback device; means for converting the sound information into audio data representing a sound field based on the translation; and means for transmitting the audio data as streaming data to the playback device via wireless transmission.

[0479] Example 37. A device includes one or more processors configured to acquire data associated with location and orientation related to tracking movement of the device at multiple time instances; transmit the data to a remote device via wireless transmission; receive compressed audio data representing a sound field from the remote device via wireless transmission; decompress the compressed audio data representing the sound field; adjust the decompressed audio data to change the sound field based on the orientation associated with the device; and output the adjusted decompressed audio data to two or more speakers.

[0480] Example 38. The device according to Example 37 further includes a memory configured to store decompressed audio data, wherein one or more processors are configured to adjust the decompressed audio data based on data associated with the location and orientation of the device and the application tracking the movement of the device.

[0481] Example 39. The device according to Example 37, wherein the decompressed audio data includes panoramic sound data, which corresponds to at least one of two-dimensional (2D) audio data representing a 2D sound field or three-dimensional (3D) audio data identifying a 3D sound field.

[0482] Example 40. A device according to Example 37, wherein one or more processors are configured to further adjust decompressed audio data to translate the sound field based on the difference between the device’s location and the location associated with the sound field, wherein the adjustment of the decompressed audio data based on the difference is limited to a translation of the sound field forward, backward, left, or right.

[0483] Example 41. The device according to Example 37, wherein one or more processors are further configured to receive head-locked audio data via wireless transmission; and to combine the head-locked audio data with adjusted decompressed audio data for output to two or more speakers.

[0484] Example 42. The device according to Example 41, wherein the adjusted decompressed audio data corresponds to posture-adjusted binaural audio, and wherein the head-locked audio data corresponds to posture-independent binaural audio.

[0485] Example 43. The device according to Example 37 further includes a buffer accessible by one or more processors, wherein the one or more processors are further configured to receive sound effects data in advance via wireless transmission; and to pre-buffer the sound effects data in the buffer.

[0486] Example 44. The device according to Example 43, wherein one or more processors are further configured to, in response to an instruction received of an interaction between a user and a virtual object associated with sound effect data, retrieve a portion of pre-buffered sound effect data corresponding to the virtual object from a buffer; and combine the portion of the pre-buffered sound effect data with adjusted decompressed audio data for output to two or more speakers.

[0487] Example 45. The device according to Example 37, wherein the audio data includes immersive sound data, and wherein one or more processors are further configured to send an indication of an immersive sound level to a remote device; and in response to sending the indication, to receive updated audio data having the immersive sound level via wireless transmission.

[0488] Example 46. A device according to Example 37, wherein one or more processors are integrated in the headphone device.

[0489] Example 47. The device according to Example 46, wherein the headphone device further includes a memory configured to store instructions executable by one or more processors; one or more sensors configured to generate data associated with tracking position and orientation; a first speaker configured to be placed close to a user's first ear when the user wears the headphone device; a second speaker configured to be placed close to a user's second ear when the user wears the headphone device; and a receiver configured to receive compressed audio data via wireless transmission.

[0490] Example 48. The device according to Example 37, wherein one or more processors are integrated into the vehicle, and wherein data associated with tracking position and orientation indicates the position and orientation of the vehicle.

[0491] Example 49. A device according to Example 37, wherein one or more processors are integrated into a speaker array device and are also configured to perform beam steering operations to direct binaural signals to a location associated with a user.

[0492] Example 50. The device according to Example 37, wherein one or more processors are integrated in at least one of a mobile phone, a tablet computer device, a wearable electronic device, a camera device, a virtual reality headset, a mixed reality headset, or an augmented reality headset.

[0493] Example 51. A method comprising: acquiring data associated with location and orientation at multiple time instances related to tracking movement of a device; transmitting the data to a remote device via wireless transmission; receiving compressed audio data representing a sound field from the remote device via wireless transmission; decompressing the compressed audio data representing the sound field; adjusting the decompressed audio data to alter the sound field based on the orientation associated with the device; and outputting the adjusted decompressed audio data to two or more speakers.

[0494] Example 52. According to the method of Example 51, wherein the decompressed audio data is adjusted based on data associated with the location and orientation of the device and the application tracking the movement of the device.

[0495] Example 53. According to the method of Example 51, wherein the compressed audio data includes panoramic sound data.

[0496] Example 54. The method according to Example 51 further includes adjusting the decompressed audio data to translate the sound field based on the difference between the location of the device and the location associated with the sound field, wherein the adjustment of the decompressed audio data based on the difference is limited to a translation of the sound field forward, backward, left, or right.

[0497] Example 55. The method of Example 51 further includes receiving head-locked audio data from a remote device via wireless transmission; and combining the head-locked audio data with adjusted decompressed audio data for output to two or more speakers.

[0498] Example 56. According to the method of Example 55, wherein the adjusted decompressed audio data corresponds to posture-adjusted binaural audio, and wherein the head-locked audio data corresponds to posture-independent binaural audio.

[0499] Example 57. The method according to Example 51 further includes receiving sound effect data in advance via wireless transmission; and pre-buffering the sound effect data.

[0500] Example 58. The method of Example 57 further includes, in response to an instruction from an interaction between a user and a virtual object associated with the sound effects data, retrieving a portion of the pre-buffered sound effects data corresponding to the virtual object; and combining that portion of the pre-buffered sound effects data with adjusted decompressed audio data for output to two or more speakers.

[0501] Example 59. The method of Example 51, wherein the audio data includes panoramic sound data, and further includes sending an indication of panoramic sound order to a remote device; and receiving updated audio data having the panoramic sound order in response to sending the indication.

[0502] Example 60. A non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to: acquire data associated with position and orientation at multiple time instances in relation to tracking movement of a device; transmit the data to a remote device via wireless transmission; receive compressed audio data representing a sound field from the remote device via wireless transmission; decompress the compressed audio data representing the sound field; adjust the decompressed audio data to alter the sound field based on the orientation associated with the device; and output the adjusted decompressed audio data to two or more speakers.

[0503] Example 61. An apparatus comprising: means for acquiring data associated with position and orientation related to tracking movement of the apparatus at multiple time instances; means for transmitting the data to a remote device via wireless transmission; means for receiving compressed audio data representing a sound field from the remote device via wireless transmission; means for decompressing the compressed audio data representing the sound field; means for adjusting the decompressed audio data to change the sound field based on the orientation associated with the apparatus; and means for outputting the adjusted decompressed audio data to two or more loudspeakers.

[0504] Example 62. A device includes one or more processors configured to receive sound information from an audio source; receive data from a playback device corresponding to positions associated with the playback device at multiple time instances; convert the sound information into audio data representing a sound field based on the data corresponding to the positions associated with the playback device; and transmit the audio data as streaming data to one or both of the playback device or a second device coupled to the playback device via wireless transmission.

[0505] Example 63. The device according to Example 62, wherein one or more processors are further configured to generate a predicted location of the playback device based on data corresponding to a location associated with the playback device, the predicted location indicating a prediction of where the playback device will be when audio data is played at the playback device; and to convert sound information into audio data representing a sound field based on the predicted location.

[0506] Example 64. The device according to Example 62, wherein one or more processors are further configured to send sound effect data from an audio source to one or both of a playback device or a second device, the sound effect data being buffered and accessible by the playback device for future playback, wherein at least a portion of the sound effect data is sent independently of any scheduled playback of that portion of the sound effect data.

[0507] Example 65. The device according to Example 62, wherein one or more processors are further configured to receive a head-locked audio portion from an audio source; generate head-locked audio data corresponding to posture-independent binaural audio based on the head-locked audio portion; and transmit the head-locked audio data via wireless transmission to one or both of a playback device or a second device for playback on the playback device.

[0508] Example 66. The device according to Example 62, wherein the audio data corresponds to immersive sound data, and wherein one or more processors are further configured to receive an indication of the immersive sound level from the playback device; and to adjust the audio data to have the immersive sound level.

[0509] Example 67. The device according to Example 62, wherein one or more processors are further configured to, after receiving data corresponding to a location associated with the playback device, receive additional data corresponding to the location associated with the playback device; generate updated audio data based on the additional data; and send the updated audio data to the playback device.

[0510] Example 68. A method comprising: receiving sound information from an audio source; receiving data from a playback device corresponding to positions associated with the playback device at multiple time instances; converting the sound information into audio data representing a sound field based on the data corresponding to the positions associated with the playback device; and transmitting the audio data as streaming data to one or both of the playback device or a second device coupled to the playback device via wireless transmission.

[0511] Example 69. A non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause one or more processors to: receive sound information from an audio source; receive data from a playback device corresponding to positions associated with the playback device at a plurality of time instances; convert the sound information into audio data representing a sound field based on the data corresponding to the positions associated with the playback device; and transmit the audio data as streaming data to one or both of the playback device or a second device coupled to the playback device via wireless transmission.

[0512] Example 70. An apparatus comprising: means for receiving sound information from an audio source; means for receiving data from a playback device corresponding to positions associated with the playback device at a plurality of time instances; means for converting the sound information into audio data representing a sound field based on the data corresponding to the positions associated with the playback device; and means for transmitting the audio data as streaming data to one or both of the playback device or a second device coupled to the playback device via wireless transmission.

[0513] Example 71. A device includes one or more processors configured to receive compressed audio data representing a sound field from a streaming device; receive data from a playback device corresponding to positions associated with the playback device at multiple time instances; generate a predicted position of the device based on the data corresponding to the positions associated with the playback device, the predicted position indicating a prediction of where the playback device will be when the audio data is played on the playback device; decompress the compressed audio data; adjust the decompressed audio data based on the predicted position to translate the sound field; compress the adjusted audio data; and transmit the compressed adjusted audio data as streaming data to the playback device via wireless transmission.

[0514] Example 72. A method comprising: receiving compressed audio data representing a sound field from a streaming device; receiving data corresponding to positions associated with the playback device at multiple time instances from a playback device; generating a predicted position of the device based on the data corresponding to the positions associated with the playback device, the predicted position indicating a prediction of where the playback device will be when the audio data is played on the playback device; decompressing the compressed audio data; adjusting the decompressed audio data to translate the sound field based on the predicted position; compressing the adjusted audio data; and transmitting the compressed adjusted audio data as streaming data to the playback device via wireless transmission.

[0515] Example 73. A non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to receive compressed audio data representing a sound field from a streaming device; receive data from a playback device corresponding to positions associated with the playback device at multiple time instances; generate a predicted position of the playback device based on the data corresponding to the positions associated with the playback device, the predicted position indicating a prediction of where the playback device will be when the audio data is played on the playback device; decompress the compressed audio data; adjust the decompressed audio data based on the predicted position to translate the sound field; compress the adjusted audio data; and transmit the compressed adjusted audio data as streaming data to the playback device via wireless transmission.

[0516] Example 74. An apparatus comprising: means for receiving compressed audio data representing a sound field from a streaming device; means for receiving data from a playback device corresponding to positions associated with the playback device at multiple time instances; means for generating a predicted position of the device based on the data corresponding to the positions associated with the playback device, the predicted position indicating a prediction of where the playback device will be when the audio data is played on the playback device; means for decompressing the compressed audio data; means for adjusting the decompressed audio data based on the predicted position to translate the sound field; means for compressing the adjusted audio data; and means for transmitting the compressed adjusted audio data as streaming data to the playback device via wireless transmission.

[0517] Example 75. A device comprising one or more processors configured to receive, via wireless transmission, compressed audio data corresponding to a first representation of a sound field, the first representation corresponding to a first viewport field of view associated with a first orientation of the device; decompress the compressed audio data; output the decompressed audio data to two or more speakers; transmit to the streaming device data associated with a second orientation of the device; receive compressed, updated audio data from the streaming device, the compressed, updated audio data corresponding to a second representation of the sound field, and the second representation corresponding to a second viewport field of view that partially overlaps with the first viewport field of view and is associated with the second orientation; decompress the compressed, updated audio data; and output the decompressed, updated audio data to two or more speakers.

[0518] Example 76. The device according to Example 75, wherein the decompressed audio data includes immersive sound data.

[0519] Example 77. The device according to Example 75, wherein a first representation of the sound field provides a higher resolution for an audio source in a first viewport field of view than for an audio source outside the first viewport field of view, and wherein a second representation of the sound field provides a higher resolution for an audio source in a second viewport field of view than for an audio source outside the second viewport field of view.

[0520] Example 78. The device according to Example 75, wherein the compressed updated audio data corresponds to the rotation of the sound field (which is based on the rotation of the device between a first posture and a second posture); and the translation of the sound field.

[0521] Example 79. The device according to Example 78, wherein the translation of the sound field corresponds to the translation of the device between a first posture and a second posture.

[0522] Example 80. The device according to Example 78, wherein the translation of the sound field exceeds the translation of the device between a first pose and a second pose.

[0523] Example 81. The device according to Example 75, wherein one or more processors are further configured to adjust decompressed audio data to change a first representation of the sound field based on data associated with at least one of translation or orientation related to movement of the device toward a second attitude; and to output the adjusted decompressed audio data to two or more speakers.

[0524] Example 82. A device according to Example 75, wherein one or more processors are integrated in the headphone device.

[0525] Example 83. The device according to Example 82, wherein the headphone device further includes a memory configured to store instructions executable by one or more processors; one or more sensors configured to generate data corresponding to the orientation of the device; a first speaker configured to be placed close to a user's first ear when the user wears the headphone device; a second speaker configured to be placed close to a user's second ear when the user wears the headphone device; and a wireless receiver configured to receive compressed audio data via wireless transmission.

[0526] Example 84. A method comprising: receiving, at one or more processors of a device and via wireless transmission, compressed audio data corresponding to a first representation of a sound field, the first representation corresponding to a first viewport field of view associated with a first orientation of the device; decompressing the compressed audio data; outputting the decompressed audio data to two or more speakers; transmitting to the streaming device data associated with a second orientation of the device; receiving, from the streaming device, compressed updated audio data corresponding to a second representation of the sound field, and the second representation corresponding to a second viewport field of view that partially overlaps with the first viewport field of view and is associated with the second orientation; decompressing the compressed updated audio data; and outputting the decompressed updated audio data to two or more speakers.

[0527] Example 85. A non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause one or more processors to: receive, via wireless transmission, compressed audio data corresponding to a first representation of a sound field from a streaming device, the first representation corresponding to a first viewport field of view associated with a first orientation of the device; decompress the compressed audio data; output the decompressed audio data to two or more speakers; send to the streaming device data associated with a second orientation of the device; receive from the streaming device compressed updated audio data corresponding to a second representation of the sound field, and the second representation corresponding to a second viewport field of view that partially overlaps with the first viewport field of view and is associated with the second orientation; decompress the compressed updated audio data; and output the decompressed updated audio data to two or more speakers.

[0528] Example 86. An apparatus comprising: means for receiving compressed audio data corresponding to a first representation of a sound field from a streaming device via wireless transmission, the first representation corresponding to a first viewport field of view associated with a first orientation of the device; means for decompressing the compressed audio data; means for outputting the decompressed audio data to two or more speakers; means for transmitting data associated with a second orientation of the streaming device to the streaming device; means for receiving compressed updated audio data from the streaming device, the compressed updated audio data corresponding to a second representation of the sound field, and the second representation corresponding to a second viewport field of view that partially overlaps with the first viewport field of view and is associated with the second orientation; means for decompressing the compressed updated audio data; and means for outputting the decompressed updated audio data to two or more speakers.

[0529] Example 87. A device comprising one or more processors configured to receive, via wireless transmission, data associated with the attitude of the playback device from a playback device; select, based on the data, a specific representation of a sound field from a plurality of representations of a sound field, each corresponding representation of the sound field corresponding to a different viewport field in a plurality of overlapping viewport field sets; generate compressed audio data corresponding to the selected representation of the sound field; and transmit, via wireless transmission, the compressed audio data as streaming data to the playback device.

[0530] Example 88. A device according to Example 87, wherein the selected representation corresponds to a first representation of the sound field corresponding to a first sector, and wherein one or more processors are further configured to receive second data associated with a second pose of the playback device from the playback device; select a second representation of the sound field from a plurality of representations of the sound field based on the second data, the second representation corresponding to a second viewport field of view in a set, wherein the second viewport field of view partially overlaps with the first viewport field of view; generate second compressed audio data corresponding to the second representation of the sound field; and transmit the second compressed audio data to the playback device via wireless transmission.

[0531] Example 89. A device according to Example 87, wherein the selected representation corresponds to a first representation of the sound field corresponding to a first sector, and wherein one or more processors are further configured to receive second data associated with a second orientation of the playback device from the playback device; select a second representation of the sound field from a plurality of representations of the sound field based on the second data, the second representation corresponding to a translation of the sound field; generate second compressed audio data corresponding to the second representation of the sound field; and transmit the second compressed audio data to the playback device via wireless transmission.

[0532] Example 90. The device according to Example 89, wherein the translation of the sound field c...

Claims

1. A device for sound field adjustment, comprising: One or more processors, which are configured as follows: Obtain sound information from the audio source; Based on the actual or predicted amount of motion of the playback device, select the complete set of uncompressed Atmos coefficients, or the truncated set of Atmos coefficients in which one or more higher Atmos orders of the full-order uncompressed Atmos data have been discarded. as well as The selected complete uncompressed or truncated set of Atmos coefficients is transmitted wirelessly as streaming data to the playback device.

2. The device according to claim 1, wherein, The sound information corresponds to at least one of 2D audio data representing a two-dimensional 2D sound field or 3D audio data representing a three-dimensional 3D sound field.

3. The device according to claim 1, wherein, The one or more processors are configured to: Determine whether the wireless link to the playback device corresponds to a higher bandwidth wireless link or a lower bandwidth wireless link; as well as The complete set of uncompressed immersive sound coefficients is selected based on the wireless link corresponding to the higher bandwidth wireless link.

4. The device according to claim 3, wherein, The higher bandwidth wireless link corresponds to a fifth-generation (5G) cellular digital network or a WiFi-type network.

5. The device according to claim 1, wherein, The one or more processors are configured to: Determine whether the wireless link to the playback device corresponds to a higher bandwidth wireless link or a lower bandwidth wireless link; as well as The truncated set of immersive sound coefficients is selected based on the wireless link corresponding to the lower bandwidth wireless link.

6. The device according to claim 5, wherein, The lower bandwidth wireless link corresponds to a Bluetooth network.

7. The device according to claim 1, wherein, The one or more processors are further configured to: Receive a request from the playback device for the complete uncompressed Atmos coefficient set or the truncated Atmos coefficient set; as well as Based on the request, select either the complete uncompressed Atmos coefficient set or the truncated Atmos coefficient set.

8. The device according to claim 1, wherein, The one or more processors are further configured to: Receive data from the playback device corresponding to the position and orientation associated with the movement of the playback device; The sound information is updated based on the received data to change the sound field; as well as Compressed audio data representing the sound field is transmitted wirelessly to the playback device, enabling the playback device to decompress the compressed audio data representing the sound field, adjust the decompressed audio data based on the orientation associated with the device to change the sound field, and output the adjusted decompressed audio data to two or more speakers.

9. The device according to claim 1, wherein, The one or more processors are further configured to: Receive data from the playback device corresponding to the position and orientation associated with the movement of the playback device; The sound information is updated based on the received data to change the sound field; as well as Uncompressed audio data representing the sound field is transmitted wirelessly to the playback device, enabling the playback device to adjust the audio data to alter the sound field based on the orientation associated with the device, and output the adjusted audio data to two or more speakers.

10. The device according to claim 1, wherein, The one or more processors are further configured to: Receive translation data from the playback device, the translation data corresponding to a translation associated with the playback device; and The sound information is converted into audio data representing the sound field based on the translation.

11. The device according to claim 1, wherein, The one or more processors are integrated in a portable electronic device.

12. The device according to claim 1, wherein, The one or more processors are integrated in the server.

13. The device of claim 1, further comprising a modem configured to modulate audio data for transmission to the playback device.

14. The device of claim 13, further comprising an antenna coupled to the modem and configured to transmit modulated audio data to the playback device.

15. A method for sound field adjustment, comprising: Obtain sound information from the audio source; Based on the actual or predicted amount of motion of the playback device, select the complete set of uncompressed Atmos coefficients, or the truncated set of Atmos coefficients in which one or more higher Atmos orders of the full-order uncompressed Atmos data have been discarded. The selected complete uncompressed or truncated set of Atmos coefficients is transmitted wirelessly as streaming data to the playback device.

16. The method according to claim 15, wherein, The sound information corresponds to at least one of 2D audio data representing a two-dimensional 2D sound field or 3D audio data representing a three-dimensional 3D sound field.

17. The method of claim 15, further comprising: Determine whether the wireless link to the playback device corresponds to a higher bandwidth wireless link or a lower bandwidth wireless link; as well as The complete set of uncompressed immersive sound coefficients is selected based on the wireless link corresponding to the higher bandwidth wireless link.

18. The method of claim 15, further comprising: Determine whether the wireless link to the playback device corresponds to a higher bandwidth wireless link or a lower bandwidth wireless link; as well as The truncated set of immersive sound coefficients is selected based on the wireless link corresponding to the lower bandwidth wireless link.

19. The method of claim 15, further comprising: Receive a request from the playback device for the complete uncompressed Atmos coefficient set or the truncated Atmos coefficient set; as well as Based on the request, select either the complete uncompressed Atmos coefficient set or the truncated Atmos coefficient set.

20. The method of claim 15, further comprising: Receive translation data from the playback device, the translation data corresponding to a translation associated with the playback device; as well as The sound information is converted into audio data representing the sound field based on the translation.

21. A device for sound field adjustment, comprising: A component used to acquire sound information from an audio source; A component for selecting a complete set of uncompressed Atmos coefficients or a truncated set of Atmos coefficients in which one or more higher Atmos orders of full-order uncompressed Atmos data have been discarded, based on the actual or predicted amount of motion of the playback device. A component for transmitting, via wireless transmission, the selected complete uncompressed Atmos coefficient set or the truncated Atmos coefficient set as streaming data to the playback device.

22. The apparatus according to claim 21, wherein, The sound information corresponds to at least one of 2D audio data representing a two-dimensional 2D sound field or 3D audio data representing a three-dimensional 3D sound field.

23. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to: Obtain sound information from the audio source; Based on the actual or predicted amount of motion of the playback device, select the complete set of uncompressed Atmos coefficients, or the truncated set of Atmos coefficients in which one or more higher Atmos orders of the full-order uncompressed Atmos data have been discarded. The selected complete uncompressed or truncated set of Atmos coefficients is transmitted wirelessly as streaming data to the playback device.

24. The non-transitory computer-readable medium according to claim 23, wherein, When the instruction is executed, it further causes the one or more processors to: Determine whether the wireless link to the playback device corresponds to a higher bandwidth wireless link or a lower bandwidth wireless link; as well as The complete set of uncompressed immersive sound coefficients is selected based on the wireless link corresponding to the higher bandwidth wireless link.

25. The non-transitory computer-readable medium according to claim 23, wherein, When the instruction is executed, it further causes the one or more processors to: Determine whether the wireless link to the playback device corresponds to a higher bandwidth wireless link or a lower bandwidth wireless link; as well as The truncated set of immersive sound coefficients is selected based on the wireless link corresponding to the lower bandwidth wireless link.

26. A computer program product comprising computer-readable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 15-20.

Images