Synchronization of header tracking data

Abstract

Methods, systems, and media are provided for tracking data with a head. In some embodiments, a method involves receiving sensor data from one or more sensors at each of a pair of communicatively coupled earbud headphones. The method may involve determining head orientation information at each earbud headphone of the pair of communicatively coupled earbud headphones. The method may involve transmitting the determined head orientation information between the pair of communicatively coupled earbud headphones, such that the master earbud headphone sends the head orientation information determined by the master earbud headphone to the slave earbud headphone. The method may involve synchronizing, at each earbud headset, the determined head orientation data based at least in part on timing information associated with a timestamp at which the head orientation information was transmitted. The method may involve utilizing the synchronized head orientation data to present audio content by each of the pair of communicatively coupled earbud headsets.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to PCT Application No. PCT / CN2022 / 118753 filed on September 14, 2022, U.S. Provisional Application No. 63 / 423,265 filed on November 7, 2022, and U.S. Provisional Application No. 63 / 492,724 filed on March 28, 2023, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to systems, methods, and media for synchronization of head tracking related data. Background Art

[0004] Audio content listeners may be interested in an immersive listening experience in which the audio content is rendered based on head orientation. However, it may be difficult to accurately monitor head orientation and render audio content based on the head orientation.

[0005] Symbols and terminology

[0006] Throughout this disclosure, including in the claims, the terms "speaker," "loudspeaker," and "audio reproduction transducer" are used synonymously to refer to any sound-producing transducer (or set of transducers). A typical set of headphones includes two speakers. Speakers may be implemented to include multiple transducers (e.g., a woofer and a tweeter), which may be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker feed(s) may undergo different processing in different circuit branches coupled to different transducers.

[0007] Throughout this disclosure, including in the claims, expressions referring to performing an operation “on” a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) are used in a broad sense to mean performing the operation directly on the signal or data or on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed on it).

[0008] Throughout this disclosure, including in the claims, the expression "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M inputs and the other XM inputs are received from external sources) may also be referred to as a decoder system.

[0009] Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and / or otherwise configured to perform pipeline processing of audio or other sound data, programmable general purpose processors or computers, and programmable microprocessor chips or chipsets. Summary of the invention

[0010] Methods, systems, and media for utilizing head tracking data are provided. In some embodiments, a method involves receiving sensor data from one or more sensors at each earbud in a pair of communicatively coupled earbuds. The method may involve determining head orientation information at each earbud in the pair of communicatively coupled earbuds. The method may involve transmitting the determined head orientation information between the pair of communicatively coupled earbuds so that a master earbud in the pair of communicatively coupled earbuds sends the head orientation information determined by the master earbud to a slave earbud in the pair of communicatively coupled earbuds. The method may involve synchronizing the determined head orientation data at each earbud in the pair of communicatively coupled earbuds based at least in part on timing information associated with a timestamp at which the head orientation information was transmitted. The method may involve using the synchronized head orientation data to present audio content by each earbud in the pair of communicatively coupled earbuds.

[0011] In some examples, determining head orientation information at each earbud includes fusing data from two or more sensors disposed in or on the earbud.

[0012] In some examples, determining the head orientation information includes determining predicted head orientation data. In some examples, the slave earbud is configured to utilize the predicted head orientation data to compensate for a delay in receiving the head orientation information from the master earbud.

[0013] In some examples, the method further involves smoothing the synchronized head orientation data, wherein the synchronized head orientation data used to present the audio content includes the smoothed synchronized head orientation data.

[0014] In some examples, transmitting the determined head orientation information between the pair of communicatively coupled earbuds includes including the timing information in a data stream transmitted from the master earbud to the slave earbud. In some examples, the method further involves determining at the slave earbud whether data is missing in the head orientation information received from the master earbud based at least in part on the timing information received from the master earbud. In some examples, synchronizing the determined head orientation data at the slave earbud includes synchronizing the determined head orientation data determined at the slave earbud based on one or more sensors of the slave earbud with the head orientation data received from the master earbud using the timing information.

[0015] In some examples, determining head orientation information at each earbud includes re-centering head orientation data generated based on the sensor data to compensate for drift of the sensor data.

[0016] In some examples, each earbud includes a cache that stores at least head orientation data. In some examples, the head orientation data is transmitted from the cache of the master earbud to the slave earbud at a timing interval that depends at least on a data transmission rate and a latency associated with transmitting the determined head orientation information between the pair of communicatively coupled earbuds.

[0017] In some examples, the method further involves resampling the sensor data before determining the head orientation information to compensate for a difference between a sampling rate used to acquire the sensor data and a sampling rate used to determine the head orientation information.

[0018] In some examples, the method further involves resampling the determined head orientation information prior to transmitting the determined head orientation information to compensate for a difference between a sampling rate used to determine the head orientation information and a sampling rate used to transmit the determined head orientation information.

[0019] In some examples, the method further involves resampling the synchronized head orientation data prior to presenting the audio content using the synchronized head orientation data to compensate for a difference between a sampling rate used to synchronize the determined head orientation data and a sampling rate used to process the audio content.

[0020] In some examples, transmitting the determined head orientation information includes utilizing a Bluetooth communication protocol.

[0021] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Thus, some innovative aspects of the subject matter described in this disclosure may be implemented via one or more non-transitory media having software stored thereon.

[0022] At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices may be capable of at least partially performing the methods disclosed herein. In some embodiments, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or a combination thereof.

[0023] The details of one or more embodiments of the subject matter described in this specification are set forth in the following drawings and description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Note that the relative sizes of the following drawings may not be drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a diagram illustrating a pair of earbud headphones according to some embodiments.

[0025] Figure 2 is a schematic block diagram of a system for synchronizing head tracking data according to some embodiments.

[0026] Figure 3 is a schematic block diagram of components of an earbud headphone according to some embodiments.

[0027] Figure 4 is a schematic block diagram of an example system for determining head tracking information in accordance with some embodiments.

[0028] Figure 5A and Figure 5B is a schematic block diagram of an example data synchronization system for a master earbud and a slave earbud, respectively, according to some embodiments.

[0029] Fig. 6A and Figure 6B is a schematic block diagram of an example data transmission system for a master earbud and a slave earbud, respectively, according to some embodiments.

[0030] Figure 7is a flow chart of an example process for synchronizing and utilizing head orientation information by a pair of earbuds in accordance with some implementations.

[0031] Figure 8 A block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure is shown.

[0032] Like reference numbers and designations throughout the various drawings indicate like elements. DETAILED DESCRIPTION

[0033] Audio content consumers are increasingly interested in immersive audio content. Immersive audio content can be audio content including spatial positioning information (e.g., to be perceived as an audio object located in a specific spatial position). When rendered by a headset, this immersive content can be rendered based on the head orientation of the listener wearing the headset. As used herein, "immersive content" refers to audio data rendered with reference to an external reference system (such as a reference system defined by a display screen that is presenting a movie, television program, etc.). Conventionally, a headset can include one or more processors and one or more sensors that are used to generate head orientation data, and audio content is rendered based on these head orientation data. However, for earbuds (where each earbud in a pair of earbuds is physically separated), rendering based on head orientation may be more difficult. For example, each earbud may have its own set of sensors that generate sensor data, and head orientation is determined based on the sensor data. However, due to the two physically separated earbuds, the head orientation determined by each earbud may be different, and may even be drastically different, resulting in errors or inaccuracies in the rendered audio content.

[0034] Disclosed herein are systems, methods, and techniques for synchronizing head orientation data and state and control information on a pair of earbuds. Each earbud may have its own set of sensors, which may include one or more accelerometers, one or more gyroscopes, one or more magnetometers, and the like. Each earbud may then generate head orientation data based on the sensor data. The head orientation data may be shared between the pair of earbuds using a wireless communication channel. For example, the wireless communication channel may utilize a Bluetooth communication protocol or any other suitable wireless communication protocol. Each earbud may then synchronize its own generated head orientation data with the head orientation data received by the paired earbuds. This may allow a more robust determination of head orientation, which in turn may allow a more robust rendering of audio content. Note that, as used herein, state / control information may include timing information (e.g., timestamp information, count information, and any other timing-related information), which may be used for delay compensation (e.g., for compensating for a transmission delay in transmitting data between two paired earbuds). Additionally or alternatively, the state / control information may include data generated at the first earbud and transmitted to the paired earbud, the data being configured to be used to re-center the head orientation data (e.g., to account for and / or compensate for system drift associated with the sensor). The data may be used, for example, by a re-centering block, such as in Figure 4 FIG. 1 is shown in FIG. 1 and described below in conjunction with the figure.

[0035] In some embodiments, the first earbud in the pair of earbuds can be designated as the master earbud, and the second earbud in the pair of earbuds can be designated as the slave earbud. The master earbud can generate timestamp or count data that can be included in a data stream transmitted to the slave earbud. In some embodiments, the slave earbud can utilize the timestamp or count data to detect missing or delayed data (e.g., due to weak signal strength of a wireless communication channel, interference from nearby devices, etc.), and can then compensate for the missing or delayed data. In some embodiments, each earbud can be configured to generate predicted head orientation data, e.g., a prediction of a listener's future head orientation. The predicted head orientation data can be used to compensate for missing or delayed head orientation data from the paired earbud.

[0036] Figure 1An example listening scenario according to some embodiments is illustrated. As shown, a listener 100 may listen to audio content using a pair of earbuds, which may include a left earbud 102a and a right earbud 102b. The two earbuds may be communicatively coupled via a wireless communication channel 106. In one example, the wireless communication channel may be a Bluetooth communication channel, such as Figure 1 As shown. Each earbud may include hardware circuitry that may include one or more sensors configured to provide sensor data indicative of a user's head orientation. For example, the one or more sensors may include one or more accelerometers, one or more gyroscopes, one or more magnetometers, and the like. The circuitry of each earbud may further include one or more processors configured to process the sensor data, determine the head orientation based on the sensor data, predict future head orientation based on the sensor data, transmit and / or receive communications from a paired mobile device and / or another earbud, and the like. Figure 1 In the example of FIG. 1 , the left earbud 102a includes the circuit 104a, and the right earbud 102b includes the circuit 104b.

[0037] In some embodiments, each earbud in a pair of earbuds may include one or more sensors configured to provide sensor data that can be used to determine head orientation. The two earbuds may be communicatively coupled (e.g., via a wireless communication channel) so that the two earbuds can share information associated with head orientation, timing information, status, and control information, etc. Each earbud may also be communicatively coupled (e.g., via a wireless communication channel) to a user device that provides audio content. Each earbud may be configured to render audio content received from a paired user device using head orientation information, and then play the rendered audio content by outputting a rendered audio signal. Note that in some embodiments, one of the earbuds in a pair of earbuds may be considered a master earbud, and the other earbud may be considered a slave earbud. The master earbud may provide instructions, timing information, etc. to the slave earbud. The slave earbud may synchronize the head orientation information obtained using its own sensor with the head orientation information obtained from the master earbud. Such synchronization may be used to overcome differences between sensors in each of the two earbuds, jitter or communication delays between the two earbuds and / or the mobile device, and the like.

[0038] Figure 2is a diagram of an example system for synchronizing head orientation and utilizing synchronized head orientation information according to some embodiments. As shown, user device 202 provides audio streaming data to master earbud 204a and slave earbud 204b. Note that although Figure 2 Indicates that the master earbud 204a is the left earbud and the slave earbud 204b is the right earbud, but in other embodiments, the master earbud may be the right earbud and the slave earbud may be the left earbud. The audio stream data may be audio content of a movie or television program, audio content associated with music, audio content associated with a podcast, etc. The audio stream data may include one or more audio objects to be rendered at a specific spatial orientation relative to the user's head based on the user's head orientation data.

[0039] Each earbud may include an audio processing block configured to receive audio stream data from the user device 202 and render the audio stream data based on the head orientation data. The audio processing block may then be configured to play the rendered audio data. As shown, the master earbud 204a includes an audio processing block 206a, while the slave earbud 204b includes an audio processing block 206b. It should be understood that, as used herein, a "block" may be implemented in software, for example as one or more functions. The software may be executed by one or more processors or one or more control systems, each of which is disposed in or on a given earbud, in or on a user device paired with the earbud, and the like. Examples of processors or control systems that may be used to implement a "block" are provided in Figure 8 , and described below in conjunction with this figure. Figure 2 In the example shown, control system 250a is configured to implement audio processing block 206a, and control system 250b is configured to implement audio processing block 206b. Control systems 250a and 250a are referred to herein as Figure 8 An example of a control system 810 is described.

[0040] Each audio processing block is configured to receive head orientation data from a corresponding head tracking block. For example, audio processing block 206a receives head orientation data from head tracking block 208a, and audio processing block 206b receives head orientation data from head tracking block 208b. Each head tracking block is configured to generate head orientation data based on sensor data received from one or more sensors. Figure 2In the example shown, control system 250a is configured to implement head tracking block 206a, and control system 250b is configured to implement head tracking block 206b. Note that each earbud has its own set of one or more sensors disposed in or on the earbud. For example, master earbud 204a may utilize set of sensors 210a, and slave earbud 204b may utilize set of sensors 210b. The set of sensors may include one or more accelerometers, one or more magnetometers, one or more gyroscopes, and the like.

[0041] In some embodiments, each earbud can obtain sensor data from one or more sensors disposed in or on the earbud, and can use the sensor data to determine the head orientation of the earbud wearer. The head orientation data can be determined by one or more processors of the earbud. The head orientation data can be shared with the paired earbud. For example, the left earbud can share head orientation data with the paired right earbud, and vice versa. The head orientation data can be shared via a wireless communication channel between two paired earbuds. The shared head orientation data can be used to determine a more robust head orientation (e.g., due to the use of multiple sensors on two different earbuds), thereby allowing more robust rendering of audio data. Note that in some embodiments, each earbud can determine predicted head orientation data, which can be used to compensate for the discarded data packets and / or delays when receiving the shared head orientation data from the paired earbud. For example, the predicted head orientation data determined at the first earbud can be used in the case where the head orientation data from the paired second earbud is delayed or discarded. Data between two paired earbuds can be synchronized. Data can be synchronized using timing information shared between the two paired earbuds. For example, timing information can be shared by adding timestamp data to a data stream including head orientation data. In some embodiments, the synchronized head orientation data can be used to re-center control data for rendering audio content. This can compensate for drift in the sensor data that causes drift in the head orientation data.

[0042] Note that in some embodiments, different aspects of the head orientation data synchronization and utilization system can operate at different sampling frequencies. For example, sensor data can be collected by a sensor having a sensor driver operating at a specific sensor sampling frequency (e.g., 100Hz, 104Hz, 110Hz, etc.), and head tracking orientation determination can be performed with a processor operating at a different rate (e.g., 40Hz, 50Hz, 60Hz, etc.). As another example, audio processing can be performed in the range of about 90Hz to 120Hz. In one example, audio processing can be performed at 93.75Hz (e.g., by processing 512 samples at a sampling rate of 48KHz). As yet another example, data can be transmitted between two earbuds at a data communication rate of 20Hz. In order to address different sampling rates between different subsystems, each earbud may include a resampling block configured to resample data from a first sampling rate to a second sampling rate. The resampling block can be executed by one or more processors of the earbud.

[0043] Figure 3 is a schematic block diagram of components of an example earbud headphone. Note that although Figure 3 Indicates that the earbuds are earbuds 204a, corresponding to Figure 2 , but the same or similar components may be included in the slave earbuds, such as Figure 2 In some embodiments, the various components may be implemented in software and may be executed by one or more processors or one or more control systems disposed in or on the earbuds. Examples of such processors or control systems are described in Figure 8 FIG. 1 is shown in FIG. 1 and described below in conjunction with the figure. Figure 3 As shown, earbud 204a includes a head tracking block 208a configured to determine head orientation data based on data from sensor 210a. Head tracking block 208a provides the head orientation data to audio processing block 206a and sends an indication of the head orientation data to a paired earbud (which may be Figure 2 from earbud headphones 204b).

[0044] like Figure 3 As shown, head orientation data may be determined by a data processing block 302 that receives sensor data and determines head orientation data. More detailed techniques that may be utilized by the data processing block 302 are described in Figure 4 FIG. 1 is shown in FIG. 1 and described below in conjunction with the figure.

[0045] The data processing block 302 communicates with the data synchronization block 304. As shown, the data synchronization block 304 also communicates with the data transfer block 306. With respect to outbound data, the data synchronization block 304 can generate timestamp data and apply it to a data stream that includes head orientation data (and predicted head orientation data and / or status and control information) generated by the data processing block 302. The head orientation data, predicted head orientation data, status and control information, and timestamp data can be provided to the data transfer block 306. With respect to inbound data, the data synchronization block 304 can receive head orientation data and predicted head orientation data from the data transfer block 306 (which in turn receives data from paired earbuds), and can synchronize the data from the paired earbuds with the head orientation data obtained using the sensors of the given earbuds. The synchronization data and status / control information can be provided to the data processing block 302 for use by the audio processing block 206a. More detailed techniques that can be implemented by the data synchronization block 304 are described in detail in the accompanying drawings. Figure 5A (for primary earbuds) and Figure 5B (for earbud type headphones) and are described below in conjunction with these two figures.

[0046] The data transfer block 306 may send head orientation data, predicted head orientation data, status / control information, and timestamp information to the paired earbuds. The data transfer block 306 may also receive head orientation data, predicted head orientation data, and timestamp information from the paired earbuds. More detailed techniques that may be implemented by the data transfer block 306 are described in Fig. 6A (for primary earbuds) and Figure 6B (for earbud type headphones) and are described below in conjunction with these two figures.

[0047] As shown, the system includes a resampler block 308, which is configured to convert data from a first sampling rate to a second sampling rate. The resampler block 308 can be used by any one of the data processing block 302, the data synchronization block 304, the data transmission block 306, or the audio processing block 206a to convert from one sampling rate to another sampling rate.

[0048] In some embodiments, one or more processors of the earbuds may receive sensor data from one or more sensors. The sensor data may include motion data from, for example, one or more accelerometers and / or one or more gyroscopes. The one or more processors may utilize data fusion to transform the motion data into head orientation data. In addition, one or more sensors may predict future head orientation data. The predicted head orientation data may be used to compensate for delays, such as when receiving head orientation data from a paired earbud. The one or more processors may perform automatic re-centering to correct drift of the motion data preview provided by the one or more sensors. In some embodiments, automatic re-centering may be performed by generating state-control information. In some embodiments, the master earbud may transmit state-control information to the slave earbud so that the master earbud generates re-centering data utilized by both the master earbud and the slave earbud.

[0049] Figure 4 is a schematic block diagram illustrating components of an example data processing block according to some embodiments. In some embodiments, the data processing block and / or components of the data processing block may be implemented as software that may be executed by one or more processors and / or one or more control systems of a given earbud headphone. Examples of such processors or control systems are described in Figure 8 is shown in and described below in conjunction with this figure. Note that although Figure 4 Depicted as including a primary earbud headset (e.g. Figure 2 The head tracking block 208a of the master earbud 204a of the embodiment of the present invention may be included in the slave earbuds (e.g., Figure 2204b). As shown, the data processing block 302 may include a data fusion block 402, a data prediction block 404, and an automatic re-centering block 406. The data fusion block 202 may be configured to receive sensor data from the sensor 210a, which may be motion data (e.g., from one or more accelerometers and / or one or more gyroscopes), and transform the sensor data into head orientation data. In some embodiments, the head orientation data may be provided to the data prediction block 404. The data prediction block 404 may, for example, generate predicted head orientation data at one or more future times. The predicted head orientation data may be provided to the automatic re-centering block 406 and / or the data synchronization block 304. The automatic re-centering block 406 may be configured to generate state / control information that may be used to adjust the head orientation data to compensate for the drift of the sensor 210a. The re-centered head orientation data may be provided to the data synchronization block 304, which may synchronize the re-centered head orientation data with the head orientation data received from the paired earbuds. Note that the automatic re-centering block 406 may additionally receive data from the data synchronization block 304, such as head orientation data received from the paired earbuds.

[0050] In some embodiments, data synchronization can use a cache configured to store orientation data. Data can be placed in the cache and / or retrieved from the cache at various timestamps. The timestamps can depend on a data transmission rate (e.g., a data transmission rate associated with data transmission between two paired earbuds), a data transmission latency (e.g., a latency associated with data transmission between two paired earbuds), a data transmission jitter, and / or a data buffer delay. In some embodiments, data can be stored in the cache and / or retrieved from the cache in a different manner for a master earbud relative to a slave earbud. For example, for a master earbud, data can be placed in the cache at a time T L (0) is placed in the cache. Continuing with this example, the synchronization data can be placed in the cache at time T L The index n may be set to the audio processing block at time T (-n), where -n refers to n samples before time 0 at which the new data is placed in the buffer. In other words, data older than the current time may be transmitted to the audio processing block. The index n may be based on the data transmission rate, data transmission latency, and / or data buffer delay. The data to be synchronized at the paired earbuds may be transmitted at time T L(-k) is sent to the transport block at time T(-k), where -k refers to k samples before time 0 at which the new data is placed in the cache. The index k can be based on the time difference between data processing and data transmission, and / or a margin for data transmission jitter. Note that index k corresponds to the data of the paired earbuds, and the time difference between index n and index k corresponds to compensation for the data transmission delay between the paired earbuds. In some embodiments, both the master earbud and the slave earbud can process data for T(-k) substantially simultaneously. In some embodiments, index n is greater than index k because the nth data can be sent to the processing block at the current time, while the kth data can be sent to the paired earbuds and processed later. With reference to the slave earbuds, the cache can be processed at time T F Synchronous data (e.g., synchronized with data from the master earbud) can be received at time T F The data at index m may correspond to data generated m samples before the current time, and the data may be predicted for a time corresponding to the current time.

[0051] Figure 5A depicts example components of a data sync block for a master earbud, and Figure 5B Example components of a data synchronization block from an earbud are depicted. In some embodiments, the data synchronization block and / or components of the data synchronization block may be implemented as software that may be executed by one or more processors or one or more control systems of a given earbud. Examples of such processors or control systems are described in Figure 8 FIG. 1 is shown in FIG. 1 and described below in conjunction with the figure. Figure 5A , the data synchronization block 304a may include a data rate and delay block 502a and a buffer 504a. The predicted head orientation data may be sent from the data processing block 302a of the master earbud to the data rate and delay block 502a. The data processing block 302a may send the orientation data to the buffer 504a. L (0) head orientation data predicted at. Data rate and latency block 502a may generate control information for synchronizing data. The cache may include data stored in the cache at a previous (eg, earlier) time. For example, data stored in T L (-k+1) and / or T L The data of (-k) may be provided to data transmission block 306a for transmission to the paired slave earbuds. Note that the data may be resampled by resampler block 308 before transmission. As another example, the data stored in T L(-n+1) and / or T L The data of (-n) may be provided to the data processing block 302a for providing to the data processing block 302a.

[0052] refer to Figure 5B , depicts example components of a data synchronization block for a slave earbud, according to some embodiments. The data synchronization block 304b may include a data rate and latency block 502b and a buffer 504b. The data rate and latency block 502b may receive data from the data transfer block 306b, the data including head orientation data and / or predicted head orientation data from the master earbud. The data rate and latency block 502b may additionally receive predicted head orientation data from the data processing block 302b. New data may be received at time T F (0) is stored in cache 504b. The data can be retrieved from cache 504b and at an earlier timestamp (e.g., T F (-k+1) and / or T F (-k)) is transmitted to the data processing block 302b.

[0053] As described above, a pair of earbuds may be communicatively coupled, for example, using a wireless communication channel, such as a communication channel that complies with the Bluetooth protocol. Due to limited bandwidth, weak signals, and / or interference from nearby devices, there may be discarded data packets and / or delays in delivered data packets. In order to compensate for discarded data or delayed data, the master earbud may transmit timing information that can be used by the slave earbud to detect discarded packets. For example, the timing information may include packet counts or other counting information that can be used by the slave earbud to detect missing or delayed packets. Note that in some embodiments, the counting information may be used to check for discarded data packets, and the timestamp information may provide more accurate information for monitoring and / or generating time-based delays and / or sampling calculations. Additionally or alternatively, the timing information may include a timestamp, such as a timestamp at which the head orientation information is transmitted. Note that the timing information may be used by the slave earbud to synchronize the head orientation data. In some embodiments, in response to detecting discarded or delayed data, the slave earbud may compensate for the missing or delayed data. For example, the slave earbud may utilize the last received head orientation data, smooth the most recent head orientation data (e.g., the most recent N samples), resample the previously received head orientation data, or any combination thereof. This compensated orientation data may then be used for synchronization and / or processing.

[0054] Fig. 6A1 illustrates example components that may be used by a data transfer block of a master earbud to generate and utilize timing information in accordance with some embodiments. In some implementations, the data transfer block or components of the data transfer block may be implemented as software that may be executed by one or more processors and / or one or more control systems of a given earbud. Examples of such processors or control systems are described in Figure 8 306a. As shown in the figure and described in conjunction with the figure below. As shown, the data transmission block 306a may include a timestamp and count data generation block 610a and a transmission control block 612a. The timestamp and count data generation block 610a can receive the orientation data and / or control data generated by the data synchronization block 304a, and can provide timing information and / or counting information to the transmission control block 612a. The transmission control block 612a can then generate a data stream including both the head orientation data and the timing information and / or counting information generated by the master earbud headset, and cause the generated data stream to be transmitted to the slave earbud headset (e.g., via a Bluetooth communication channel or any other suitable wireless communication channel). The transmission control block 612a can trigger the transmission of the data stream at a predefined interval and / or within a predetermined timing.

[0055] Figure 6B 1 illustrates example components that may be used by a data transfer block of a slave earbud to receive and utilize timing information in accordance with some embodiments. In some implementations, the data transfer block or components of the data transfer block may be implemented as software that may be executed by one or more processors and / or one or more control systems of a given earbud. Examples of such processors or control systems are described in Figure 8 6 and described in conjunction with the figure below. As shown in the figure, the data transmission block 306b may include a timestamp and count data check block 610b and a missing and discarded data compensation block 614. The timestamp and count data check block 610b may receive a data stream from the master earbud headset, and the data stream may include head orientation data and timestamp and / or count data. Based on the timestamp and / or count data, the timestamp and count data check block 610b may detect discarded and / or delayed data packets of the head orientation data. The timestamp and count data check block 610b may send information indicating discarded and / or delayed data packets to the missing and discarded data compensation block 614. The missing and discarded data compensation block 614 may then generate replacement data (e.g., by utilizing previously received data, resampling previously received data, smoothing previously received data, or any combination thereof). The received and / or replacement data may be provided to the data synchronization block 304b so that the head orientation data from the master earbud headset is synchronized with the head orientation data generated by the slave earbud headset.

[0056] Figure 7700 is a flow chart of an example process 700 for generating and utilizing head orientation data by a pair of earbuds. Note that the blocks of process 700 may be performed by a processor or control system on each earbud in a pair of earbuds. Examples of such processors or control systems are described in Figure 8 As shown in and described below in conjunction with the figure. As described above, the pair of earbuds may include a master earbud and a slave earbud. In some embodiments, the blocks of process 700 may be in addition to Figure 7 In some implementations, two or more blocks of process 700 may be performed substantially in parallel. In some implementations, one or more blocks of process 700 may be omitted.

[0057] At 702, process 700 may begin by receiving sensor data from one or more sensors at each earbud in a pair of communicatively coupled earbuds. The one or more sensors may be disposed in or on each earbud. The one or more sensors may include one or more accelerometers, one or more gyroscopes, one or more magnetometers, and the like.

[0058] At 704, process 700 may determine head orientation information at each earbud. For example, process 700 may transform raw sensor data indicating motion into head orientation information using data fusion. Note that in some embodiments, the head orientation information may include head orientation prediction data indicating a predicted head orientation at one or more future time points. The predicted head orientation data may be used to compensate for missing or dropped data packets in the paired earbuds. In some embodiments, the head orientation information may be determined by a data processing block for each earbud. Example components of the data processing blocks are described in Figure 4 , and described above in conjunction with this figure.

[0059] At 706, process 700 can transmit the determined head orientation information between the pair of earbuds. For example, the master earbud can generate a data stream that includes head orientation information and timing information that can be used by the slave earbud to detect missing or delayed data. As another example, the slave earbud can send head orientation data to the master earbud. By sharing head orientation information between the pair of earbuds, more robust head orientation information can be determined, which in turn can allow for more robust rendering of audio content. The head orientation information can be sent from a given earbud using a data transfer block. Example components of the data transfer block are described in Fig. 6A (for primary earbuds) and Figure 6B (for earbud type headphones) and described above in conjunction with these two figures.

[0060] At 708, process 700 may synchronize head orientation data at each earbud based at least in part on timing information associated with a timestamp of transmitting the head orientation information. For example, each earbud may synchronize head orientation information generated based on a sensor disposed in or on the earbud itself with head orientation information received from a paired earbud. Synchronization may be performed by a data synchronization block. Example components of the data synchronization block are described in Figure 5A (for primary earbuds) and Figure 5B (for earbud type headphones) and described above in conjunction with these two figures.

[0061] At 710, process 700 can utilize the synchronized head orientation data to render audio content by each earbud. For example, process 700 can render audio stream data provided by a paired user device (e.g., a mobile phone, a tablet computer, a smart TV, a game console, a desktop computer, a laptop computer, etc.) based on the synchronized head orientation data. The audio stream can be rendered using spatial metadata associated with the audio stream data, the spatial metadata indicating how to render the audio stream using the head orientation data. The rendered audio stream can then be played using each of the earbuds.

[0062] Figure 8 is a block diagram showing an example of components of an apparatus capable of implementing various aspects of the present disclosure. Figure 1 Sample, Figure 8 The types and quantities of the elements shown in are provided only as examples. Other embodiments may include more, less and / or different types and quantities of elements. According to some examples, device 800 may be configured to perform at least some of the methods disclosed herein. In some embodiments, device 800 may be or may include one or more parts of a television, an audio system, a mobile device (such as a cellular phone), a laptop computer, a tablet device, an intelligent speaker or another type of device.

[0063] According to some alternative embodiments, the apparatus 800 may be or may include a server. In some such examples, the apparatus 800 may be or may include an encoder. Thus, in some instances, the apparatus 800 may be a device configured for use within an audio environment such as a home audio environment, whereas in other instances, the apparatus 700 may be a device configured for use in the "cloud", e.g., a server.

[0064] In this example, the device 800 includes an interface system 805 and a control system 810. In some embodiments, the interface system 805 can be configured to communicate with one or more other devices of the audio environment. In some examples, the audio environment can be a home audio environment. In other examples, the audio environment can be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. In some embodiments, the interface system 805 can be configured to exchange control information and associated data with the audio devices of the audio environment. In some examples, the control information and associated data can be related to one or more software applications being executed by the device 800.

[0065] In some embodiments, the interface system 805 may be configured to receive a content stream or to provide a content stream. The content stream may include audio data. The audio data may include but may not be limited to an audio signal. In some instances, the audio data may include spatial data such as channel data and / or spatial metadata. In some examples, the content stream may include video data and audio data corresponding to the video data.

[0066] The interface system 805 may include one or more network interfaces and / or one or more external device interfaces (e.g., one or more universal serial bus (USB) interfaces). According to some embodiments, the interface system 805 may include one or more wireless interfaces. The interface system 805 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 805 may include a control system 810 and a memory system (e.g., Figure 8 815). However, in some instances, the control system 810 may include a memory system. In some implementations, the interface system 805 may be configured to receive input from one or more microphones in the environment.

[0067] For example, control system 810 may include a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0068] In some embodiments, the control system 810 may reside in more than one device. For example, in some embodiments, a portion of the control system 810 may reside in a device within one of the environments described herein, and another portion of the control system 810 may reside in a device outside the environment, such as a server, a mobile device (e.g., a smart phone or a tablet computer), etc. In other examples, a portion of the control system 810 may reside in a device within an environment, and another portion of the control system 810 may reside in one or more other devices of the environment. For example, a portion of the control system 810 may reside in a device (e.g., a server) that implements a cloud-based service, and another portion of the control system 810 may reside in another device (e.g., another server, a memory device, etc.) that implements a cloud-based service. In some examples, the interface system 805 may also reside in more than one device. In some embodiments, a portion of the control system may reside in or on an earbud headset.

[0069] In some implementations, the control system 810 may be configured to at least partially perform the methods disclosed herein. According to some examples, the control system 810 may be configured to implement the following methods: synchronizing head tracking information, resampling data, predicting head tracking information, rendering audio data, and the like.

[0070] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transitory media may reside, for example, on a computer. Figure 8 810 and / or the control system 810 shown in FIG. 815. Thus, various innovative aspects of the subject matter described in this disclosure may be implemented in one or more non-transitory media having software stored thereon. The software may, for example, perform scene analysis, determine gain limits for different clusters, determine gains for different frequency bands, apply gains to audio signals to generate modified or enhanced audio signals, etc. For example, the software may be implemented by a processor such as a processor. Figure 8 The control system 810 and other control system components are executed.

[0071] In some examples, apparatus 800 may include Figure 8Optional microphone system 820 shown in . Optional microphone system 820 may include one or more microphones. In some embodiments, one or more microphones may be part of or associated with another device (such as a speaker of a speaker system, a smart audio device, etc.). In some examples, device 800 may not include microphone system 820. However, in some such embodiments, device 800 may still be configured to receive microphone data from one or more microphones in the audio environment via interface system 810. In some such embodiments, a cloud-based implementation of device 800 may be configured to receive microphone data or noise indicators corresponding at least in part to the microphone data from one or more microphones in the audio environment via interface system 810.

[0072] According to some embodiments, the apparatus 800 may include Figure 8 Optional loudspeaker system 825 is shown in . Optional loudspeaker system 825 may include one or more loudspeakers, which may also be referred to herein as "speakers," or more generally as "audio reproduction transducers." In some examples (e.g., cloud-based implementations), device 800 may not include loudspeaker system 825. In some implementations, device 800 may include headphones. The headphones may be connected or coupled to device 800 via a headphone jack or via a wireless connection (e.g., Bluetooth).

[0073] Some aspects of the present disclosure include a system or device configured (e.g., programmed) to perform one or more examples of the disclosed methods, and a tangible computer-readable medium (e.g., a disk) storing code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general-purpose processor, a digital signal processor, or a microprocessor that is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data asserted thereto.

[0074] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP), which is configured (e.g., programmed and otherwise configured) to perform the required processing on (multiple) audio signals, including the execution of one or more examples of the disclosed method. Alternatively, embodiments of the disclosed system (or its elements) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include an input device and a memory), which is programmed with software or firmware and / or otherwise configured to perform any of various operations, including one or more examples of the disclosed method. Alternatively, the elements of some embodiments of the system of the present invention are implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform one or more examples of the disclosed method, and the system also includes other elements (e.g., one or more loudspeakers and / or one or more microphones). The general-purpose processor configured to perform one or more examples of the disclosed method may be coupled to an input device (e.g., a mouse and / or keyboard), a memory, and a display device.

[0075] Another aspect of the present disclosure is a computer-readable medium (e.g., a disk or other tangible storage medium) storing code for executing one or more examples of the disclosed method or its steps (e.g., an encoder executable to execute one or more examples of the disclosed method or its steps).

[0076] Although specific embodiments of the present disclosure and applications of the present disclosure have been described herein, it will be apparent to those skilled in the art that many changes may be made to the embodiments and applications described herein without departing from the scope of the present disclosure as described and claimed herein. It should be understood that although certain forms of the present disclosure have been shown and described, the present disclosure is not limited to the specific embodiments described and shown or the specific methods described.

[0077] Various aspects of the present disclosure may be understood from the following enumerated example embodiments (EEE):

[0078] EEE 1. A method for processing audio, the method comprising:

[0079] receiving raw sensor data from one or more sensors in at least two wearable audio playback devices;

[0080] Fusing raw sensor data to generate head movement data;

[0081] predicting future head movement based on the head movement data to generate orientation data;

[0082] re-centering the audio scene using the synchronized orientation data or the re-centering state control data, the re-centering comprising at least one of avoiding drifting from the audio scene or re-centering the head orientation; and

[0083] providing the orientation data or the re-centered state control data for synchronization between a first device of the wearable audio playback device and a second device of the wearable audio playback device;

[0084] EEE 2. The method of EEE 1, wherein the wearable audio playback device is a wireless earbud, the first device is a master earbud, providing the orientation data or re-centering state control data for synchronization including prediction of future head movement data or future orientation data for compensating for transmission jitter, transmission delay and processing delay; the second device is a slave earbud; and

[0085] The orientation data or the re-centered status data is transmitted from the master earbud to the slave earbud.

[0086] EEE 3. The method as described in EEE 2, wherein predicting future head movement includes predicting at least one of transmission delay, correcting jitter, or adjusting head movement based on known transmission state information.

[0087] EEE 4. The method as described in EEE 3, wherein the known transmission status information includes at least one of a transmission interval, a data discard rate, or a buffer length.

[0088] EEE 5. The method of any one of EEE 1 to EEE 4, wherein providing the orientation data comprises indexing the orientation data based on a calculated data rate and a data transfer delay. ...

[0089] EEE 6. The method of EEE 5, wherein indexing the orientation data or the re-centered state data based on the calculated data rate and data transmission delay comprises:

[0090] Determining based on timestamps or data counts that certain data is missing or delayed during transmission; and

[0091] Compensate for missing data during indexing.

[0092] EEE 7. The method of EEE 6, wherein providing the re-centering status data comprises logic for determining an orientation of the virtual audio scene and a time for triggering the re-centering. ...

[0093] EEE 8. The method as described in any one of EEE 1 to EEE 6, wherein the dominant earbud earphone and the slave earbud earphone can be interchanged to achieve power consumption balance.

[0094] EEE 9. A system comprising:

[0095] A processor; and a computer-readable medium storing instructions, which, when executed by the processor, cause the processor to perform the operations of any one of EEE 1 to EEE 8.

[0096] EEE 10. A computer readable medium storing instructions, which when executed by a processor cause the processor to perform the operations of any one of EEE 1 to EEE 8.

Claims

1. A method for utilizing head tracking data, the method comprising: receiving sensor data from one or more sensors at each earbud of a pair of communicatively coupled earbuds; determining head orientation information at each earbud of the pair of communicatively coupled earbuds; transmitting the determined head orientation information between the pair of communicatively coupled earbuds such that a master earbud of the pair of communicatively coupled earbuds transmits the head orientation information determined by the master earbud to a slave earbud of the pair of communicatively coupled earbuds; synchronizing, at each earbud of the pair of communicatively coupled earbuds, the determined head orientation data based at least in part on timing information associated with a timestamp at which the head orientation information was transmitted; as well as The synchronized head orientation data is utilized to render audio content by each earbud of the pair of communicatively coupled earbuds.

2. The method of claim 1, wherein: Determining the head orientation information at each earbud includes fusing data from two or more sensors disposed in or on the earbud.

3. The method according to any one of claims 1 or 2, wherein: Determining the head orientation information includes determining predicted head orientation data.

4. The method of claim 3, wherein: The slave earbud is configured to utilize the predicted head orientation data to compensate for a delay in receiving the head orientation information from the master earbud.

5. The method of any one of claims 1 to 4, further comprising smoothing the synchronized head orientation data, wherein: The synchronized head orientation data used to render audio content includes smoothed synchronized head orientation data.

6. The method according to any one of claims 1 to 5, wherein: Transmitting the determined head orientation information between the pair of communicatively coupled earbuds includes including the timing information in a data stream transmitted from the master earbud to the slave earbud.

7. The method of claim 6, further comprising determining, at the slave earbud, whether data is missing in the head orientation information received from the master earbud based at least in part on the timing information received from the master earbud.

8. The method according to any one of claims 6 or 7, wherein: Synchronizing the determined head orientation data at the slave earbud includes synchronizing the determined head orientation data determined at the slave earbud based on one or more sensors of the slave earbud with the head orientation data received from the master earbud using the timing information.

9. The method according to any one of claims 1 to 8, wherein: Determining the head orientation information at each earbud includes re-centering head orientation data generated based on the sensor data to compensate for drift of the sensor data.

10. The method according to any one of claims 1 to 9, wherein: Each earbud comprises a buffer storing at least said head orientation data.

11. The method of claim 10, wherein: Head orientation data is transmitted from the buffer of the master earbud to the slave earbud at a timing interval dependent at least on a data transmission rate and latency associated with transmitting the determined head orientation information between the pair of communicatively coupled earbuds.

12. The method of any one of claims 1 to 11, further comprising resampling the sensor data before determining the head orientation information to compensate for a difference between a sampling rate used to acquire the sensor data and a sampling rate used to determine the head orientation information.

13. The method as described in any one of claims 1 to 12 further includes resampling the determined head orientation information before transmitting the determined head orientation information to compensate for the difference between the sampling rate used to determine the head orientation information and the sampling rate used to transmit the determined head orientation information.

14. The method of any one of claims 1 to 13, further comprising resampling the synchronized head orientation data before presenting the audio content using the synchronized head orientation data to compensate for a difference between a sampling rate used to synchronize the determined head orientation data and a sampling rate used to process the audio content.

15. The method according to any one of claims 1 to 14, wherein: Transmitting the determined head orientation information includes utilizing a Bluetooth communication protocol.

16. A system comprising: processor; as well as A computer readable medium storing instructions which, when executed by the processor, cause the processor to perform the operations of any one of claims 1 to 15.

17. A computer readable medium storing instructions, which when executed by a processor cause the processor to perform the operations of any one of claims 1 to 15.

Images