Head tracking data synchronization

Abstract

Methods, systems, and media for utilizing head tracking data are provided. In some embodiments, the method includes receiving sensor data from one or more sensors at each earbud of a pair of communicatively coupled earbuds. The method may include determining head orientation information at each earbud of the pair of communicatively coupled earbuds. The method may include transmitting the determined head orientation information between the pair of communicatively coupled earbuds, such that the leader earbud transmits the head orientation information determined by the leader earbud to the follower earbud. The method may include synchronizing the determined head orientation data at each earbud based at least in part on timing information associated with a timestamp at which the head orientation information was transmitted. The method may include utilizing the synchronized head orientation data to present audio content via each earbud of the pair of communicatively coupled earbuds.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of priority to PCT Application No. PCT / CN2022 / 118753, filed September 14, 2022, and U.S. Provisional Application No. 63 / 492,724, filed March 28, 2023, each of which is incorporated herein by reference in its entirety.

[0002] [Technical field] The present disclosure relates to systems, methods, and media for synchronization of head-tracking related data. [Background technology]

[0003] Listeners of audio content may be interested in an immersive listening experience in which audio content is rendered based on head orientation. However, accurately monitoring head orientation and rendering audio content based on head orientation can be difficult.

[0004] [Notation and Nomenclature] Throughout this disclosure, including the claims, the terms "speaker," "loudspeaker," and "audio reproduction transducer" are used interchangeably to refer to any sound-emitting transducer (or set of transducers). A typical headphone set includes two speakers. A speaker may be implemented to include multiple transducers (e.g., woofers and tweeters) that may be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker feeds may undergo different processing in different circuit branches coupled to different transducers.

[0005] Throughout this disclosure, including the claims, the phrase performing an operation "on" a signal or data (e.g., filtering, scaling, transforming, or applying a gain to a signal or data) is used broadly to indicate performing an operation on the signal or data directly, or on a processed version of the signal or data (e.g., on a version of the signal that has undergone pre-filtering or pre-processing before the operation is performed).

[0006] Throughout this disclosure, including the claims, the term "system" is used broadly to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M inputs and the other XM inputs are received from external sources) may also be referred to as a decoder system.

[0007] Throughout this disclosure, including the claims, the term "processor" is used broadly to denote a system or device that is programmable or configurable (e.g., with software or firmware) to perform operations on data (e.g., audio, video, or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and / or otherwise configured to perform pipeline processing on audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets. Summary of the Invention [Means for solving the problem]

[0008] Methods, systems, and media for utilizing head tracking data are provided. In some embodiments, the method includes receiving sensor data from one or more sensors at each earbud of a pair of communicatively coupled earbuds. The method may include determining head orientation information at each earbud of the pair of communicatively coupled earbuds. The method may include transmitting the determined head orientation information between the pair of communicatively coupled earbuds, such that a leader earbud of the pair of communicatively coupled earbuds transmits the head orientation information determined by the leader earbud to a follower earbud of the pair of communicatively coupled earbuds. The method may include synchronizing the determined head orientation data at each earbud of the pair of communicatively coupled earbuds based at least in part on timing information associated with a timestamp at which the head orientation information was transmitted. The method may include utilizing the synchronized head orientation data to present audio content via each earbud of the pair of communicatively coupled earbuds.

[0009] In some embodiments, determining head orientation information at each earbud includes fusing data from two or more sensors located in or on the earbud.

[0010] In some examples, determining the head orientation information includes determining predicted head orientation data, and in some embodiments, the follower earbuds are configured to utilize the predicted head orientation data to compensate for latency in receiving head orientation information from the leader earbud.

[0011] In some examples, the method further includes smoothing the synchronized head orientation data, wherein the synchronized head orientation data utilized to present the audio content comprises the smoothed synchronized head orientation data.

[0012] In some examples, transmitting the determined head orientation information between the pair of communicatively coupled earbuds includes including timing information in a data stream transmitted from the leader earbud to the follower earbud. In some embodiments, the method further includes determining, at the follower earbud, whether data is missing in the head orientation information received from the leader earbud based at least in part on the timing information received from the leader earbud. In some embodiments, synchronizing the determined head orientation data at the follower earbud includes synchronizing the determined head orientation data determined at the follower earbud based on one or more sensors in the follower earbud with the head orientation data received from the leader earbud using the timing information.

[0013] In some examples, at each earbud, determining the head orientation information includes recentering the head orientation data generated based on the sensor data to compensate for drift in the sensor data.

[0014] In some examples, each earbud includes a cache buffer for storing at least the head orientation data, and in some examples, the head orientation data is transmitted from the cache buffer of the leader earbud to the follower earbud at timing intervals that depend at least on the data transmission rate and latency associated with transmitting the determined head orientation information between a pair of communicatively coupled earbuds.

[0015] In some examples, the method further includes resampling the sensor data before determining the head orientation information to compensate for a difference between a sampling rate used to acquire the sensor data and a sampling rate used to determine the head orientation information.

[0016] In some examples, the method further includes resampling the determined head orientation information before transmitting the determined head orientation information to compensate for a difference between a sampling rate used to determine the head orientation information and a sampling rate used to transmit the determined head orientation information.

[0017] In some examples, the method further includes resampling the synchronized head orientation data before utilizing the synchronized head orientation data to present the audio content to compensate for a difference between a sampling rate used to synchronize the determined head orientation data and a sampling rate used to process the audio content.

[0018] In some examples, transmitting the determined head orientation information includes utilizing a BLUETOOTH® communication protocol.

[0019] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including, but not limited to, random access memory (RAM) devices, read-only memory (ROM) devices, etc. Accordingly, some inventive aspects of the subject matter described in this disclosure may be implemented via one or more non-transitory media having software stored thereon.

[0020] At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices may be capable of at least partially performing the methods disclosed herein. In some implementations, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single- or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or combinations thereof.

[0021] The details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. It should be noted that the relative dimensions of the following figures may not be drawn to scale. Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [Brief explanation of the drawings]

[0022] [Figure 1] 1 illustrates a pair of earbuds according to some embodiments.

[0023] [Figure 2] FIG. 1 is a schematic block diagram of a system for synchronizing head tracking data according to some embodiments.

[0024] [Figure 3] FIG. 1 is a schematic block diagram of components of an earbud according to some embodiments.

[0025] [Figure 4] FIG. 1 is a schematic block diagram of an exemplary system for determining head tracking information according to some embodiments.

[0026] [Figure 5A] FIG. 1 is a schematic block diagram of an exemplary data synchronization system for reader earbuds, according to some implementations. [Figure 5B] FIG. 1 is a schematic block diagram of an exemplary data synchronization system for follower earbuds, according to some implementations.

[0027] [Figure 6A] FIG. 1 is a schematic block diagram of an exemplary data transfer system for a reader earbud, according to some implementations. [Figure 6B] FIG. 1 is a schematic block diagram of an exemplary data transfer system for follower earbuds, according to some implementations.

[0028] [Figure 7] 1 is a flowchart of an example process for synchronizing and utilizing head orientation information by a pair of earbuds, according to some implementations.

[0029] [Figure 8] FIG. 1 shows a block diagram illustrating example components of an apparatus capable of implementing various aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0030] Like reference numbers and designations in the various drawings indicate like elements.

[0031] Audio content consumers are increasingly interested in immersive audio content. Immersive audio content may be audio content that includes spatial positioning information (e.g., audio objects that are to be perceived as located at a specific spatial location). When rendered by headphones, such immersive content may be rendered based on the head orientation of a listener wearing the headphones. As used herein, "immersive content" refers to audio data that is rendered with reference to an external reference frame, such as a reference frame defined by a display screen presenting a movie, television program, or the like. Traditionally, over-the-head headphones may include one or more processors and one or more sensors used to generate head orientation data from which audio content is rendered. However, with earbuds, each earbud of a pair of earbuds is physically separate, and rendering based on head orientation may be more difficult. For example, each earbud may have its own set of sensors that generate sensor data from which head orientation is determined. However, by having two physically separate earbuds, the head orientation determined by each earbud may be different, perhaps even significantly different, thereby causing errors or inaccuracies in the rendered audio content.

[0032] Disclosed herein are systems, methods, and techniques for synchronizing head orientation data and status and control information between a pair of earbuds. Each earbud may have its own set of sensors, which may include one or more accelerometers, one or more gyroscopes, one or more magnetometers, etc. Each earbud may then generate head orientation data based on the sensor data. The head orientation data may be shared between the pair of earbuds using a wireless communication channel. For example, the wireless communication channel may utilize the BLUETOOTH® communication protocol or any other suitable wireless communication protocol. Each earbud may then synchronize the head orientation data generated by itself with the head orientation data received by the paired earbud. This allows for more robust determination of head orientation, which in turn allows for more robust rendering of audio content. It should be noted that, as used herein, status / control information may include timing information (e.g., timestamp information, count information, and any other timing-related information) that may be used for latency compensation (e.g., to compensate for transfer latency of transferring data between two paired earbuds). Additionally or alternatively, status / control information may include data generated in a first earbud and transmitted to a paired earbud configured for use in recentering head orientation data (e.g., to account for and / or compensate for system drift associated with the sensors). The data may be used, for example, by the recentering block shown in and described below in connection with FIG. 4.

[0033] In some implementations, the first earbud of the pair may be designated as the leader earbud, and the second earbud of the pair may be designated as the follower earbud. The leader earbud may generate timestamp or count data that may be included in the data stream transmitted to the follower earbud. In some embodiments, the follower earbud may utilize the timestamp or count data to detect missing or delayed data (e.g., due to weak signal strength in the wireless communication channel, interference from nearby devices, etc.) and may then compensate for the missing or delayed data. In some embodiments, each earbud may be configured to generate predicted head orientation data, e.g., a prediction of the listener's future head orientation. The predicted head orientation data may be used to compensate for missing or delayed head orientation data from paired earbuds.

[0034] FIG. 1 illustrates an exemplary listening scenario, according to some embodiments. As illustrated, a listener 100 may be listening to audio content using a pair of earbuds, which may include a left earbud 102a and a right earbud 102b. The two earbuds may be communicatively coupled via a wireless communication channel 106. In one example, the wireless communication channel may be a BLUETOOTH® communication channel, as shown in FIG. 1. Each earbud may include hardware circuitry, which may include one or more sensors configured to provide sensor data indicative of a user's head orientation. For example, the one or more sensors may include one or more accelerometers, one or more gyroscopes, one or more magnetometers, etc. The circuitry of each earbud may further include one or more processors configured to process the sensor data, determine head orientation based on the sensor data, predict future head orientation based on the sensor data, send and / or receive communications from a paired mobile device and / or other earbuds, etc. In the example of FIG. 1, left earbud 102a includes circuitry 104a and right earbud 102b includes circuitry 104b.

[0035] In some implementations, each earbud of a pair of earbuds may include one or more sensors configured to provide sensor data that can be used to determine head orientation. The two earbuds may be communicatively coupled (e.g., via a wireless communication channel) such that the two earbuds may share information associated with head orientation, timing information, status and control information, etc. Each earbud may also be communicatively coupled (e.g., via a wireless communication channel) to a user device that provides audio content. Each earbud may be configured to render audio content received from a paired user device using the head orientation information and then play the rendered audio content by outputting the rendered audio signal. Note that in some implementations, one of the earbuds of a pair of earbuds may be considered a leader, and the other earbud may be considered a follower. The leader earbud may provide instructions, timing information, etc. to the follower earbud. The follower earbud may synchronize head orientation information obtained using its own sensors with head orientation information obtained from the leader earbud. This synchronization can be used to overcome differences between the sensors in each of the two earbuds, jitter or communication latency between the two earbuds and / or the mobile device, etc.

[0036] 2 is a schematic diagram of an exemplary system for synchronizing head orientation and utilizing synchronized head orientation information, according to some embodiments. As shown, a user device 202 provides audio stream data to a leader earbud 204a and a follower earbud 204b. While FIG. 2 shows the leader earbud 204a as the left earbud and the follower earbud 204b as the right earbud, it should be noted that in other embodiments, the leader earbud may be the right earbud and the follower earbud may be the left earbud. The audio stream data may be audio content for a movie or television program, audio content associated with music, audio content associated with a podcast, etc. The audio stream data may include one or more audio objects to be rendered at a particular spatial location relative to the user's head based on the user's head orientation data.

[0037] Each earbud may include an audio processing block configured to receive audio stream data from the user device 202 and render the audio stream data based on the head orientation data. The audio processing block may then be configured to play the rendered audio data. As shown, the leader earbud 204a includes an audio processing block 206a, and the follower earbud 204b includes an audio processing block 206b. As used herein, it should be understood that a “block” may be implemented in software, for example, as one or more functions. The software may be executed by one or more processors or one or more control systems, each located in or on a given earbud, in or on a user device paired with the earbud, etc. Examples of processors or control systems that may be used to implement a “block” are shown in and described below in connection with FIG. 8. In the example shown in FIG. 2, control system 250a is configured to implement audio processing block 206a, and control system 250b is configured to implement audio processing block 206b. Control systems 250a and 250a are instances of control system 810 described herein with reference to FIG.

[0038] Each audio processing block is configured to receive head orientation data from a corresponding head tracking block. For example, audio processing block 206a receives head orientation data from head tracking block 208a, and audio processing block 206b receives head orientation data from head tracking block 208b. Each head tracking block is configured to generate head orientation data based on sensor data received from one or more sensors. In the example shown in FIG. 2, control system 250a is configured to implement head tracking block 206a, and control system 250b is configured to implement head tracking block 206b. Note that each earbud has its own set of one or more sensors located within or on the earbud. For example, leader earbud 204a may utilize sensor set 210a, and follower earbud 204b may utilize sensor set 210b. The sensor sets may include one or more accelerometers, one or more magnetometers, one or more gyroscopes, etc.

[0039] In some implementations, each earbud may obtain sensor data from one or more sensors disposed in or on the earbud and may use the sensor data to determine the head orientation of the earbud wearer. The head orientation data may be determined by one or more processors in the earbud. The head orientation data may be shared with paired earbuds. For example, a left earbud may share head orientation data with its paired right earbud, and vice versa. The head orientation data may be shared between two paired earbuds via a wireless communication channel. The shared head orientation data may be used to determine a more robust head orientation (e.g., by using multiple sensors on two different earbuds), thereby enabling more robust rendering of audio data. Note that in some implementations, each earbud may determine predicted head orientation data that may be used to compensate for dropped data packets and / or latency when receiving shared head orientation data from its paired earbud. For example, predicted head orientation data determined in a first earbud may be used in instances where head orientation data from a second paired earbud is delayed or dropped. Data between the two paired earbuds may be synchronized. The data may be synchronized using timing information shared between the two paired earbuds. For example, the timing information may be shared by adding timestamp data to a data stream that includes the head orientation data. In some embodiments, the synchronized head orientation data may be used to recenter control data used to render audio content. This may compensate for drift in sensor data that causes drift in the head orientation data.

[0040] Note that in some embodiments, different aspects of the head orientation data synchronization and utilization system may operate at different sampling frequencies. For example, sensor data may be collected by a sensor with a sensor driver operating at a particular sensor sampling frequency (e.g., 100 Hz, 104 Hz, 110 Hz, etc.), while head tracking orientation determination may be performed using a processor operating at a different rate (e.g., 40 Hz, 50 Hz, 60 Hz, etc.). As another example, audio processing may be performed within a range of approximately 90 Hz to 120 Hz. In one example, audio processing may be performed at 93.75 Hz (e.g., by processing 512 samples at a 48 KHz sampling rate). As yet another example, data may be transferred between two earbuds at a data communication rate of 20 Hz. To account for different sampling rates between different subsystems, each earbud may include a resampling block configured to resample data from a first sampling rate to a second sampling rate. The resampling block may be executed by one or more processors in the earbud.

[0041] FIG. 3 is a schematic block diagram of components of an example earbud. While FIG. 3 illustrates the earbud as earbud 204a, which corresponds to the leader earbud shown in and described above in connection with FIG. 2, it should be noted that the same or similar components may be included in a follower earbud, such as follower earbud 204b of FIG. 2. In some implementations, various components may be implemented in software and executed by one or more processors or one or more control systems located in or on the earbud. An example of such a processor or control system is shown in and described below in connection with FIG. 8. As shown in FIG. 3, earbud 204a includes a head tracking block 208a configured to determine head orientation data based on data from sensor 210a. Head tracking block 208a provides the head orientation data to audio processing block 206a, which sends an indication of the head orientation data to the paired earbud (which may be follower earbud 204b of FIG. 2).

[0042] As shown in Figure 3, head orientation data may be determined by a data processing block 302 that receives sensor data and determines head orientation data. More detailed techniques that may be utilized by data processing block 302 are shown in and described below in connection with Figure 4.

[0043] The data processing block 302 communicates with a data synchronization block 304. As shown, the data synchronization block 304 also communicates with a data transfer block 306. With respect to outbound data, the data synchronization block 304 may generate a data stream including the head orientation data (and predicted head orientation data, and / or status and control information) generated by the data processing block 302 and apply timestamp data to the data stream. The head orientation data, predicted head orientation data, and status and control information timestamp data may be provided to the data transfer block 306. With respect to inbound data, the data synchronization block 304 may receive head orientation data and predicted head orientation data from the data transfer block 306 (which in turn receives data from paired earbuds) and may synchronize the data from the paired earbuds with head orientation data obtained using sensors in a given earbud. The synchronized data and status / control information may be provided to the data processing block 302 for use by the audio processing block 206a. More detailed techniques that may be implemented by data synchronization block 304 are shown in and described below in connection with Figure 5A (for the leader earbud) and Figure 5B (for the follower earbud).

[0044] The data transfer block 306 may transmit head orientation data, predicted head orientation data, status / control information, and timestamp information to the paired earbuds. The data transfer block 306 may also receive head orientation data, predicted head orientation data, and timestamp information from the paired earbuds. More detailed techniques that may be implemented by the data transfer block 306 are shown in and described below in connection with Figure 6A (for the leader earbud) and Figure 6B (for the follower earbud).

[0045] As shown, the system includes a resampler block 308 configured to convert data from a first sampling rate to a second sampling rate. The resampler block 308 may be used by any of the data processing block 302, the data synchronization block 304, the data transfer block 306, or the audio processing block 206a to convert from one sampling rate to another.

[0046] In some implementations, one or more processors in the earbuds may receive sensor data from one or more sensors. The sensor data may include, for example, motion data from one or more accelerometers and / or one or more gyroscopes. The one or more processors may use data fusion to convert the motion data into head orientation data. The one or more sensors may further predict future head orientation data. The predicted head orientation data may be used, for example, to compensate for latency in receiving head orientation data from paired earbuds. The one or more processors may perform auto-recentering to correct for drift in predictions of the motion data provided by the one or more sensors. In some implementations, auto-recentering may be performed by generating status control information. In some embodiments, the leader earbud may send status control information to the follower earbuds so that the leader earbuds generate recentering data utilized by both the leader earbud and the follower earbuds.

[0047] FIG. 4 is a schematic block diagram illustrating components of an example data processing block according to some implementations. In some implementations, the data processing block and / or its components may be implemented as software that may be executed by one or more processors and / or one or more control systems of a given earbud. An example of such a processor or control system is shown in FIG. 8 and described below in connection with that figure. Note that while FIG. 4 is shown as including a head tracking block 208a of a leader earbud (e.g., leader earbud 204a of FIG. 2), the same or similar components may be included in the head tracking block of a follower earbud (e.g., follower earbud 204b of FIG. 2). As shown, the data processing block 302 may include a data fusion block 402, a data prediction block 404, and an auto-recentering block 406. The data fusion block 202 may be configured to receive sensor data from sensors 210a, which may be motion data (e.g., from one or more accelerometers and / or one or more gyroscopes), and convert the sensor data into head orientation data. In some implementations, the head orientation data may be provided to a data prediction block 404. The data prediction block 404 may, for example, generate predicted head orientation data for one or more future times. The predicted head orientation data may be provided to an auto-recentering block 406 and / or a data synchronization block 304. The auto-recentering block 406 may be configured to generate status / control information usable to adjust the head orientation data to compensate for drift of the sensor 210a. The recentered head orientation data may be provided to a data synchronization block 304, which may synchronize the recentered head orientation data with head orientation data received from a paired earbud. Note that the auto-recentering block 406 may additionally receive data from the data synchronization block 304, e.g., head orientation data received from a paired earbud.

[0048] In some implementations, data synchronization may use a cache buffer configured to store orientation data. Data may be placed in and / or retrieved from the cache buffer at various timestamps. The timestamps may depend on a data transmission rate (e.g., a data transmission rate associated with data transmission between two paired earbuds), a data transmission latency (e.g., a latency associated with data transmission between two paired earbuds), data transmission jitter, and / or a data buffer delay. In some embodiments, data may be stored in and / or retrieved from the cache buffer in a different manner for a leader earbud than for a follower earbud. For example, for a leader earbud, data may be stored in and / or retrieved from the cache buffer at time T L Continuing with this example, the synchronized data can be placed in the cache buffer at time T L The index n may be set to the audio processing block at (-n), where -n refers to n samples before time 0 when new data was cached in the cache buffer. That is, data older than the current time may be sent to the audio processing block. The index n may be based on the data transmission rate, data transmission latency, and / or data buffer delay. The data synchronized in the paired earbuds may be set to T LThe new data may be sent to the transport block at time T(-k), where -k refers to k samples before time 0 that the new data was put into the cache buffer. Index k may be based on the time difference between data processing and data transmission, and / or headroom for data transmission jittering. Note that index k corresponds to the data of the paired earbuds, and the time difference between index n and index k corresponds to compensation for data transmission latency between the paired earbuds. In some embodiments, both the leader and follower earbuds may process data for T(-k) substantially simultaneously. In some embodiments, index n is greater than index k, because the nth data may be sent to the processing block at the current time, and the kth data may be sent to the paired earbud for processing at a later time. Referring to the follower earbud, the cache buffer is T F The data may be received at time T(0). The synchronized data (e.g., synchronized with data from the reader earbuds) may be received at time T F (-m), where index m may be based on a data transmission rate, a data transmission latency, a data transmission jitter, and / or a data buffer delay. Note that the data at index m corresponds to data generated m samples before the current time and is predicted for a time corresponding to the current time.

[0049] FIG. 5A illustrates exemplary components of a data synchronization block in a leader earbud, and FIG. 5B illustrates exemplary components of a data synchronization block in a follower earbud. In some embodiments, the data synchronization block and / or components of the data synchronization block may be implemented as software that may be executed by one or more processors or one or more control systems of a given earbud. An example of such a processor or control system is shown in FIG. 8 and described below in connection with that figure. Referring to FIG. 5A, the data synchronization block 304a includes a data rate and latency block 502a and a cache buffer 504a. Predicted head orientation data may be sent from the data processing block 302a of the leader earbud to the data rate and latency block 502a. The data processing block 302a may send the orientation data to the cache buffer 504a. T L The predicted head orientation data at (0). The data rate and latency block 502a may generate control information used to synchronize the data. The cache buffer may contain data stored in the cache buffer at a previous (e.g., older) time. For example, T L (-k+1) and / or T L The data stored in (-k) may be provided to the data forwarding block 306a for transmission to the paired follower earbud. Note that before forwarding, the data may be resampled by the resampler block 308. As another example, T L (-n+1) and / or T L The data stored in (-n) may be provided to the data processing block 302a for presentation to the data processing block 302a.

[0050] 5B, exemplary components of a data synchronization block of a follower earbud are shown, according to some implementations. The data synchronization block 304b may include a data rate and latency block 502b and a cache buffer 504b. The data rate and latency block 502b may receive data from the data transfer block 306b, including head orientation data and / or predicted head orientation data from the leader earbud. The data rate and latency block 502b may further receive predicted head orientation data from the data processing block 302b. New data is received at time T F The data may be stored in the cache buffer 504b at (0). The data may be retrieved from the cache buffer 504b and stored at an earlier timestamp, e.g., T F (-k+1) and / or T F (-k) to the data processing block 302b.

[0051] As described above, a pair of earbuds may be communicatively coupled using a wireless communication channel, such as a communication channel conforming to the BLUETOOTH® protocol. Due to limited bandwidth, a weak signal, and / or interference from nearby devices, there may be latency in dropped and / or delivered data packets. To compensate for dropped or delayed data, the leader earbud may transmit timing information that can be used by the follower earbud to detect dropped packets. For example, the timing information may include packet count or other count information that can be used by the follower earbud to detect missing or delayed packets. Note that in some embodiments, the count information may be used to check for dropped data packets, while the timestamp information may provide more accurate information for monitoring and / or generating time-based calculations of latency and / or sampling. The timing information may additionally or alternatively include a timestamp, e.g., a timestamp of when the head orientation information was transmitted. Note that this timing information may be used by the follower earbuds to synchronize head orientation data. In some implementations, in response to detecting dropped or delayed data, the follower earbud may compensate for the missing or delayed data. For example, the follower earbud may utilize the last received head orientation data, smoothed recent head orientation data (e.g., the most recent N samples), resampling of previously received head orientation data, or any combination thereof. This compensated orientation data may then be used for synchronization and / or processing.

[0052] FIG. 6A shows example components that may be used by a data transfer block of a leader earbud to generate and utilize timing information, according to some embodiments. In some embodiments, the data transfer block, or components of the data transfer block, may be implemented as software that may be executed by one or more processors and / or one or more control systems of a given earbud. An example of such a processor or control system is shown in and described below in connection with FIG. 8. As shown, the data transfer block 306a includes a timestamp and counting data generation block 610a and a transfer control block 612a. The timestamp and count data generation block 610a may receive the orientation data and / or control data generated by the data synchronization block 304a and may provide the timing information and / or count information to the transfer control block 612a. The transfer control block 612a may then generate a data stream including both the head orientation data and the timing information and / or count information generated by the leader earbud and cause the generated data stream to be transmitted to the follower earbud (e.g., via a BLUETOOTH® communication channel or any other suitable wireless communication channel). The transfer control block 612a may trigger the transmission of the data stream at predefined intervals and / or within predetermined timing.

[0053] FIG. 6B shows example components that may be used by a data transfer block of a follower earbud to receive and utilize timing information, according to some embodiments. In some embodiments, the data transfer block, or components of the data transfer block, may be implemented as software that may be executed by one or more processors and / or one or more control systems of a given earbud. An example of such a processor or control system is shown in FIG. 8 and described below in connection with that figure. As shown, the data transfer block 306b includes a timestamp and count data check block 610b and a missing and dropped data compensation block 614. The timestamp and count data check block 610b may receive a data stream from the leader earbud that may include head orientation data and timestamp and / or count data. Based on the timestamp and / or count data, the timestamp and count data check block 610b may detect dropped and / or delayed data packets of head orientation data. The timestamp and count data check block 610b may send information indicative of the dropped and / or delayed data packets to the missing and dropped data compensation block 614. The missing and dropped data compensation block 614 may then generate replacement data (e.g., by utilizing previously received data, resampling previously received data, smoothing previously received data, or any combination thereof). The received and / or replacement data may be provided to the data synchronization block 304b to synchronize the head orientation data from the leader earbud with the head orientation data generated by the follower earbud.

[0054] FIG. 7 is a flowchart illustrating an example process 700 for generating and utilizing head orientation data by a pair of earbuds. Note that the blocks of process 700 may be performed by a processor or control system on each earbud of the pair. An example of such a processor or control system is shown in FIG. 8 and described below in connection with that figure. As mentioned above, the pair of earbuds may include a leader earbud and a follower earbud. In some embodiments, the blocks of process 700 may be performed in an order other than that shown in FIG. 7. In some implementations, two or more blocks of process 700 may be performed substantially in parallel. In some implementations, one or more blocks of process 700 may be omitted.

[0055] At 702, process 700 may begin by receiving sensor data from one or more sensors in each earbud of a pair of communicatively coupled earbuds. The one or more sensors may be located in or on each earbud. The one or more sensors may include one or more accelerometers, one or more gyroscopes, one or more magnetometers, etc.

[0056] At 704, process 700 may determine head orientation information in each earbud. For example, process 700 may use data fusion to convert raw sensor data indicative of movement into head orientation information. Note that in some embodiments, the head orientation information may include head orientation prediction data indicative of predicted head orientation at one or more future points in time. The predicted head orientation data may be used to compensate for missing or dropped data packets from a paired earbud. In some embodiments, the head orientation information may be determined by a data processing block in each earbud. Exemplary components of the data processing block are shown in FIG. 4 and described above in connection with that figure.

[0057] At 706, process 700 may transmit the determined head orientation information between the pair of earbuds. For example, the leader earbud may generate a data stream including head orientation information and timing information that can be used by the follower earbud to detect missing or delayed data. As another example, the follower earbud may transmit head orientation data to the leader earbud. By sharing head orientation information between the pair of earbuds, more robust head orientation information can be determined, which may enable more robust rendering of audio content. The head orientation information can be transmitted from a given earbud using a data transfer block. Exemplary components of the data transfer block are shown in FIG. 6A (for the leader earbud) and FIG. 6B (for the follower earbud) and described above in connection therewith.

[0058] At 708, process 700 may synchronize head orientation data at each earbud based at least in part on timing information associated with the timestamp at which the head orientation information was transmitted. For example, each earbud may synchronize head orientation information generated based on sensors located in or on the earbud itself with head orientation information received from the paired earbud. The synchronization may be performed by a data synchronization block. Exemplary components of the data synchronization block are shown in and described above in connection with FIG. 5A (for the leader earbud) and FIG. 5B (for the follower earbud).

[0059] At 710, process 700 may utilize the synchronized head orientation data to present audio content through each earbud. For example, process 700 may render audio stream data provided by a paired user device (e.g., a mobile phone, a tablet computer, a smart TV, a game console, a desktop computer, a laptop computer, etc.) based on the synchronized head orientation data. The audio stream may be rendered using spatial metadata associated with the audio stream data that indicates how the head orientation data should be used to render the audio stream. The rendered audio stream may then be played using each of the earbuds.

[0060] 8 is a block diagram illustrating example components of a device capable of implementing various aspects of the present disclosure. As with other figures provided herein, the types and number of elements shown in FIG. 8 are given merely as examples. Other implementations may include more, fewer, and / or different types and numbers of elements. According to some examples, device 800 may be configured to perform at least some of the methods disclosed herein. In some implementations, device 800 may be or include a television, one or more components of an audio system, a mobile device (such as a cellular phone), a laptop computer, a tablet device, a smart speaker, or other type of device.

[0061] According to some alternative implementations, apparatus 800 may be or include a server. In some such examples, apparatus 800 may be or include an encoder. Thus, in some instances, apparatus 800 may be a device configured for use in an audio environment, such as a home audio environment, while in other instances, apparatus 700 may be a device configured for use in the “cloud,” e.g., a server.

[0062] In this example, device 800 includes an interface system 805 and a control system 810. Interface system 805, in some implementations, may be configured to communicate with one or more other devices in an audio environment. The audio environment, in some examples, may be a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, an automobile environment, a train environment, a street or sidewalk environment, a park environment, etc. Interface system 805, in some implementations, may be configured to exchange control information and associated data with audio devices in the audio environment. The control information and associated data, in some examples, may relate to one or more software applications running by device 800.

[0063] The interface system 805, in some implementations, may be configured to receive or provide a content stream. The content stream may include audio data. The audio data may include, but is not limited to, an audio signal. In some instances, the audio data may include spatial data, such as channel data and / or spatial metadata. In some examples, the content stream may include video data and audio data corresponding to the video data.

[0064] The interface system 805 may include one or more network interfaces and / or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some implementations, the interface system 805 may include one or more wireless interfaces. The interface system 805 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 805 may include one or more interfaces between the control system 810 and a memory system, such as the optional memory system 815 shown in FIG. 8 . However, the control system 810 may include a memory system in some instances. The interface system 805, in some implementations, may be configured to receive input from one or more microphones in the environment.

[0065] The control system 810 may include, for example, a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0066] In some implementations, control system 810 may reside in more than one device. For example, in some implementations, a portion of control system 810 may reside in a device within one of the environments described herein, while another portion of control system 810 may reside in a device outside the environment, such as a server, a mobile device (e.g., a smartphone or tablet computer), etc. In other embodiments, a portion of control system 810 may reside in a device within one environment, while another portion of control system 810 may reside in one or more other devices within the environment. For example, a portion of control system 810 may reside in a device implementing a cloud-based service, such as a server, while another portion of control system 810 may reside in another device implementing the cloud-based service, such as another server, a memory device, etc. Interface system 805 may also reside in more than one device, in some examples. In some implementations, a portion of the control system may reside in or on an earbud.

[0067] In some implementations, control system 810 may be configured to perform, at least in part, the methods disclosed herein. According to some examples, control system 810 may be configured to perform methods for synchronizing head tracking information, resampling data, predicting head tracking information, rendering audio data, etc.

[0068] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including, but not limited to, random access memory (RAM) devices, read-only memory (ROM) devices, etc. The one or more non-transitory media may reside, for example, in optional memory system 815 and / or control system 810 shown in FIG. 8. Accordingly, various inventive aspects of the subject matter described in this disclosure may be implemented in one or more non-transitory media having software stored thereon. The software may, for example, perform scene analysis, determine gain limits for different clusters, determine gains for different frequency bands, apply gains to audio signals to generate modified or enhanced audio signals, etc. The software may be executable by one or more components of a control system, such as, for example, control system 810 of FIG. 8.

[0069] In some examples, device 800 may include optional microphone system 820 shown in FIG. 8. Optional microphone system 820 may include one or more microphones. In some implementations, one or more of the microphones may be part of or associated with another device, such as a speaker of a speaker system, a smart audio device, or the like. In some examples, device 800 may not include microphone system 820. However, in some such implementations, device 800 may still be configured to receive microphone data for one or more microphones in the audio environment via interface system 810. In some such implementations, a cloud-based implementation of device 800 may be configured to receive microphone data, or noise metrics corresponding at least in part to the microphone data, from one or more microphones in the audio environment via interface system 810.

[0070] According to some implementations, device 800 may include an optional loudspeaker system 825 shown in FIG. 8. Optional loudspeaker system 825 may include one or more loudspeakers, which may also be referred to herein as "speakers" or, more generally, "audio reproduction transducers." In some examples (e.g., cloud-based implementations), device 800 may not include loudspeaker system 825. In some implementations, device 800 may include headphones. Headphones may be connected or coupled to device 800 via a headphone jack or via a wireless connection (e.g., BLUETOOTH®).

[0071] Some aspects of the present disclosure include systems or devices configured (e.g., programmed) to perform one or more examples of the disclosed methods, and tangible computer-readable media (e.g., disks) storing code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general-purpose processor, digital signal processor, or microprocessor programmed with software or firmware and / or configured to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system that includes input devices, memory, and a processing subsystem programmed (and / or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to asserted data.

[0072] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) configured (e.g., programmed and otherwise configured) to perform necessary processing on audio signal(s), including performing one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed systems (or elements thereof) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include input devices and memory) that is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations, including one or more examples of the disclosed methods. Alternatively, elements of some embodiments of the inventive systems are implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform one or more examples of the disclosed methods, and the system also includes other elements (e.g., one or more loudspeakers and / or one or more microphones). A general-purpose processor configured to perform one or more examples of the disclosed methods may be coupled to an input device (e.g., a mouse and / or keyboard), memory, and a display device.

[0073] Another aspect of the present disclosure is a computer-readable medium (e.g., a disk or other tangible storage medium) that stores code (e.g., executable code) for performing one or more examples of the disclosed methods or steps thereof.

[0074] While particular embodiments of and applications of the present disclosure have been described herein, it will be apparent to those skilled in the art that many variations on the embodiments and applications described herein are possible without departing from the scope of the present disclosure as described and claimed herein. While particular forms of the present disclosure have been illustrated and described, it should be understood that the disclosure should not be limited to the specific embodiments described and illustrated, or to the particular manner described.

[0075] Various aspects of the present disclosure can be understood from the following enumerated exemplary embodiments (EEE).

[0076] EEE1. Receiving raw sensor data from one or more sensors of at least two wearable audio playback devices; fusing the raw sensor data to generate head movement data; predicting future head movements from the head movement data to generate orientation data; recentering an audio scene using the synchronized orientation data or recentering status control data, wherein the recentering includes at least one of avoiding drift from the audio scene or resetting head orientation; and providing the orientation data or recentering status control data for synchronization between the first one of the wearable audio playback devices and the second one of the wearable audio playback devices; How to process audio.

[0077] EEE2. The wearable audio playback devices are wireless earbuds, and the first device is a master earbud that provides the orientation data or recentering status control data for synchronization, including predicting future head movement data or future orientation data to compensate for transmission jitter, transmission delay, and processing delay, and the second device is a slave earbud; The method of EEE1, further comprising transferring the orientation data or recentering status data from the master earbud to the slave earbud.

[0078] EEE3. The method of EEE2, wherein predicting future head movement includes at least one of predicting transfer latency, compensating for jitter, or adjusting for head movement based on known transfer status information.

[0079] EEE4. The method of EEE3, wherein said known transfer status information includes at least one of a transfer interval, a data drop rate, or a cache buffer length.

[0080] EEE5. The method of any of EEE1 to EEE4, wherein providing said orientation data includes indexing said orientation data based on a calculated data rate and data transfer latency.

[0081] EEE6. Indexing the orientation data or recentering status data based on the calculated data rate and data transfer latency, determining that certain data was missed or delayed during transmission based on a timestamp or data count; and compensating for said missing data during indexing.

[0082] EEE7. The method of EEE6, wherein providing the recentering status data includes the logic for determining the orientation of the virtual audio scene and the time for triggering the recentering.

[0083] EEE8. The method of any one of EEE1 to EEE6, wherein the master earbud and the slave earbud may be swapped to balance power consumption.

[0084] EEE9. A system comprising: 10. A system comprising: a processor; and a computer-readable medium storing instructions that, when executed by the processor, cause the processor to perform the operations recited in any one of EEE1 to EEE8.

[0085] EEE10. A computer readable medium having stored thereon instructions which, when executed by a processor, cause the processor to perform the operations recited in any one of EEE1 to EEE8.

Claims

1. A method for utilizing head tracking data, wherein the method is In each earbud of a pair of communicably coupled earbuds, sensor data is received from one or more sensors located in or on the earbud, and an audio stream is received from a user device. In each of the pair of communicably coupled earbuds, the orientation information of the head is determined based on the sensor data received by the earbud. The leader earbud of the pair of communicably coupled earbuds transmits the determined head orientation information to the follower earbud of the pair of communicably coupled earbuds, and the determined head orientation information is transmitted between the pair of communicably coupled earbuds. At least the follower earbud of the pair of communicably coupled earbuds synchronizes the head orientation information determined in the follower earbud with the head orientation information determined in the leader earbud, at least in part, based on timing information associated with the timestamp when the head orientation information was transmitted. In order to present audio content by each earbud of the pair of communicably coupled earbuds, each earbud of the pair of communicably coupled earbuds renders the audio stream based on the synchronized head orientation data, A method that includes this.

2. The method according to claim 1, wherein determining the head orientation information in each earbud includes fusing data from two or more sensors located in or on the earbud.

3. The method according to claim 1 or 2, wherein determining the head orientation information includes determining predicted head orientation data.

4. The method according to claim 3, wherein the follower earbud is configured to use the predicted head orientation data to compensate for the latency when receiving the head orientation information from the leader earbud.

5. The method according to claim 1, further comprising smoothing the synchronized head orientation data, wherein the synchronized head orientation data used to present audio content includes the smoothed synchronized head orientation data.

6. The method according to claim 1, wherein transmitting the determined head orientation information between the pair of communicably coupled earbuds includes including the timing information in a data stream transmitted from the leader earbud to the follower earbud.

7. The method according to claim 6, further comprising determining whether data is missing in the head orientation information received from the leader earbud, based at least in part on the timing information received from the leader earbud.

8. The method according to claim 6, wherein synchronizing the determined head orientation data in the follower earbud includes synchronizing the determined head orientation data determined in the follower earbud based on one or more sensors of the follower earbud with the head orientation data received from the leader earbud using the timing information.

9. The method according to claim 1, wherein determining the head orientation information in each earbud includes recentering the head orientation data generated based on the sensor data in order to compensate for the drift of the sensor data.

10. The method according to claim 1, wherein each earbud comprises a cache buffer for storing at least the head orientation data.

11. The method according to claim 10, wherein the head orientation data is transmitted from the reader earbud to the follower earbud at a timing interval that depends at least on the data transmission rate and latency associated with transmitting the determined head orientation information between the pair of communicably coupled earbuds.

12. The method according to claim 1, further comprising resampling the sensor data before determining the head orientation information in order to compensate for the difference between the sampling rate used to acquire the sensor data and the sampling rate used to determine the head orientation information.

13. The method according to claim 1, further comprising resampling the determined head orientation information before transmitting the determined head orientation information in order to compensate for the difference between the sampling rate used to determine the head orientation information and the sampling rate used to transmit the determined head orientation information.

14. The method of claim 1, further comprising resampling the synchronized head orientation data before using the synchronized head orientation data to present the audio content, in order to compensate for the difference between the sampling rate used to synchronize the determined head orientation data and the sampling rate used to process the audio content.

15. The method according to claim 1, wherein transmitting the determined head orientation information includes using a wireless communication protocol.

16. Each earbud comprises a pair of communicatively coupled earbuds, each including a processor and one or more sensors. A computer-readable medium storing instructions that, when executed by the processor of the communicatively coupled earbud, cause the processor to perform the operation described in claim 1, A system that includes this.

17. A computer-readable medium storing instructions that, when executed by the processors of a pair of communicably coupled earbuds, cause the processors to perform the operation described in claim 1.

Images