Distributed head tracking

Abstract

A technique for determining head tracking information is provided. In some embodiments, the method involves receiving motion information from a pair of earphones paired with a mobile device. The method may involve determining rotational information related to rotation applied to a sound field according to a head tracking operating mode. In a first mode, the motion information received from the earphones corresponds to the rotational information to be determined. In a second mode, the motion information received from the earphones includes initial rotational information, which is combined with motion information acquired via sensors in the mobile device to determine the rotational information. In a third mode, the motion information received from the earphones includes motion sensor data, which is combined with motion information acquired via sensors in the mobile device to determine the rotational information.

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 517,561, filed on August 3, 2023, and PCT Application No. PCT / CN2023 / 081947, filed on March 16, 2023, all of which are hereby incorporated by reference in their entirety.

[0002] [Technical Field] The present disclosure relates to systems, methods, and media for distributed head tracking.

Background Art

[0003] Listeners of audio content may wish to listen to audio content, such as music, movies, or audio content associated with television programs, in an immersive way, and the audio content is rendered as if it is generated from a specific spatial position relative to the listener's head orientation. However, it can be difficult to determine the spatial rotation of the sound field according to the listener's head orientation.

[0004] Notation and Nomenclature Throughout the present disclosure, including the claims, the terms "speaker", "loudspeaker", and "audio reproduction transducer" are used synonymously to denote any transducer (or set of transducers) that emits sound. A typical set of headphones includes two speakers. A speaker can be implemented to include multiple transducers (e.g., woofers and tweeters) that can be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker feeds can undergo different processing in different circuit branches coupled to different transducers.

[0005] Throughout this disclosure, including the claims, the expression "performing an operation on" a signal or data (e.g., filtering, scaling, transforming, or applying gain to a signal or data) is used in a broad sense to indicate that an operation is performed directly on the signal or data, or on a processed version of the signal or data (e.g., on a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed).

[0006] Throughout this disclosure, including the claims, the term “system” is used in a broad sense to describe a device, system, or subsystem. For example, a subsystem implementing a decoder may be called a decoder system, and a system containing such a subsystem (for example, a system that generates X output signals in response to multiple inputs, wherein the subsystem generates M inputs and XM other inputs are received from an external source) may also be called a decoder system.

[0007] Throughout this disclosure, including the claims, the term “processor” is used in a broad sense to describe a system or device that is programmable or otherwise configurable (for example, using software or firmware) to perform actions on data (for example, audio, or video, or other image data). Examples of processors include field-programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and / or otherwise configured to perform pipeline processing on audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets. [Overview of the project] [Means for solving the problem]

[0008] Techniques for determining head tracking information are provided. In some embodiments, a method for determining head tracking information may involve receiving motion information from a set of earphones paired with a mobile device in a mobile device. The method may further involve the step of determining rotational information associated with rotation applied to a sound field according to a head tracking operating mode, wherein in a first mode of the head tracking operating mode, motion information received from a pair of earphones corresponds to rotational information to be determined; in a second mode of the head tracking operating mode, motion information received from a pair of earphones includes initial rotational information, which is combined with motion information acquired via one or more sensors of the mobile device to determine rotational information; and in a third mode of the head tracking operating mode, motion information received from a pair of earphones includes motion sensor data, which is combined with motion information acquired via one or more sensors of the mobile device to determine rotational information.

[0009] In some examples, the method further includes detecting the motion activity of a wearer of a pair of earphones, and the rotational information determined is at least partially based on the detected motion activity. In some examples, detecting motion activity includes determining whether the wearer is walking, at least partially based on a comparison of the power of the frequency domain representation of the motion information with a threshold. In some examples, detecting motion activity is performed by a pair of earphones. In some examples, detecting motion activity is performed by a mobile device.

[0010] In some examples, a pair of earphones is configured to perform an automatic zeroing procedure to determine an updated reference axis corresponding to the direction the user is facing, and the rotation information determined is at least partially based on the reference axis. In some examples, determining the reference axis includes comparing the head orientation angle variance to a threshold, determining that the head orientation angle variance is below the threshold for a given duration, and determining an updated reference axis in response to the determination that the head orientation angle variance is below the threshold for a given duration. In some examples, the threshold is set based at least partially on the variance.

[0011] In some examples, a pair of earphones is configured to perform a calibration procedure which is configured to correct the tilt of at least one of the earphones in the ear of the wearer of the pair of earphones. In some examples, the calibration procedure includes determining a corrective rotation based on a quaternion that represents the wearer's head orientation.

[0012] In some examples, the mobile device is further configured to receive motion data from at least one other wearable device, including at least one of a smartwatch or a fitness tracker. In some examples, in a third mode of the head-tracking motion mode, the mobile device is configured to combine motion data from at least one other wearable device with motion information obtained through one or more sensors of the mobile device in order to determine rotational information.

[0013] In some examples, the mobile device determines rotational information associated with the mobile device's rotation used to determine the rotation applied to the sound field, and determines rotational information used to determine the rotation applied to the sound field based on the mobile device's yaw angle. In some examples, determining rotational information associated with the mobile device's rotation used to determine the rotation applied to the sound field includes determining that the rotational information associated with the mobile device's rotation has reliability above a reliability threshold and that the mobile device has a static metric within a static threshold.

[0014] In some embodiments, the system includes a pair of earphones and a mobile device paired with the pair of earphones, the mobile device including at least one processor configured to receive motion information from the pair of earphones paired with the mobile device and to determine rotational information associated with rotation applied to the sound field according to a head-tracking operation mode. In a first mode of the head-tracking operation mode, the motion information received from the pair of earphones corresponds to the rotational information to be determined. In a second mode of the head-tracking operation mode, the motion information received from the pair of earphones includes initial rotational information, which is combined with motion information acquired via one or more sensors of the mobile device to determine the rotational information. In a third mode of the head-tracking operation mode, the motion information received from the pair of earphones includes motion sensor data, which is combined with motion information acquired via one or more sensors of the mobile device to determine the rotational information.

[0015] In some examples, at least one of a pair of earphones or mobile devices is configured to detect the motion activity of the wearer of the pair of earphones, and the rotational information determined is at least partially based on the detected motion activity. In some examples, detecting motion activity includes determining whether the wearer is walking, at least partially based on a comparison of the power of the frequency domain representation of the motion information with a threshold.

[0016] In some examples, a pair of earphones is configured to perform an automatic zeroing procedure to determine an updated reference axis corresponding to the direction the user is facing, and the rotation information determined is at least partially based on the reference axis. In some examples, determining the reference axis includes comparing the head orientation angle variance to a threshold, determining that the head orientation angle variance is below the threshold for a given duration, and determining an updated reference axis in response to the determination that the head orientation angle variance is below the threshold for a given duration. In some examples, the threshold is set based at least partially on the variance.

[0017] In some examples, a pair of earphones is configured to perform a calibration procedure which is configured to correct the tilt of at least one of the earphones in the ear of the wearer of the pair of earphones. In some examples, the calibration procedure includes determining a corrective rotation based on a quaternion that represents the wearer's head orientation.

[0018] In some examples, the mobile device is further configured to receive motion data from at least one other wearable device, which may include at least one of the following: a smartwatch or a fitness tracker.

[0019] In some examples, the mobile device determines rotational information associated with the mobile device's rotation used to determine the rotation applied to the sound field, and determines rotational information used to determine the rotation applied to the sound field based on the mobile device's yaw angle. In some examples, determining rotational information associated with the mobile device's rotation used to determine the rotation applied to the sound field includes determining that the rotational information associated with the mobile device's rotation has reliability above a reliability threshold and that the mobile device has a static metric within a static threshold.

[0020] In some embodiments, a pair of earphones includes one or more motion sensors and at least one processor. The at least one processor may be configured to pair with a mobile device and to determine rotational information associated with rotation applied to the sound field according to a head-tracking operating mode. In a first mode of the head-tracking operating mode, the rotational information includes motion data acquired from one or more motion sensors of the pair of earphones. In a second mode of the head-tracking operating mode, the rotational information includes a rotation vector determined by at least one processor based on motion data acquired from one or more motion sensors of the pair of earphones. In a third mode of the head-tracking operating mode, the rotational information includes yaw angle, pitch angle, and roll angle associated with the current head orientation of the wearer of the pair of earphones. The at least one processor may be further configured to provide the rotational information to the paired mobile device.

[0021] In some examples, at least one processor is further configured to detect the motion activity of a wearer of a pair of earphones, and the rotational information determined is at least partially based on the detected motion activity. In some examples, detecting motion activity includes determining whether the wearer is walking, at least partially based on a comparison of the power of the frequency domain representation of the motion information with a threshold.

[0022] In some examples, at least one processor is configured to perform an automatic zeroing procedure to determine an updated reference axis corresponding to the direction the user is facing, and the rotation information determined is at least partially based on the reference axis. In some examples, determining the reference axis includes comparing the head-turning angle variance to a threshold, determining that the head-turning angle variance is below the threshold for a given duration, and determining an updated reference axis in response to the determination that the head-turning angle variance is below the threshold for a given duration. In some examples, the threshold is set based at least partially on the variance.

[0023] In some examples, at least one processor is further configured to perform a calibration procedure configured to correct the tilt of at least one earbud of a pair of earbuds in the ear of a wearer of a pair of earbuds. In some examples, the calibration procedure includes determining a corrective rotation based on a quaternion representing the wearer's head orientation.

[0024] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices in accordance with instructions (e.g., software) stored on one or more non-temporary media. Such non-temporary media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices and read-only memory (ROM) devices.

[0025] At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices may be capable of at least partially performing the methods disclosed herein. In some implementations, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or combinations thereof.

[0026] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Note that the relative dimensions of the following figures may not be drawn to scale.

Brief Description of the Drawings

[0027] [Figure 1A] An exemplary configuration of a distributed head tracking system according to some embodiments is shown. [Figure 1B] An exemplary configuration of a distributed head tracking system according to some embodiments is shown. [Figure 2A] Exemplary outputs of various configurations of a distributed head tracking system according to some embodiments are shown. [Figure 2B] Exemplary outputs of various configurations of a distributed head tracking system according to some embodiments are shown. [Figure 2C] Exemplary outputs of various configurations of a distributed head tracking system according to some embodiments are shown. [Figure 2D] Exemplary outputs of various configurations of a distributed head tracking system according to some embodiments are shown. [Figure 2E]The following shows exemplary outputs of various configurations of a distributed head-tracking system according to several embodiments. [Figure 3] The following illustrates exemplary instances of a distributed head-tracking system implemented on a pair of earphones, according to several embodiments. [Figure 4] The following illustrates exemplary instances of a distributed head-tracking system implemented on a wearable device, according to several embodiments. [Figure 5] The following are exemplary instances of a distributed head-tracking system implemented on a host device, according to several embodiments. [Figure 6] This is a flowchart illustrating an exemplary process for recentering the reference axis for sound field orientation, according to several embodiments. [Figure 7] This is an exemplary graph of sensor data that can be used to recenter a reference axis according to several embodiments. [Figure 8] This is a flowchart illustrating an exemplary process for determining a threshold used to recenter a reference axis, according to several embodiments. [Figure 9] This is a graph of exemplary sensor data that can be used to determine a threshold used to recenter a reference axis, according to several embodiments. [Figure 10] This is a flowchart illustrating an exemplary process for performing activity detection according to several embodiments. [Figure 11] The graphs shown are exemplary filters that may be applied to motion data to perform activity detection according to several embodiments. [Figure 12] This is a flowchart illustrating an exemplary process for calibrating motion data according to several embodiments. [Figure 13] According to several embodiments, exemplary configurations are shown for utilizing the orientation of a host device to determine sound field orientation. [Figure 14A]According to several embodiments, exemplary state machines for determining whether data is reliable and / or static are shown. [Figure 14B] The diagrams shown illustrate how to determine whether the orientation is static or not, according to several embodiments. [Figure 15] This is a graph showing the tracking status of an exemplary host device in several embodiments. [Figure 16] This is a flowchart illustrating an exemplary process for determining rotational information by a mobile device according to a head tracking motion mode, according to several embodiments. [Figure 17] This is a flowchart illustrating an exemplary process for determining rotational information using a pair of earphones, according to a head-tracking operation mode, as in some embodiments. [Figure 18] A block diagram is shown illustrating an example of the components of a device capable of implementing various aspects of this disclosure.

[0028] Similar reference numbers and names in various drawings refer to the same elements. [Modes for carrying out the invention]

[0029] In some cases, audio content may be rendered immersively so that audio content objects are rendered as having a specific spatial location relative to the listener's head. If the listener is wearing headphones or earphones, the audio content may be rendered based on the listener's head orientation, which may be determined based on one or more sensors placed inside or on the headphones or earphones. For example, the sensors may include one or more accelerometers and / or one or more gyroscopes, which may provide sensor data that can be used to determine the listener's head orientation and / or changes in the listener's head orientation. Based on the listener's head orientation, the audio content may be rendered so that the spatial location of the audio objects is perceived as being in a specific location relative to the listener's head orientation. For example, in some cases, it may be desirable to render audio objects as being in front of the listener, regardless of the direction the listener is facing. Another example is when it may be desirable to render audio objects in a fixed location relative to an external reference frame, regardless of the direction the listener's head is facing. In some cases, whether audio objects are rendered using a fixed spatial location relative to an external reference frame or a reference frame centered on the listener's head may depend on the type of audio content, the activity the listener is involved in, and other factors.

[0030] Data from headphones or earphones may be provided to a host device (e.g., a mobile device, tablet computer, etc.) via a communication channel such as a BLUETOOTH® communication channel or other types of communication channels. The host device may render audio content based on the data and have the rendered audio content presented through the headphones / earphones. Note that, as used herein, “host device” generally refers to a device that is communicably coupled to a pair of earphones and renders the audio content presented by the pair of earphones. Examples of host devices include mobile phones, tablet computers, laptop computers, desktop computers, and game consoles. Different headphones / earphones and different host devices may have different capabilities. For example, some headphones / earphones may be equipped with both an accelerometer and a gyroscope, while others may have only one or more types of sensors (e.g., one or more gyroscopes). Another example is that some headphones / earphones may have a more advanced processor capable of more advanced processing to determine the listener’s current activity and / or head orientation, while others may have more limited processing capabilities.

[0031] This specification discloses systems, methods, and techniques for flexible modes of performing listener head tracking and determining sound field orientation for earphones, host devices, and other wearable devices having different capabilities. In particular, this specification discloses various configurations for distributed head tracking systems, where a distributed head tracking block instance may be implemented on a pair of earphones and a paired host device. Each distributed head tracking block instance may be configured to perform various functions depending on the sensors disposed on each device, the capabilities of each device, the operating system requirements or limitations of the devices, etc. For example, a distributed head tracking block instance implemented on a pair of earphones may be configured to provide rotation information to the host device via a BLUETOOTH communication channel or other communication channel, and the specific rotation information provided by the pair of earphones may depend on the capabilities of the pair of earphones. The host device may be configured to combine the rotation information provided by the pair of earphones with rotation information and / or motion data acquired by the host device's sensors to determine aggregated rotation information, which may be used by the host device to determine the sound field orientation or rotation on which audio content can be rendered. In some embodiments, an additional wearable device, such as a smartwatch or fitness tracker, may further include a distributed head-tracking block instance configured to provide additional motion data associated with the listener to the host device. In such embodiments, the host device can use the additional motion data to determine the listener's orientation or activity (e.g., whether the listener is walking / running, substantially static, etc.).

[0032] In some embodiments, a distributed head-tracking system may be implemented such that a host device (e.g., a mobile device, laptop computer, etc.) implements an instance of the distributed head-tracking block, and one or more paired wearable devices (e.g., a pair of earphones, a fitness tracker, a smartwatch, etc.) implement instances of the distributed head-tracking block. Each instance of the distributed head-tracking block may be configured to take motion data as input, which may include acceleration data and / or raw rotation data (e.g., from a gyroscope), and to produce rotation information as output. The rotation information may be in the format of a rotation vector or rotation matrix, a quaternion, or yaw-pitch-roll angles (e.g., Euler angles). The host device may receive rotation information from one or more paired devices and may generate aggregated rotation information based on the rotation information generated from the host device's sensors and the rotation information generated from the sensors of one or more paired devices. The aggregated rotation information may then be used to determine sound field orientation based on which audio content is being rendered. It should be noted that, regardless of the device on which each distributed head-tracking instance is implemented, the distributed head-tracking instance, and any block or module implemented therein, may be implemented using one or more control systems and / or one or more processors of the device. An example of such a control system is shown in Figure 18 and is described below in relation to Figure 18.

[0033] Figure 1A shows an exemplary implementation of a distributed head tracking system, including distributed head tracking block instances on a host device and on paired devices, according to several embodiments. As shown, a first wearable device 101 includes a device sensor 102 and a distributed head tracking instance 104. The device sensor 102 may include one or more accelerometers and / or one or more gyroscopes configured to provide raw rotational data and / or raw acceleration data to the distributed head tracking instance 104. The distributed head tracking instance 104 may be configured to provide rotational information, which is a fusion of the acceleration data and raw rotational data provided by the device sensor 102, to the distributed head tracking instance 112 on the host device 105. Similarly, a second wearable device 103 also includes a device sensor 106 and a distributed head tracking instance 108. The distributed head tracking instance 108 may be configured to provide rotational information to the distributed head tracking instance 112 on the host device 105. In some embodiments, the first wearable device 101 may be a pair of earphones paired with a host device, and the second wearable device 103 may be, for example, a fitness tracker, smartwatch, smart glasses, or augmented reality headset. The host device 105 includes a device sensor 110 which may be configured to provide raw rotational and acceleration data to the distributed head tracking instance 112. The host device 105 may be a mobile phone, tablet computer, or the like. As shown in the figure, the distributed head tracking instance 112 may generate output rotation by aggregating rotational information provided by each of the distributed head tracking instances 104 and 108, along with rotational and acceleration data from the host device's device sensor 110. The output rotation may be used to determine sound field orientation or rotation based on which audio content is rendered by the host device. Note that while two wearable devices are shown in Figure 1, in some embodiments the distributed head tracking system may utilize one, two, three, five, and so on.Wearable device. For example, in some embodiments, a pair of earphones paired with a host device may be used, and a second wearable device may be omitted.

[0034] In some embodiments, a wearable device such as a pair of earphones, a fitness tracker, or a smartwatch may not implement an instance of a distributed head tracking block. In some such implementations, the wearable device may provide raw rotational data (e.g., from a gyroscope) and / or acceleration data (e.g., from one or more accelerometers) to an instance of a distributed head tracking block in a host device. The host device may then determine the output rotational information, from which the sound field orientation may be determined based on the aggregation of raw data from the wearable device and sensor data from the host device.

[0035] Figure 1B shows an exemplary implementation of a distributed head tracking system in which the distributed head tracking instance is implemented only on a host device, according to several embodiments. As shown, the first wearable device 151 includes a device sensor 152 configured to provide raw rotational and acceleration data to the distributed head tracking instance 158 on the host device 155. Similarly, the second wearable device 153 includes a device sensor 154 configured to provide raw rotational and acceleration data to the distributed head tracking instance 158 on the host device 155. The distributed head tracking instance 158 is configured to determine an output rotation value using the raw rotational and acceleration data from the first wearable device 151 and the second wearable device 153, in conjunction with the raw rotational and acceleration data from the device sensor 156 of the host device 155. The output rotation value from the host device 155 can then be used to determine the sound field orientation or rotation to which audio content can be rendered by the host device 155.

[0036] In some embodiments, the final output rotation may include a yaw-pitch-roll (YPR) angle, sometimes referred to herein as the Euler angle. The YPR angle may be used to determine sound field orientation or rotation based on which audio content is rendered by the host device and can be played back by a pair of earphones. The YPR angle may be determined by a distributed head-tracking block instance. In some embodiments, the distributed head-tracking block instance that determines the YPR angle may be implemented on the host device. Alternatively, in some embodiments, the distributed head-tracking block instance that determines the YPR angle may be implemented on a pair of earphones paired with the host device and transmitted to the host device so that audio content can be rendered by the host device according to the YPR angle.

[0037] Figures 2A, 2B, 2C, 2D, and 2E show various exemplary configurations of a distributed head-tracking system according to several embodiments.

[0038] In the exemplary configuration shown in Figure 2A, a distributed head tracking block instance 202 on a pair of earphones receives accelerometer and gyroscope data and generates a rotation vector as an output. The rotation vector may include a fusion of acceleration and raw rotation data generated by the accelerometer and gyroscope, respectively. The rotation vector may be provided to a distributed head tracking block instance 204 on a host device. The distributed head tracking block instance 204 may combine the rotation vector with quaternion information (which may be generated based on one or more accelerometers and / or gyroscopes of the host device) to generate an output YPR angle. Note that the host device may also receive gyroscope data, as shown in Figure 2A.

[0039] In the exemplary configuration shown in Figure 2B, the distributed head tracking block instance 208 of the host device may receive acceleration and raw rotation data from the earphone 206. Note that the earphone 206 may be configured to transmit raw acceleration and raw rotation data without processing the data to generate, for example, a rotation vector or rotation matrix. The distributed head tracking block instance 208 may be configured to combine the acceleration and raw rotation data from the earphone 206 with quaternion information obtained from the host device's sensors to generate an output YPR angle.

[0040] In the exemplary configuration shown in Figure 2C, a distributed head-tracking block instance 210 of a pair of earphones may be directly configured to generate an output YPR angle based on gyroscope data from one or more gyroscopes of the pair of earphones.

[0041] In the exemplary configuration shown in Figure 2D, a distributed head tracking block instance 212 of a pair of earphones may be configured to generate a rotation vector based on acceleration data and raw rotation data from one or more accelerometers and one or more gyroscopes. A distributed head tracking block instance 214 of a wearable device (e.g., a smartwatch, fitness tracker, etc.) may be configured to generate motion state data from one or more accelerometers of the wearable device. A distributed head tracking block instance 216 of a host device may be configured to combine the rotation vector received from a pair of earphones with motion state data received from the wearable device, along with quaternion data generated from the host device's sensors, in order to generate an output YPR angle. Note that, as shown in Figure 2D, the host device may also receive gyroscope data from the pair of earphones.

[0042] In the exemplary configuration shown in Figure 2E, a pair of earphones may implement an auto-zero block 218 configured to convert gyroscope data (e.g., raw rotation data) into a rotation vector. The rotation vector can then be sent to a distributed head tracking block instance 220 of the host device. The distributed head tracking block instance 220 of the host device may be configured to combine the rotation vector received from the pair of earphones with quaternion data generated based on the host device's sensors to generate an output YPR angle.

[0043] In some implementations, a distributed head-tracking block instance may be configured to perform any combination of functions, regardless of the device on which it is implemented, including converting rotation data from one format to another, calibrating rotation data, performing activity detection, recentering a reference frame based on user orientation and / or movement, and / or generating rotation information. In some embodiments, a distributed head-tracking block instance implemented on a pair of earphones may be configured to determine walking direction by, for example, performing signal processing on acceleration data acquired from one or more accelerometers on the pair of earphones. It should be noted that the flexible implementation of a distributed head-tracking block instance may allow each device to generate data based on its capabilities.

[0044] Figure 3 is a schematic diagram of an exemplary implementation of a distributed head tracking block instance 300 implemented on a pair of earphones. As shown in the figure, the transformation block 302 may be configured to take raw rotation data, for example, from one or more gyroscopes on the pair of earphones, as input. The raw rotation data may be transformed by the transformation block 302 into any suitable format, such as a rotation vector, rotation matrix, or quaternion. The transformed rotation data may optionally be provided to a calibration block 304, which may be configured to calibrate the rotation data. An exemplary technique that may be implemented by the calibration block 304 is shown in Figure 12 and described below in relation to Figure 12. Acceleration data, for optionally, from one or more accelerometers on the pair of earphones, may be provided to a walking direction detection block 306. The walking direction detection block 306 may use the rotation data and acceleration data to determine the walking direction, for example, by performing signal processing to determine the walking direction. The activity detection block 308 may use the acceleration data to detect whether the user is stationary or engaged in an activity, based on signal processing of the acceleration data. Exemplary techniques that may be implemented by the activity detection block 308 are shown in Figures 10 and 11 and are described below in relation to Figures 10 and 11. Optionally calibrated converted rotation data may be provided to an auto-zero block 310, which may be configured to recenter the reference axis based on user rotation data. Exemplary techniques that may be implemented by the auto-zero block 310 are shown in Figures 6, 7, 8, and 9 and are described below in relation to Figures 6, 7, 8, and 9. Based on the recentered reference axis, the output selector block 312 may be configured to output rotation information, which may include rotation vectors, YPR angles, etc., depending on the head tracking operating mode and / or the capabilities of a pair of earphones. The output rotation information may be transmitted to a paired host device (e.g., a paired mobile phone).

[0045] Figure 4 is a schematic diagram of an exemplary implementation of a distributed head tracking block instance 400 implemented on a wearable device other than a pair of earphones. For example, the distributed head tracking block instance 400 may be implemented in a fitness tracker, smartwatch, augmented reality headset, etc. As shown in the figure, the distributed head tracking block instance 400 may include a transformation block 402 configured to take raw rotation data (e.g., obtained from a gyroscope) as input and convert the raw rotation data into, for example, a rotation vector, rotation matrix, quaternion, etc. The transformed rotation data may be provided to an activity detection block 404. The activity detection block 404 may be configured to perform activity detection using the transformed rotation data and acceleration data. Exemplary techniques that may be implemented by the activity detection block 308 are shown in Figures 10 and 11 and are described below in relation to Figures 10 and 11. Output rotation information from the distributed head tracking block instance 400 may be provided to a host device.

[0046] Figure 5 is a schematic diagram of an exemplary implementation of a distributed head tracking block instance 500 implemented on a host device according to several embodiments. As shown, the distributed head tracking block instance 500 is configured to receive raw rotation and acceleration data from a pair of earphones 502 and optionally from another wearable device 504. The rotation data from the pair of earphones can be converted by a conversion block 506 to, for example, a rotation vector or matrix, a quaternion, etc. The converted rotation data can then be calibrated by a calibration block 508. The calibrated rotation data can then be used by an auto-zero block 512 to determine a re-centered reference axis. The calibrated rotation data and acceleration data from the earphones can be processed by an activity detection block 514a to determine a motion state based on the earphone data. Rotation and acceleration data from any wearable device may be processed by an activity detection block 514b to determine a motion state based on the data from the wearable device. Data from the host device sensor 510 may be processed by the activity detection block 514c to determine the motion state based on the host device sensor data. Note that exemplary techniques that may be implemented by any of the activity detection blocks 514a, 514b, and / or 514c are shown in Figures 10 and 11, and will be described below with respect to Figures 10 and 11. The motion state determined by the earphone data, optional wearable device data, and host device sensor data, along with a recentered reference axis, can be processed by the core host device tracker block 516 to determine the output rotation, which may be the output YPR angle.

[0047] The reference coordinate system is updated, for example, when the listener moves or changes orientation, thereby allowing the sound field to rotate according to the listener's updated reference coordinate system. The reference axis may be updated based on rotation data. For example, in some implementations, the reference axis may be updated in response to a determination that the variation in rotation data falls below a threshold over a predetermined duration. In response, the reference axis can be updated to a new reference axis, for example, using a crossfade technique or other smoothing technique, allowing the reference axis to be updated without causing an unpleasant listener experience. For example, crossfading can be performed using linear smoothing techniques, exponential smoothing techniques, cosine smoothing techniques, etc. In some embodiments, the new reference axis may be determined at least in part based on user activity. For example, the new reference axis may be set as the listener's walking direction in response to a determination that the listener is walking in a single linear direction. As another example, the new reference axis may be set as a short-time smoothed input orientation in response to a determination that the listener is moving, but the movement is not in a single direction.

[0048] Figure 6 is a flowchart of an exemplary process 600 for updating a reference axis, according to several embodiments. Blocks of process 600 may be implemented on a host device, such as a mobile phone, which is configured to update the sound field orientation based on the reference axis. In some embodiments, blocks of process 600 may be executed in an order other than that shown in Figure 6. In some implementations, two or more blocks of process 600 may be executed substantially in parallel. In some implementations, one or more blocks of process 600 may be omitted.

[0049] Process 600 may begin in 602 by determining the conversion of rotational information to generate yaw angle, pitch angle, and roll angle. Rotational information may be obtained by one or more sensors of a pair of earphones paired with the host device, one or more sensors of another wearable device paired with the host device, and / or one or more sensors of the host device. One or more sensors may include one or more accelerometers, one or more gyroscopes, and so on.

[0050] In 604, process 600 may smooth the yaw angle, pitch angle, and roll angle. The smoothing may be performed over any suitable time window, for example, 10 milliseconds, 50 milliseconds, 0.5 seconds, 1 second, etc.

[0051] In 606, process 600 can determine the variations in the smoothed yaw angle, pitch angle, and roll angle. In some embodiments, the variations may be determined by the following equations.

[0052]

number

[0053] Here, θ[t0] is the current angle, and N+1 is the length of the window used to calculate the average.

[0054] In 608, in response to the determination that the yaw, pitch, and roll variations are greater than a predetermined threshold and stable, process 600 can determine an updated reference axis. In some embodiments, the threshold may be determined based on data variance, as shown in and described below in relation to Figures 8 and 9. In such implementations, the variance may be estimated from the portion of rotation data in which the rotation angle is stable over a given time period. Alternatively, in some embodiments, the threshold may be set as a predetermined constant value.

[0055] Process 600 may determine that yaw, pitch, and roll fluctuations are stable by using a counter to count the duration for which fluctuations are below a threshold. In some embodiments, in response to the determination that the counter has reached a specified duration (generally represented herein as Tstable), Process 600 may determine an updated reference axis. The updated reference axis may be determined based on the yaw angle, pitch angle, and roll angle, based on the listener's current activity, or any combination thereof.

[0056] At 610, process 600 can transition to the updated reference axis. For example, process 600 can crossfade from the current reference axis to the updated reference axis. More specifically, crossfading can be implemented using linear transitions, exponential transitions, cosine transitions, etc.

[0057] Figure 7 shows a graph of the rotation angle 702. As shown in the figure, in response to the rotation angle 702 being smaller than a predetermined fluctuation threshold 704 over a predetermined duration 706 (represented as “stabilization time”), the reference axis angle (represented by the curve 708) transitions to the updated reference axis. This transition occurs over a time represented as “recentering time” in Figure 7.

[0058] As described above in relation to Figure 6, in some embodiments, the threshold for recentering the reference axis may be set based on the variance of the rotation angle. For example, in some embodiments, the threshold may be proportional to the variance. The variance may be determined for a portion of the rotation angle data that has been determined to be stable. Figure 8 is a flowchart of an exemplary process 800 for setting the threshold used to recenter the reference axis, according to some embodiments. In some embodiments, blocks of process 800 may be executed by one or more control systems and / or processors of a host device and / or a pair of earphones (e.g., the control systems shown in Figure 18 and described below in relation thereto). In some embodiments, blocks of process 800 may be executed in an order other than that shown in Figure 8. In some implementations, two or more blocks of process 800 may be executed substantially in parallel. In some implementations, one or more blocks of process 800 may be omitted.

[0059] Process 800 can be initiated in 802 by estimating the variance of the rotation angle response for determining that the rotation angle is stable. For example, to determine that the rotation angle is stable, process 800 may determine that the rotation angle has been within a predetermined range for a predetermined period of time. The variance may be determined based on a time series of rotation angle data. In some embodiments, the variance may be referred to herein as the variation in rotation angle.

[0060] In 804, process 800 can determine a threshold for recentering the reference axis based on the variance. For example, in some embodiments, the threshold may be determined by the following formula:

[0061]

number

[0062] Here, α is a constant, V is the variance of the rotation angle, and Tc is the input constant threshold. In some examples, α may have a value greater than 1 / V when the user determines it to be stable. Exemplary values ​​of α may include 9, 10, 11, etc. Exemplary values ​​of Tc may include 5 degrees, 10 degrees, 15 degrees, etc.

[0063] Figure 9 is a graph showing an exemplary rotation angle represented by curve 902, and the time periods during which the rotation angle is stable and unstable. As shown in the figure, during time period 904, the rotation angle is stable for a predetermined time period. In response to the stability of the rotation angle over the predetermined time period, the threshold is updated based on the variance of the rotation angle during time period 904. During time period 906, the rotation angle is unstable. During time period 908, the rotation angle is stable for a predetermined time period. In response to the stability of the rotation angle over the predetermined time period, the threshold is updated based on the variance of the rotation angle during time period 908. Note that since the variance of the rotation angle is greater in time period 908 than in time period 904, the threshold is higher after updating from time period 908 onwards than after updating from time period 904 onwards.

[0064] In some implementations, one or more control systems or processors of a pair of earphones and / or mobile devices may be configured to perform activity detection on motion data acquired from motion sensors (e.g., one or more accelerometers and / or gyroscopes) of the pair of earphones. For example, activity detection may indicate whether the listener is walking / running or stationary. In some embodiments, the sound field orientation may be set to the direction in which the listener is moving if the listener is walking or running, but if the listener is determined to be stationary (e.g., not moving forward in a straight line), the sound field orientation may be set to the direction the listener is looking (e.g., head orientation). In some embodiments, activity detection may be performed by filtering acceleration data, for example, acceleration data in the z direction corresponding to the acceleration of the listener's head along an axis pointing outward from the top of the listener's head. For example, the acceleration data may be low-pass and / or band-pass filtered. Continuing this example, the power of the filtered acceleration data may be compared to a threshold. The listener may be determined to be walking or running in response to a power estimate exceeding a predetermined threshold. Conversely, listeners may be considered static in response to power estimates falling below a predetermined threshold.

[0065] Figure 10 is a flowchart of an exemplary process 1000 for performing activity detection according to several embodiments. Blocks of process 1000 may be executed by a control system and / or processor of a pair of earphones and / or wearable devices. An example of such a control system is shown in Figure 18 and is described below in relation to Figure 18. In some embodiments, blocks of process 1000 may be executed in an order other than that shown in Figure 10. In some implementations, two or more blocks of process 1000 may be executed substantially in parallel. In some implementations, one or more blocks of process 1000 may be omitted.

[0066] Process 1000 can be initiated in 1002 by converting rotation information. For example, the rotation information may correspond to raw gyroscope data, which can be converted into rotation information that is generally represented as a matrix M in this specification.

[0067] In 1004, process 1000 can rotate acceleration information based on the transformed rotation information. For example, the rotation data may correspond to three-axis rotation data, which are generally represented as ax, ay, and az in this specification. Continuing this example, acceleration data can be rotated by multiplying the acceleration data by a rotation matrix M. As an example, the rotated acceleration information may be determined by the following formula:

[0068]

number

[0069] In 1006, process 1000 can filter the rotated acceleration information. For example, in some embodiments, process 1000 filters α as described above. z The z-axis acceleration information rotated as represented by the bar can be filtered. In some embodiments, filtering may involve applying a low-pass filter (e.g., having a 3 Hz cutoff, 5 Hz cutoff, 7 Hz cutoff, etc.) and then band-pass filtering the signal. The band-pass filter may have a cutoff frequency in the range of approximately 1–5 Hz, 2–3 Hz, etc. Such filtering may function to isolate portions of the z-axis acceleration signal related to the up-and-down movement of the listener's head during walking or running.

[0070] Figure 11 shows an exemplary low-pass filter 1102 and an exemplary band-pass filter 1104 according to several embodiments.

[0071] Referring back to Figure 10, in 1008, process 1000 can perform activity detection based on filtered acceleration information. For example, process 1000 can generate a power estimate, which is generally expressed as Pm in this specification. As an example, the power estimate may be determined by the following formula:

[0072]

number

[0073] In the above equation, α z The hat represents filtered and rotated z-axis acceleration information, where t0 is the initial time and t1 is the current time. The duration between t1 and t0 can be the duration over approximately 10 samples (e.g., 8 samples, 10 samples, 12 samples, etc.). In one example, assuming a sampling rate of 50 Hz and a duration over 10 samples, the duration is 200 milliseconds.

[0074] In some embodiments, process 1000 can determine whether the current activity is walking, running, or static by comparing a power estimate with a predetermined threshold, which is generally expressed herein as Tm. For example, process 1000 can determine that the listener is walking or running in response to the power estimate meeting or exceeding Tm. Conversely, process 1000 can determine that the listener is static in response to the power estimate falling below Tm. In some embodiments, Tm may have a value between 0.02 and 0.3, e.g., 0.05, 0.1, 0.15, 0.25, etc.

[0075] In some cases, one of the earbuds in a pair may be tilted in the listener's ear, which can then make it difficult to determine the listener's head orientation due to the tilt of the sensor within the earbud. In some embodiments, the tilt can be compensated for when determining the listener's head orientation. For example, in some embodiments, a calibration procedure may be performed to determine a reverse rotation that is performed to rotate from the sensor-measured orientation to a known initial head orientation. The reverse rotation determined during the calibration procedure can then be stored for future use, for example, to compensate for the tilt of the earbud with respect to the listener's ear when it is worn by the listener.

[0076] Figure 12 is a flowchart of an exemplary process 1200 for performing and utilizing a calibration procedure to correct earphone tilt, according to several embodiments. In some implementations, blocks of process 1200 may be executed by one or more control systems or processors of a pair of earphones and / or a mobile device paired with a pair of earphones. Examples of such control systems are shown in Figure 18 and described below in relation to Figure 18. In some embodiments, blocks of process 1200 may be executed in an order other than that shown in Figure 12. In some implementations, two or more blocks of process 1200 may be executed substantially in parallel. In some implementations, one or more blocks of process 1200 may be omitted.

[0077] Process 1200 can be initiated in 1202 by determining manual calibration information by determining an inverse sensor rotation that rotates the initial sensor orientation to the initial head orientation. The initial head orientation may be a known head orientation, for example, with the listener's head pointed straight forward and / or at a neutral angle. In some embodiments, the initial head orientation may be represented by an initial head quaternion, which is generally represented herein as qh0. In some cases, qh0 may be represented by [1, 0, 0, 0]. The initial sensor orientation may be represented by a sensor measurement quaternion qs0. The inverse sensor rotation may be represented by q's0, and may be determined such that qs0 is rotated to qh0 by multiplying qs0 and q's0, for example, such that the rotation by q's0 rotates the initial sensor orientation to the initial head orientation. Note that the inverse sensor rotation may be stored for future use (e.g., in memory associated with the earphone and / or mobile device).

[0078] In step 1204, process 1200 can acquire the updated sensor orientation. The updated sensor rotation can be acquired at any appropriate time after manual calibration has been performed, e.g., several hours, several days, several weeks, several months, several years, etc. The updated sensor orientation can be acquired from one or more sensors of a pair of earphones (e.g., one or more accelerometers, one or more gyroscopes, etc.). The updated sensor orientation is generally represented herein as qs1.

[0079] In step 1206, process 1200 can use the inverse sensor rotation to determine the updated head orientation corresponding to the updated sensor orientation. For example, in some embodiments, process 1200 can determine the updated head orientation by multiplying the inverse sensor rotation by the updated sensor orientation. As an example, assuming the inverse sensor rotation of q is 0 and the updated sensor rotation is represented by qs1, the updated head orientation represented by qh1 may be determined by the following formula:

[0080]

number

[0081] In some cases, it may be advantageous to determine the sound field orientation based on the orientation of the host device presenting the content. For example, in a case where a listener is viewing video content presented from a host device (e.g., watching a video on a phone or tablet computer), the sound field orientation may be determined by referring to the host device's display. In another example, if the host device is in a moving vehicle, the direction and / or orientation of the host device's movement may represent the orientation of the environment, and therefore the sound field orientation may be determined to be consistent with the orientation of the host device. Generally, determining the sound field orientation based on the orientation of the host device is referred to herein as the “host device tracking state,” while cases where the sound field orientation is determined based on the orientation of the listener's head, such as by sensors in the earphones, are referred to as the “host device non-tracking state.” In the “host device tracking state,” the host device may be configured to determine the overall orientation by combining orientation information obtained from the earphones and rotational information determined from the host device. For example, yaw angles obtained from the earphones and the host device may be combined to determine a single sound field orientation yaw angle.

[0082] Figure 13 is a diagram of an exemplary system for determining sound field orientation in either a host device tracking state or a host device non-tracking state, according to several embodiments. As shown, in the host device tracking state, the raw yaw angle 1302 from the earphones and the yaw angle 1304 from the host device may be combined by an auto-zero block 1306. The auto-zero block 1306 may be configured to determine the yaw angle between the raw yaw angle from the earphones and the yaw angle from the host device. The output of the auto-zero block 1306 may be provided to a distributed head tracking block 1308 on the host device, which may be configured to determine sound field orientation based on the yaw angle determined by the auto-zero block 1306.

[0083] In contrast, when the host device is not tracking, the processed yaw angle 1310 obtained from the earphone sensor can be provided to a distributed head tracking block 1308 on the host device, which can determine the sound field orientation regardless of the orientation of the host device.

[0084] In some embodiments, the host device tracking state can be entered in response to the necessary condition that the host device orientation data is reliable, and in response to the sufficient condition that the listener's activity is static (e.g., not walking or running). In other words, the host device tracking state is entered only if the host device orientation data is reliable, but may not be entered even if the orientation data is reliable. The determination that the listener's activity is static is sufficient to enter the host device tracking state. The host device orientation data may be considered reliable in response to the determination that the host device is neither tilted nor stationary (e.g., due to being left on a table or desk). The determination that the listener's activity is static (e.g., not walking or running) may be determined based on a consensus of data from one or more devices (e.g., the host device, a pair of earphones, another paired wearable device, etc.) that the listener is static. Note that the transition to the host device tracking state may occur after the host device orientation data has been determined to be reliable beyond a threshold confidence time, and / or after the listener's activity has been determined to be static beyond a threshold static movement time. Using threshold state time to manage transitions to host device tracking states can prevent instantaneous state changes that could lead to unpleasant sound field orientation for the listener.

[0085] Figure 14A shows a state machine for transitioning between reliable and unreliable states, and between static and non-static states, according to several embodiments. Note that state determination may be performed by one or more control systems and / or processors of the host device. An example of such a control system is shown in Figure 18 and is described below in relation to Figure 18.

[0086] A transition from a reliable state 1402 to an unreliable state 1404 may occur in response to a determination that the device is either tilted or stationary. Conversely, a transition from an unreliable state 1404 to a reliable state 1402 may occur in response to a determination that the device is neither tilted nor stationary. A device may be determined to be tilted in response to a determination that the tilt velocity along the Z-axis, generally represented herein as ωz, is greater than a threshold tilt velocity, generally represented herein as ωz_threshold. In some examples, ωz_threshold may have a value of approximately 60 degrees, such as 55 degrees, 60 degrees, or 65 degrees. The tilt velocity may be determined based on the change in angle between the current Z-axis and the previous Z-axis. Using quaternions, the angle relative to the previous Z-axis may be determined by the following formula:

[0087]

number

[0088] The slope velocity can be determined by the following formula.

[0089]

number

[0090] In the above equation, θ z0 This represents the current angle with respect to the current Z-axis, and θ z1 This represents the previous angle relative to the previous Z-axis.

[0091] In some implementations, staticity can be determined based on the device's acceleration across all three axes. For example, the staticity metric, represented herein as Ps, can be determined by the following formula:

[0092]

number

[0093] A host device may be determined to remain stationary in response to the fact that the stationary metric Ps is less than a predetermined stationary threshold.

[0094] The transition from static state 1406 to non-static state 1408 may occur in response to a determination of whether the listener is walking or running. Conversely, the transition from non-static state 1408 to static state 1406 may occur in response to a determination that the listener is not walking or running, or a determination that the listener is static. Note that detection of the listener walking or running may be performed using an activity detection block that runs on the host device, a pair of earphones, and / or another paired wearable device. Activity detection may be performed by filtering acceleration data along the Z axis, as shown in relation to Figure 10 and described above.

[0095] In some implementations, a listener may be determined to be static based on outputs from activity detection blocks of one or more devices, including a host device, a pair of earphones, and / or another paired wearable device (e.g., a paired fitness tracker, a paired smartwatch, etc.). Figure 14B shows an exemplary system for determining whether a listener is static. As shown, the output of the host device's activity detection block 1452 may produce an output indicating whether the listener is static or not. Optionally, activity detection blocks 1454 and / or 1456 of other wearable devices (e.g., a pair of earphones, a smartwatch, etc.) may produce outputs indicating whether the listener is static or not. For example, each activity block of each device may produce an output indicating a determination of whether the listener is static or not based on the sensors of each device (e.g., one or more accelerometers and / or gyroscopes). Each static determination may be provided to an AND block 1458, which may generate an aggregated determination of whether the listener is static or not. For example, if each device providing a determination of whether the listener is static agrees, the AND block 1458 may generate an aggregate determination that the listener is static. Conversely, if the determinations from multiple devices do not coincide, the AND block 1458 may generate an aggregate determination that the listener is not static.

[0096] Figure 15 shows an exemplary timing diagram including the transition from a host device tracking state to a host device non-tracking state according to several embodiments. During time period 1502, the host device is in a non-tracking state. During time period 1502, the host device is determined to be reliable (e.g., trustworthy) and static. After the threshold trust time and threshold static time have elapsed at the end of time period 1502, the host device transitions to a tracking state during time period 1504. During this time, the listener is looking at their phone as shown. At the end of time period 1504, the host device orientation data enters an unreliable or unreliable state at time 1505 due to the host device being stationary longer than the threshold quiescence time. In response to the host device orientation data being unreliable longer than the threshold trust time, the host device enters a non-tracking state during time period 1506. During part of time period 1506, the listener begins looking at their phone. During time period 1506, the host device orientation data is determined to be reliable, and the host device is determined to be stationary longer than the threshold trust time and threshold static time, respectively. Therefore, the host device enters the tracking state during time period 1508. At the end of time period 1508, the listener begins walking, which triggers the non-static state at time 1509. In response to the non-static threshold time being met, the host device enters the non-tracking state during time period 1510. Note that the non-static threshold time can be different from the threshold confidence time, as shown in Figure 15. For example, the threshold confidence time may be longer than the threshold non-static time.

[0097] Figure 16 is a flowchart of an exemplary process 1600 for determining rotational information that can be used to generate or modify sound field orientation. Blocks of process 1600 may be executed by one or more control systems and / or processors of a mobile device (sometimes referred to herein as the “host device”). An example of such a control system is shown in Figure 18 and described below in relation to Figure 18. In some embodiments, blocks of process 1600 may be executed in an order other than that shown in Figure 16. In some implementations, two or more blocks of process 1600 may be executed in an order other than that shown in Figure 16. In some implementations, one or more blocks of process 1600 may be omitted. In some implementations, two or more blocks of process 1600 may be executed substantially in parallel.

[0098] Process 1600 can be initiated in 1602 by the mobile device receiving motion information from a pair of earphones paired with the mobile device. As described above in relation to Figures 1A, 1B, and 2A to 2E, the motion information may be raw rotation data and / or processed rotation data (e.g., rotation vectors or rotation matrices). Optionally, the mobile device may further receive motion information from one or more other paired wearable devices, such as a paired fitness tracker or a paired smartwatch.

[0099] In step 1604, process 1600 can determine rotational information associated with the rotation applied to the sound field according to the head tracking operation mode. Note that the head tracking operation mode may correspond to the capabilities of a pair of earphones and / or the host device, such as whether another wearable device is paired with the host device. For example, the head tracking operation mode may correspond to the configuration of a distributed head tracking system as described above in relation to Figures 1A, 1B, and 2A to 2E. For example, in the first mode, motion information received from a pair of earphones corresponds to the rotational information to be determined. An example of the first mode is shown in Figure 2C and described above in relation to Figure 2C. As another example, in the second mode, motion information received from a pair of earphones includes initial rotational information, which is combined with motion information acquired via one or more sensors of a mobile device to determine the rotational information available for rotating the sound field. Examples of the second mode are shown in Figures 2A, 2D, and 2E and described above in relation to them. As mentioned above, Figure 2D shows a configuration of a second mode in which motion information is additionally received from an additional paired wearable device. In yet another example, in a third mode, motion information received from a pair of earphones includes raw motion sensor data (e.g., from one or more accelerometers and / or gyroscopes), and the motion sensor data is combined with motion information obtained from one or more sensors of a mobile device to determine rotational information that can be used to rotate the sound field. An exemplary configuration of the third mode is shown in Figure 2B and described above in relation to Figure 2B.

[0100] After determining rotation information, the mobile device may be configured to rotate the sound field according to the determined rotation information. The mobile device may then render audio content according to the rotated sound field. The rendered audio content may then be presented through paired earphones.

[0101] Figure 17 is a flowchart of an exemplary process 1700 for determining rotational information that can be used to generate or modify sound field orientation. Blocks of process 1700 may be executed by one or more control systems and / or processors of a pair of earphones. Examples of such control systems are shown in Figure 18 and described below in relation to Figure 18. In some embodiments, blocks of process 1700 may be executed in an order other than that shown in Figure 17. In some implementations, two or more blocks of process 1700 may be executed in an order other than that shown in Figure 17. In some implementations, one or more blocks of process 1700 may be omitted. In some implementations, two or more blocks of process 1700 may be executed substantially in parallel.

[0102] Process 1700 can be initiated in 1702 by pairing a pair of earphones with a mobile device. Pairing may involve playing audio content presented by the mobile device through the pair of earphones, or transmitting motion data and / or rotation data to the mobile device via a communication channel through the pair of earphones. The communication channel between the pair of earphones and the mobile device may be a Bluetooth communication channel or another wireless communication channel.

[0103] In step 1704, process 1700 can determine rotational information associated with rotation applied to the sound field according to the head tracking operation mode. Note that the head tracking operation mode may correspond to the capabilities of a pair of earphones and / or the host device, such as whether another wearable device is paired with the host device. For example, the head tracking operation mode may correspond to the configuration of a distributed head tracking system as described above in relation to Figures 1A, 1B, and 2A to 2E. For example, in the first mode, the rotational information may include motion data acquired from one or more motion sensors of a pair of earphones. An exemplary configuration according to the first mode is shown in Figure 2B and described above in relation to Figure 2B. As another example, in the second mode, the rotational information may include a rotation vector determined by at least one processor of a pair of earphones based on motion data acquired from one or more motion sensors of a pair of earphones. An exemplary configuration according to the second mode is shown in Figures 2A, 2D, and 2E and described above in relation to them. As yet another example, in the third mode, the rotation information may include the YPR angle associated with the current head orientation of the wearer of a pair of earphones. An exemplary configuration according to the third mode is shown in Figure 2C and described above in relation to Figure 2C.

[0104] In step 1706, process 1700 can provide rotation information to the paired mobile device. The rotation information may be provided via a wireless communication channel established during the pairing of a pair of earphones with the mobile device.

[0105] A mobile device may be configured to use rotational information, optionally in conjunction with motion data and / or rotational information generated by the mobile device using one or more sensors on the mobile device, in order to determine sound field rotation. The mobile device may then render audio content based on the determined sound field rotation. The rendered audio content may be played back by a pair of earphones.

[0106] Figure 18 is a block diagram showing examples of components of an apparatus capable of implementing various embodiments of the present disclosure. As with other figures provided herein, the types and numbers of elements shown in Figure 18 are provided merely as examples. Other implementations may include more, fewer, and / or different types and numbers of elements. According to some examples, apparatus 1800 may be configured to perform at least some of the methods disclosed herein. In some implementations, apparatus 1800 may be, or include, a television, one or more components of an audio system, a mobile device (such as a cellular phone), a laptop computer, a tablet device, a smart speaker, or another type of device.

[0107] According to several alternative implementations, device 1800 may be a server or may include a server. In some such examples, device 1800 may be an encoder or may include an encoder. Thus, in some cases, device 1800 may be a device configured for use in an audio environment such as a home audio environment, while in other cases, device 1800 may be a device configured for use in a "cloud," for example, a server.

[0108] In this example, the device 1800 includes an interface system 1805 and a control system 1810. In some implementations, the interface system 1805 may be configured to communicate with one or more other devices in an audio environment. In some examples, the audio environment may be a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, or a park environment. In some implementations, the interface system 1805 may be configured to exchange control information and related data with audio devices in the audio environment. In some examples, the control information and related data may relate to one or more software applications running on the device 1800.

[0109] In some implementations, the interface system 1805 may be configured to receive or provide content streams. Content streams may include audio data. Audio data may include, but is not limited to, audio signals. In some cases, audio data may include channel data and / or spatial data such as spatial metadata. In some examples, content streams may include video data and corresponding audio data.

[0110] The interface system 1805 may include one or more network interfaces and / or one or more external device interfaces (such as one or more Universal Serial Bus (USB) interfaces). According to some implementations, the interface system 1805 may include one or more wireless interfaces. The interface system 1805 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 1805 may include one or more interfaces between the control system 1810 and a memory system, such as the optional memory system 1815 shown in Figure 18. However, the control system 1810 may include a memory system in some examples. In some implementations, the interface system 1805 may be configured to receive input from one or more microphones in the environment.

[0111] The control system 1810 may include, for example, a general-purpose single or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0112] In some implementations, the control system 1810 may reside in two or more devices. For example, in some implementations, part of the control system 1810 may reside in a device within one of the environments described herein, while another part of the control system 1810 may reside in a device outside the environment, such as a server or a mobile device (e.g., a smartphone or tablet computer). In other examples, part of the control system 1810 may reside in a device within one environment, while another part of the control system 1810 may reside in one or more other devices within the environment. For example, part of the control system 1810 may reside in a device implementing cloud-based services, such as a server, while another part of the control system 1810 may reside in another device implementing cloud-based services, such as another server or a memory device. The interface system 1805 may also reside in two or more devices in some examples. In some implementations, part of the control system may reside in or on an earphone.

[0113] In some implementations, the control system 1810 may be configured to perform at least partially the methods disclosed herein. According to some examples, the control system 1810 may be configured to perform methods for determining rotational information, determining yaw angle, pitch angle, and roll angle, performing an automatic zeroing procedure, performing a calibration procedure, and the like.

[0114] Some or all of the methods described herein may be executed by one or more devices in accordance with instructions (e.g., software) stored on one or more non-temporary media. Such non-temporary media may include, but are not limited to, random-access memory (RAM) devices, read-only memory (ROM) devices, and other memory devices described herein. One or more non-temporary media may reside, for example, in any memory system 1815 and / or control system 1810 shown in Figure 18. Thus, various inventive aspects of the subject matter described herein may be implemented in one or more non-temporary media on which software is stored. The software may, for example, extract objects from a multi-channel audio signal, generate a spatial enhancement mask, apply the spatial enhancement mask, and generate an output binaural audio signal. The software may be executable by one or more components of a control system, such as the control system 1810 in Figure 18.

[0115] In some examples, the device 1800 may include an optional microphone system 1820, as shown in Figure 18. The optional microphone system 1820 may include one or more microphones. In some implementations, one or more of the microphones may be part of or associated with another device, such as a speaker in a speaker system or a smart audio device. In some examples, the device 1800 may not include the microphone system 1820. However, in some such implementations, the device 1800 may nevertheless be configured to receive microphone data for one or more microphones in the audio environment via the interface system 1810. In some such implementations, a cloud-based implementation of the device 1800 may be configured to receive microphone data, or a noise reference at least partially corresponding to the microphone data, from one or more microphones in the audio environment via the interface system 1810.

[0116] In some implementations, the device 1800 may include an optional loudspeaker system 1825, as shown in Figure 18. The optional loudspeaker system 1825 may include one or more loudspeakers, which may also be referred to herein as “speakers” or more commonly as “audio playback transducers.” In some examples (e.g., cloud-based implementations), the device 1800 may not include the loudspeaker system 1825. In some implementations, the device 1800 may include headphones. The headphones may be connected to or coupled to the device 1800 via a headphone jack or via a wireless connection (e.g., BLUETOOTH®).

[0117] Some aspects of this disclosure include a system or device (e.g., programmed) configured to perform one or more examples of the disclosed method, and a tangible computer-readable medium (e.g., a disk) for storing code for performing one or more examples of the disclosed method or its steps. For example, some disclosed systems are or include a programmable general-purpose processor, digital signal processor, or microprocessor programmed in software or firmware and / or configured to perform any of a variety of operations on data, including embodiments of the disclosed method or its steps. Such a general-purpose processor may be or include a computer system including an input device, memory, and a processing subsystem programmed (and / or otherwise configured) to perform one or more examples of the disclosed method (or its steps) in response to asserted data.

[0118] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) configured (e.g., programmed and otherwise configured) to perform necessary processing on an audio signal(s), including the execution of one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed system (or its elements) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include input devices and memory) programmed in software or firmware and / or otherwise configured to perform any of a variety of operations, including one or more examples of the disclosed methods. Or, elements of some embodiments of the system of the present invention may be implemented as a general-purpose processor or DSP (e.g., programmed) configured to perform one or more examples of the disclosed methods, and the system also includes other elements (e.g., one or more loudspeakers and / or one or more microphones). A general-purpose processor configured to perform one or more examples of the disclosed methods may be coupled to input devices (e.g., a mouse and / or keyboard), memory, and display devices.

[0119] Another aspect of the present disclosure is a computer-readable medium (e.g., a disk or other tangible storage medium) that stores code (e.g., an executable coder for execution) for performing one or more examples of the disclosed method or steps thereof.

[0120] While specific embodiments and applications of the Disclosure are described herein, it will be apparent to those skilled in the art that many modifications are possible to the embodiments and applications described herein without departing from the scope of the Disclosure described herein and claimed herein. Although specific forms of the Disclosure have been shown and described, it should be understood that the Disclosure should not be limited to the specific embodiments or methods described herein.

Claims

1. A method for determining head tracking information, In a mobile device, motion information is received from a pair of earphones paired with the mobile device, This involves determining rotational information associated with rotation applied to the sound field according to the head tracking operation mode, In the first mode of the head tracking operation mode, the motion information received from the pair of earphones corresponds to the rotation information that is determined. In the second mode of the head tracking operation mode, the motion information received from the pair of earphones includes initial rotation information, which is combined with motion information acquired via one or more sensors of the mobile device to determine the rotation information. A method comprising determining, in a third mode of the head tracking operation mode, the motion information received from the pair of earphones includes motion sensor data, and the motion sensor data is combined with motion information acquired via one or more sensors of the mobile device to determine the rotation information.

2. The method according to claim 1, further comprising detecting the movement activity of the wearer of the pair of earphones, wherein the rotation information determined is at least partially based on the detected movement activity.

3. The method according to claim 2, wherein detecting the motion activity includes determining whether the wearer is walking, at least in part, based on a comparison between the power and threshold of the frequency domain representation of the motion information.

4. The method according to claim 2, wherein the detection of the motion activity is performed by the pair of earphones.

5. The method according to claim 2, wherein the detection of the motion activity is performed by the mobile device.

6. The method according to claim 1, wherein the pair of earphones is configured to perform an automatic zeroing procedure to determine an updated reference axis corresponding to the direction the user is facing, and the rotation information determined is at least partially based on the reference axis.

7. Determining the aforementioned reference axis means Comparing the variance of head orientation angle to a threshold, It is determined that the distribution of the head orientation angle falls below the threshold for a predetermined duration, The method of claim 6, comprising determining the updated reference axis in response to the determination that the dispersion of the head orientation angle is below the threshold over a predetermined duration.

8. The method according to claim 7, wherein the threshold is set based at least in part on the variance.

9. The method according to claim 1, wherein the pair of earphones is configured to perform a calibration procedure which is configured to correct the tilt of at least one of the earphones in the ear of the wearer of the pair of earphones.

10. The method according to claim 9, wherein the calibration procedure includes determining a corrected rotation based on a quaternion representing the orientation of the wearer's head.

11. The method according to claim 1, wherein the mobile device is further configured to receive motion data from at least one other wearable device, including at least one of a smartwatch or a fitness tracker.

12. The method according to claim 11, wherein in the third mode of the head tracking operation mode, the mobile device is configured to combine the motion data from at least one other wearable device with the motion information obtained through one or more sensors of the mobile device in order to determine the rotation information.

13. The aforementioned mobile device Determine rotation information associated with the rotation of the mobile device used to determine the rotation applied to the sound field, The method according to claim 1, wherein rotation information used to determine the rotation applied to the sound field is determined based on the yaw angle of the mobile device.

14. The method according to claim 13, wherein determining the rotation information associated with the rotation of the mobile device used to determine the rotation applied to the sound field includes determining that the rotation information associated with the rotation of the mobile device has a reliability above a reliability threshold and that the mobile device has a static metric within a static threshold.

15. It is a system, One or more processors, A system comprising: a non-temporary computer-readable medium that stores instructions causing one or more processors to perform the operations described in any one of claims 1 to 14 when the one or more processors are executed;

16. A non-temporary computer-readable medium that stores instructions causing one or more processors to perform the operations described in any one of claims 1 to 14 when executed by one or more processors.

17. It is a system, One pair of earphones, The set of earphones includes a mobile device paired with the aforementioned pair of earphones, and the mobile device is The aforementioned mobile device receives motion information from a pair of earphones paired with the mobile device, This involves determining rotational information associated with rotation applied to the sound field according to the head tracking operation mode, In the first mode of the head tracking operation mode, the motion information received from the pair of earphones corresponds to the rotation information that is determined. In the second mode of the head tracking operation mode, the motion information received from the pair of earphones includes initial rotation information, which is combined with motion information acquired via one or more sensors of the mobile device to determine the rotation information. In a third mode of the head tracking operation mode, the motion information received from the pair of earphones includes motion sensor data, and the motion sensor data is combined with motion information acquired via one or more sensors of the mobile device to determine the rotation information.

18. The system according to claim 17, wherein at least one of the pair of earphones or the mobile device is configured to detect the movement activity of the wearer of the pair of earphones, and the rotation information determined is at least partially based on the detected movement activity.

19. The system according to claim 18, wherein detecting the motion activity includes determining whether the wearer is walking, at least in part, based on a comparison between the power and threshold of the frequency domain representation of the motion information.

20. The system according to claim 17, wherein the pair of earphones is configured to perform an automatic zeroing procedure to determine an updated reference axis corresponding to the direction the user is facing, and the rotation information determined is at least partially based on the reference axis.

21. Determining the aforementioned reference axis means Comparing the variance of head orientation angle to a threshold, It is determined that the distribution of the head orientation angle falls below the threshold for a predetermined duration, The system according to claim 20, comprising determining the updated reference axis in response to the determination that the dispersion of the head orientation angle is below the threshold over a predetermined duration.

22. The system according to claim 21, wherein the threshold is set based at least in part on the variance.

23. The system according to claim 17, wherein the pair of earphones is configured to perform a calibration procedure which is configured to correct the tilt of at least one of the earphones in the ear of the wearer of the pair of earphones.

24. The system according to claim 23, wherein the calibration procedure includes determining a corrected rotation based on a quaternion representing the orientation of the wearer's head.

25. The system according to claim 17, wherein the mobile device is further configured to receive motion data from at least one other wearable device, which includes at least one of a smartwatch or a fitness tracker.

26. The aforementioned mobile device Determine rotation information associated with the rotation of the mobile device used to determine the rotation applied to the sound field, The system according to claim 17, wherein rotation information used to determine the rotation applied to the sound field is determined based on the yaw angle of the mobile device.

27. The system according to claim 26, wherein determining the rotation information associated with the rotation of the mobile device used to determine the rotation applied to the sound field includes determining that the rotation information associated with the rotation of the mobile device has a reliability above a reliability threshold and that the mobile device has a static metric within a static threshold.

28. It is a pair of earphones, One or more motion sensors, At least one processor, Pair with your mobile device, The rotational information associated with the rotation applied to the sound field is determined according to the head tracking operation mode. In the first mode of the head tracking operation mode, the rotation information includes motion data acquired from one or more motion sensors of the pair of earphones. In the second mode of the head tracking operation mode, the rotation information includes a rotation vector determined by the at least one processor based on the motion data acquired from the one or more motion sensors of the pair of earphones. In the third mode of the head tracking operation mode, the rotation information includes yaw, pitch, and roll angles associated with the current head orientation of the wearer of the pair of earphones. A pair of earphones, including at least one processor configured to provide the rotation information to the paired mobile device.

29. The pair of earphones according to claim 28, wherein the at least one processor is further configured to detect the motion activity of the wearer of the pair of earphones, and the rotation information determined is at least partially based on the detected motion activity.

30. The pair of earphones according to claim 29, wherein detecting the motion activity includes determining whether the wearer is walking, at least in part, based on a comparison of the power and threshold of the frequency domain representation of the motion information.

31. The pair of earphones according to claim 28, wherein the at least one processor is configured to perform an automatic zeroing procedure to determine an updated reference axis corresponding to the direction the user is facing, and the rotation information determined is at least partially based on the reference axis.

32. Determining the aforementioned reference axis means Comparing the variance of head orientation angle to a threshold, It is determined that the distribution of the head orientation angle falls below the threshold for a predetermined duration, A pair of earphones according to claim 31, comprising determining the updated reference axis in response to the determination that the dispersion of the head orientation angle is below the threshold over a predetermined duration.

33. The pair of earphones according to claim 32, wherein the threshold is set based at least in part on the variance.

34. The pair of earphones according to claim 28, wherein the at least one processor is further configured to perform a calibration procedure which is configured to correct the tilt of at least one of the earphones of the pair in the ear of a wearer of the pair of earphones.

35. The pair of earphones according to claim 34, wherein the calibration procedure includes determining a corrected rotation based on a quaternion representing the orientation of the wearer's head.

Images