Efficient direction tracking by predicting future directions

Abstract

The present disclosure relates to a method and system for predicting a future direction of a direction tracker (100). The method includes the steps of obtaining a sequence of angular velocity samples, each of which indicates an angular velocity at a time instant, and obtaining a sequence of angular acceleration samples, each of which indicates an acceleration or deceleration of the angular velocity at each time instant. The method further includes determining (S5a) a predicted direction of the direction tracker (100) based on a first-order prediction of the accumulated rotation of the direction tracker (100) for each time instant when the angular velocity is accelerating, and determining (S5c) a predicted direction of the direction tracker (100) based on a second-order prediction of the accumulated rotation of the direction tracker (100) for each time instant when the angular velocity is decelerating.

Description

[Technical field]

[0001] [Related Applications] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 245,312, filed September 17, 2021, and U.S. Provisional Patent Application No. 63 / 374633, filed September 6, 2022. Both applications are incorporated herein by reference in their entireties.

[0002] [Technical field] The present disclosure relates to a method and system for predicting the future orientation of a motion tracker. [Background technology]

[0003] In many applications, such as navigation, presentation of audio content, and presentation of image or video content, the orientation of a user or user device is tracked over time to control certain operations. For example, most smartphones include an orientation tracker configured to determine the orientation of the smartphone, such as to determine when to switch from a vertical to a horizontal display mode, to determine the orientation of a map displayed on the smartphone, or to allow the smartphone to act as a window into an AR or VR environment.

[0004] Similarly, directional tracking of a user consuming audio content is becoming increasingly important to enable a user to enhance their sense of immersion, for example by performing a rotational transformation of an audio signal rendered to a set of earphones or earplugs based on the user's orientation. For example, a complex acoustic scene including multiple spatial audio objects can be rendered into a binaural audio signal, providing an accurate representation of the acoustic scene of a user listening to the binaural audio signal. By further taking into account the user's current direction, obtained for example by measuring the user's direction with a directional sensor, a convincing acoustic effect can be created in which the user can move within the virtual acoustic scene as opposed to the virtual acoustic scene following the user (for example, as in the case of rendering conventional binaural or stereo audio to earphones).

[0005] For example, a user may be listening to binaural audio representing an acoustic scene in which the main audio source is located directly in front of the user. If the user turns around in a counterclockwise direction, the orientation tracker tracks the change in orientation and the binaural audio is processed accordingly, so that the user perceives the main audio source as being located to the right of the user, then directly behind the user, and finally to the left of the user. Orientation tracking can thus move the user relative to objects in the (virtual) acoustic scene. Summary of the Invention

[0006] The problem with this type of orientation tracking for rotational audio processing is that it takes a non-negligible time to perform audio processing and rendering that depends on the user's current orientation. Thus, there is a distracting delay between the rotated audio content and the user's actual orientation. In some cases, the orientation tracker is part of a set of wireless earphones, but the audio processing is performed by a smartphone connected to the wireless earphones. This means that in addition to the audio processing and rendering, non-negligible additional time is required for information transfer between the earphones and the smartphone. In such a situation, the time between the orientation tracker measuring a first orientation of the user and the first playback of audio content rotated to the first orientation by the wireless earphones can be 200 milliseconds or more. Within this time, the user may move in a second orientation that is different from the first orientation, causing a distracting delay effect as soon as the user's orientation changes.

[0007] To avoid these drawbacks, a solution is proposed in which the user's future direction is predicted and the predicted direction is used for audio processing instead of the current direction. If the predicted direction were accurate, distracting delays would be mitigated, but most currently proposed methods for prediction are inaccurate, computationally intensive, prone to introducing undesirable audio artifacts, or unsuitable for implementation in low-performance and / or portable battery-powered devices.

[0008] It is therefore an object of the present disclosure to provide an improved method for performing directional tracking.

[0009] A first aspect of the invention relates to a method for predicting a future direction of a direction tracker, the method comprising the steps of: acquiring a sequence of angular velocity samples, each angular velocity sample indicative of an angular velocity of the direction tracker at a time instant; and acquiring a sequence of angular acceleration samples, each angular acceleration sample indicative of one of an acceleration and a deceleration of the angular velocity of the direction tracker at each time instant. The method further comprises, for each time instant during which the angular velocity is accelerating, determining a predicted direction of the direction tracker after a predefined prediction period based on a first order prediction of an accumulated rotation of the direction tracker, the first order prediction of the accumulated rotation being based on an accumulated rotation of a current angular velocity occurring during a prediction time. For each time instant during which the angular velocity is decelerating, the method comprises determining a predicted direction of the direction tracker after a predefined prediction period based on a second order prediction of the accumulated rotation of the direction tracker, The secondary prediction of the accumulated rotation is based on the accumulated rotation of the current angular velocity and the current deceleration that occurred during a prediction time.

[0010] It is understood that the term acceleration can mean either acceleration or deceleration in some contexts. For example, where A and B are non-negative numbers, the angular velocity is A rad / s, the angular acceleration is B rad / s 2 An object moving at a speed of 1000 km / s has an angular velocity of 1000 km / s. 2 means that the angular acceleration increases by -B rad / s 2 is the angular velocity in Bragg rad / s 2 Similarly, the angular velocity is -A rad / s, and the angular acceleration is B rad / s. 2 An object moving at a speed of 1000 km / s has an angular velocity of 1000 km / s. 2 This means that the angular acceleration decreases (decels) by -B rad / s 2 is the angular velocity in Bragg rad / s 2 Therefore, acceleration and velocity values ​​with positive and negative signs can be used interchangeably with the terms acceleration (angular velocity increases) and deceleration (angular velocity decreases).

[0011] The predetermined prediction period can be expressed in time units, or equivalently in number of samples. It is further understood that the samples can be original samples from the direction tracker, or a combination (e.g., in terms of mean or average) of multiple original samples with finer granularity from the direction tracker. It is envisioned that the samples can be samples of a continuous signal, or a downsampled / upsampled version of the original sequence of samples. For example, each sample can represent angular velocity and angular acceleration during a corresponding time segment of a series of time segments, where the time segments may be partially overlapping or non-overlapping in time.

[0012] A first aspect of the present invention is based at least in part on the realization that by using a simple first-order predictor in situations where the direction tracker is accelerating and a more sophisticated second-order predictor in situations where the direction tracker is decelerating, a computationally efficient method for predicting the direction of a head tracker for audio processing is achieved. When a user listens to audio content that is rendered based on the user's direction, the inventors have realized that a first-order predictor, although less accurate, is sufficient in situations where the user's direction changes are accelerating, and that when the user's direction changes are decelerating, it is more important to allocate computational resources to accurate predictions.

[0013] When accelerating, the user is often unable to notice small discrepancies between the user's instantaneous direction and the rendered audio content. However, as soon as the user's direction starts to decelerate (even if the initial speed is high), the user immediately becomes sensitive to the discrepancy between the instantaneous direction and the rendered audio content. For this reason, when the direction tracker that tracks the user's direction detects a deceleration, a more computationally expensive second-order predictor is used, and when the direction tracker that tracks the user's direction detects an acceleration, a less computationally expensive first-order predictor is used.

[0014] In some embodiments, the method further comprises, for each time point at which the angular velocity falls below a predetermined velocity threshold, determining a predicted orientation of the direction tracker after a predetermined prediction period based on a next prediction of the accumulated rotation of the direction tracker.

[0015] That is, a computationally efficient first-order predictor is used for any time point where the angular velocity samples indicate an angular velocity below a predefined velocity threshold. For small velocities, the discrepancy between the user's instantaneous direction and the rendered audio content is small and less noticeable even when the user's direction changes are decelerating. It is understood that the angular velocity threshold imposes a limit on the absolute angular velocity, i.e., a limit regardless of the sign (positive or negative) of the angular velocity.

[0016] In some embodiments, for each time point when the angular velocity is decelerating at a deceleration that exceeds a deceleration threshold, the predicted orientation of the direction tracker after the predetermined prediction time is based on a truncated second order prediction of the accumulated rotation of the direction tracker; The shortened secondary prediction of the accumulated rotation is based on the accumulated rotation of the current angular velocity and the current deceleration that occurred during a second prediction time, the prediction time being longer than the second prediction time.

[0017] For an angular velocity to be decelerating with a current velocity exceeding a deceleration threshold, it means that the rate at which the angular velocity approaches zero exceeds a predefined deceleration threshold. For example, if an object has an angular velocity of +A rad / s and an acceleration of -B rad / s 2 (meaning deceleration is occurring) and the deceleration threshold is (in absolute value) B / 2 rad / s 2 If so, then the acceleration / deceleration exceeds the deceleration threshold because the deceleration rate exceeds the threshold rate of deceleration.

[0018] Therefore, a modified version of the quadratic predictor (the truncated quadratic predictor) is used in high rate deceleration situations, as the inventors recognize that overshooting audio artifacts can occur at high rate decelerations and are mitigated or eliminated entirely by using a truncated quadratic predictor.

[0019] In some embodiments, the method further includes determining a scaling factor for each time point when the angular velocity is decelerating, where a secondary prediction of the accumulated rotation of the orientation tracker is based on scaling the primary prediction with the scaling factor.

[0020] That is, to further improve the computational efficiency, it is realized that the primary prediction and the secondary prediction (actually even the shortened secondary prediction) are linked via a single scaling factor. Therefore, when switching from one type of prediction to another, evaluation of different functions is not necessarily required, since it is sufficient to update the value of a single scaling factor. For example, the secondary prediction is calculated by multiplying the primary prediction by the scaling factor.

[0021] According to a second aspect of the present invention, there is provided a direction prediction device configured to perform the method according to the first aspect.

[0022] According to a third aspect of the invention there is provided a computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the method of the first aspect.

[0023] Any feature described in the context of a method may have a corresponding feature in a system or apparatus, and vice versa. [Brief description of the drawings]

[0024] Aspects of the present invention will now be described in more detail with reference to the accompanying drawings showing presently preferred embodiments of the invention.

[0025] [Figure 1] 1 illustrates a direction tracker in the tracker (T) coordinate system according to some embodiments.

[0026] [Diagram 2]1 illustrates a tracker (T) and a direction tracker in a floating (F) coordinate system according to some embodiments.

[0027] [Diagram 3] 1 is a flow chart illustrating a method for predicting a future direction of a directional factor according to some embodiments.

[0028] [Figure 4] 11 is a flow chart illustrating different predictions of angular velocity that will occur during a prediction time according to some embodiments.

[0029] [Figure 5A] 1 illustrates a user facing a first direction and listening to an audio presentation that is adapted based on the direction the user is facing, according to an embodiment.

[0030] [Figure 5B] 13 illustrates a user facing in a second direction and listening to an audio presentation that is adapted based on the direction the user is facing, according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] The systems and methods disclosed in this application may be implemented as software, firmware, hardware, or a combination thereof. In a hardware implementation, the division of a task does not necessarily correspond to a division into physical units. On the contrary, one physical component may have multiple functions, and one task may be performed jointly by multiple physical components.

[0032] The computer hardware may be, for example, a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smart phone, a web appliance, a network router, a switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions that the computer hardware is to take. Furthermore, this disclosure should be directed to a collection of computer hardware that individually or collectively executes instructions to perform any one or more of the concepts discussed herein.

[0033] Certain or all components may be implemented by one or more processors that accept computer-readable (also referred to as machine-readable) code that includes an instruction set that, when executed by the one or more processors, performs at least one of the methods described herein. Any processor capable of executing an instruction set (sequential or otherwise) that specifies operations to be performed is included. Thus, an example is a standard processing system (i.e., computer hardware) that includes one or more processors. Each processor may include one or more of a CPU, an image processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem that includes a hard drive, SSD, RAM, and / or ROM. A bus subsystem may be included for communication between components. Software may reside in the memory subsystem and / or in the processor during execution by the computer system.

[0034] One or more processors may operate as stand-alone devices or may be connected to other processors, e.g., networked. Such a network may be built on a variety of different network protocols and may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

[0035] Software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those skilled in the art, the term "computer storage media" includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, various forms of physical (non-transitory) storage media, such as EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage, or any other medium accessible by a computer that can be used to store the desired information. Additionally, as known to those skilled in the art, communication media (transitory) may also typically embody computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media.

[0036] Referring to FIG. 1, a direction tracker 100 configured to determine direction about three axes 120, 121, 122 is shown.

[0037] The position of an object in the vicinity of the direction tracker may be defined with respect to the tracker (T) coordinate system, relative to axes TX 120, TY 122 and TZ 121. This coordinate system is called the T-frame, and the axes 120, 121, 122 of the T-frame are fixed to the direction tracker 100 and therefore rotate with the direction tracker 100.

[0038] To enable directional tracking, the direction tracker 100 may include one or more sensors that enable the direction of the device to be determined. The one or more sensors may be a gyro, a magnetometer, and / or an accelerometer. Also, while the embodiment of FIG. 1 determines the direction of the direction tracker 100 about three axes 120, 121, 122, the direction tracker 100 may be configured to determine the direction only about one or two axes.

[0039] The orientation tracker 100 can be attached to a user device, such as a set of earphones, a smartphone, a smartwatch, or any type of user device, so as to be able to determine the orientation of the user device (and / or the user device being worn or held by the user) about one, two or three axes.

[0040] The orientation tracker 100 can include one, two or three gyros, each measuring the rate of rotation (angular velocity 130, 131, 132) about one axis of rotation. A three gyro configuration is sometimes referred to as a three-axis gyro.

[0041] The orientation tracker 100 may include three magnetometers, each measuring a component of the local magnetic field along one axis, the three magnetometer configuration sometimes being referred to as a three-axis magnetometer.

[0042] The orientation tracker 100 may include three accelerometers, each measuring the acceleration of the device along one axis. A configuration of three accelerometers is sometimes referred to as a three-axis accelerometer.

[0043] The sensors of the orientation tracker 100 are mounted to capture orientation measurements relative to three axes of a right-handed coordinate system, the axes being labelled as X-axis 120 , Z-axis 121 and Y-axis 122 .

[0044] A gyro measurement can be represented as a vector of three time-varying quantities.

number

[0045] Note that if fewer than three gyros are used, one or more elements of the Gyr(t) vector may be zero or omitted. For example, if two gyros are used, the vector Gyr(t) contains only two elements or two non-zero elements for each time t.

[0046] The vector Gyr(t) is the unit vector U rot It may be represented by a single rotation component rotating at ω(t) radians / second about an axis defined by (t).

number

number

[0047] Similarly, the magnetometer and accelerometer measurements may be expressed as single rotational components or as vectors of three time variables each indicating angular velocity about each of the three axes 120, 121, 122, as follows:

number

[0048] Gyro, magnetometer or accelerometer measurements may be sampled at regular time intervals, and if the time between samples is ΔT, then the sampled value at sample number n may be defined for a gyro measurement as:

number

number

[0049] 2 shows a fixed (or floating) reference frame defined by three axes, the FX axis 220, the FY axis 222 and the FZ axis 221. This fixed (or floating) reference frame is called the F frame. Without loss of generality, it can be assumed that the origin of the F frame coincides with the origin of the T frame. The three axes of the F frame 220, 221, 222 define a nearly stationary reference frame.

[0050] The direction tracker 100 resides in the same space spanned by the T and F frames, respectively, and the position of any object residing in the space around the direction tracker 100 can be defined according to its coordinates in either the T or F frame.

number

number

[0051] The orientation of the orientation tracker 100 in F frames is given by the rotation matrix M TF It is understood that the rotation matrix M TF is the unit quaternion Q TF It is also understood that the invention can be fully described by:

number

number

number

[0052] In one embodiment, the direction tracker includes a gyro sensor that measures the angular velocity 131, 132, 133 of the direction tracker 100 about one or more of the three axes 120, 121, 122 of the T-frame. This angular velocity data may be in the form of three angular velocity measurements defined in Equation 1, or may be a time sampled quantity, where sample n is sampled at time t=ΔTn as shown in Equation 5.

[0053] For example, the time sampled gyroscope values ​​are converted to unit vector / rotation form according to the following formula:

number

[0054] Using the time-sampled unit vector / rotation data at sample time n, the orientation of the orientation tracker 100 at sample time n is given by the quaternion Q TF It can be calculated for [n] as follows:

number

[0055] Those skilled in the art will recognize that Q from Equation 15 TF It will be appreciated that while [n] determines the orientation of the direction tracker device relative to the F frame, the orientation of the F frame relative to the world frame is arbitrary, since at each sample interval n, the orientation is determined based on the orientation at the previous sample interval n-1. To take advantage of the additional quaternion updates at each sample interval, Equation 15 can be modified with an incremental rotation to adjust the orientation of the F frame relative to the world frame, for example, to ensure that the F frame's FZ axis 221 points approximately vertically and the FX axis 220 or FY axis 222 points approximately toward magnetic north. This incremental rotation used to slowly align the F frame with the world frame can be calculated using an accelerometer and / or magnetometer. If the direction tracker 100 only includes an accelerometer, the resulting incremental rotation operation will only maintain a valid alignment of the FZ axis 221 (using gravity in the vertical direction), while the FX axis 220 and FY axis 222 can point in any direction and may drift over time due to inaccuracies in the angular velocity measurements. However, if the orientation tracker 100 includes a magnetometer, the incremental orientation maintains a valid alignment of the F frame with respect to the world frame.

[0056] Quaternion △Q F [n] is defined as the quaternion that represents the incremental rotation of the F frame in sample interval n that realigns the F frame with the external frame. The incremental rotation ΔQ F [n] can be used to form an extended version of Equation 15 that takes into account the incremental rotation of the F frame.

number

[0057] If incremental rotation is not considered, Equation 15 can be used instead of Equation 17. Alternatively, if incremental rotation is not considered, ΔQ F [n] can be set to:

number

number

[0058] To predict the orientation of the orientation tracker 100 for a number of future samples, the future orientation:

number

number

number

number

[0059] Prediction Quaternion P L Another derivation of [n] is that the predicted rotation is the incremental quaternion ΔQ T It can be defined by observing that it is equivalent to multiple repeated applications of [n] (which is equivalent to raising the increment quaternion to a power of L).

number

[0060] Second-order prediction γ2 pred [n] is generated by considering the angular acceleration a[n], which, with units of radians per sample squared, can be determined as follows:

number

number

[0061] Using the future predicted values ​​of angular velocity, as shown in Equation 23, the cumulative angular rotation over L future sample intervals can be predicted by:

number

[0062] The quadratic predictor γ2 in Eq. pred [n] is the first-order predictor γ1 according to the following formula: pred It can also be rewritten in terms of [n] and the scaling factor β[n]:

number

number

[0063] In one embodiment, the prediction of future angular rotations of the orientation tracker 100 is adapted to use a first-order prediction method in some situations and a second-order prediction method in other situations. This is called a sesqui-order predictor, which is defined as follows:

number

[0064] Setting β[n]=1 results in a future angle prediction that corresponds to the first prediction, and the following settings:

number

[0065] With further reference to FIG. 3, a flow chart illustrating a method for selecting β[n] according to one implementation is shown.

[0066] Optional step S1 is to determine whether ω[n] is greater than or equal to a predetermined angular velocity threshold ω minFor example, determining whether ω min is between 0.02 and 0.10 rad / sec, which is suitable for head tracking purposes, or preferably about 0.05 rad / sec. min (i.e., |ω[n]|<ω min ), the method proceeds to step S5a where β[n] is selected such that β[n]=1 (indicating first order prediction) and in step S6 a second order predictor is calculated using Equation 27. Alternatively, if ω[n] exceeds the angular rate threshold, the method proceeds to step S2.

[0067] Step S2 involves calculating the value of entity r[n], which is:

number

[0068] Once r[n] has been calculated, the method proceeds to step S3 where it is determined whether r[n] is positive (i.e., whether r[n]≧0). If r[n] is positive, this indicates that the angular velocity of the direction tracker 100 is increasing. If r[n] is positive, the method proceeds to step S5a where β[n] is selected such that β[n]=1 and the suborder predictor is calculated in step S5 using Equation 27. Alternatively, if r[n]<0, the method proceeds to step S4.

[0069] Step S4 involves determining whether r[n] is less than -1 (i.e., whether r[n]<-1). If r[n]<-1, this indicates that the angular velocity of the direction tracker 100 is decreasing due to deceleration, which means that the velocity will drop to zero before sample interval n+L. If r[n]<-1, the method proceeds to step S5c, where β[n] is selected such that:

number

number

[0070] In some embodiments, step S3 is performed before step S2 (e.g., simultaneously with step S1), and step S3 is simplified to simply determine whether the direction tracker 100 is accelerating. If the direction is accelerating (or the acceleration is zero), the method proceeds to step S5a. In particular, the value r[n] is calculated only if it is necessary to distinguish between r[n]<-1 or -1≦r[n]<0.

[0071] 4, the predicted angular velocity over L prediction samples is shown. Above, when the orientation tracker is nearly stationary (e.g., ω min An implementation has been provided where the value of β[n] is chosen to be β[n]=1 (thus providing a first order prediction operation) whenever the vector is rotating with an angular velocity less than n (rotating with an angular velocity less than n) or accelerating. This corresponds to a predicted angular velocity 301 that is constant from the current sample n to the predicted sample n+L.

[0072] The value of β[n] is selected to provide a second order prediction when the direction tracker 100 is slowly decelerating with insufficient deceleration to bring the angular velocity of the direction tracker to zero during the next L sample intervals (assuming acceleration is constant for the next L sample intervals), such as when -1≦r[n]<0. This corresponds to a predicted angular velocity 302 that continues to decrease with the deceleration at sample n, but does not reach zero before sample n+L.

[0073] When the direction tracker 100 is decelerating rapidly with a deceleration sufficient to cause the angular velocity of the direction tracker 100 to zero over the next L sample intervals (assuming acceleration is constant over the next L sample intervals), the value of β[n] is selected to provide a second order prediction over a shorter time interval L′, where the shorter time interval is number of sample periods:

number

[0074] Thus, the deceleration threshold 305 is still the most rapid deceleration that is assigned a secondary prediction over L prediction samples, but if the direction tracker 100 decelerates more rapidly, the prediction of angular velocity over L samples becomes a secondary prediction only up to sample n+L', and then becomes 0 until sample n+L.

[0075] In view of the above, a method is provided for predicting the orientation of the direction tracker 100 that uses a computationally inexpensive primary predictor for situations in which the direction tracker 100 is accelerating or moving with a small angular velocity. The more computationally expensive secondary predictor is reserved for situations in which the direction tracker is decelerating; in situations in which the direction tracker is decelerating fast enough that its angular velocity reaches 0 within the prediction time L, the secondary predictor predicts that the direction tracker will reach angular velocity 0 at L' and remain stationary until L.

[0076] In one implementation, the value of β[n] is calculated using pseudocode segment 1:

number

[0077] Unit vector U rotInstead of expressing the angular velocity as a function of time (t) and a single rotational component ω(t), the angular velocity can be expressed as a vector Ω of gyroscope values ​​(scaled to units of radians per sample):

number

number

number

number

[0078] It will be appreciated that the method of Figure 3 above can be implemented similarly with the angular velocity represented by the vector Ω and the calculation of r[n] from Equation 33. Furthermore, in determining whether the angular velocity Ω is less than the threshold ωmin in step S1, a comparison can be made to determine whether:

number

number

[0079] In some implementations, the quaternion vector is determined according to a simpler approximation. For example, ΔQT[n] is the sum of ΔQ approx T [n], given by:

number

[0080] That is, the trigonometric functions from the above equations 16 and 19: [Number] may be replaced by the first term, or the 11th or more terms, in each Taylor series expansion: [Number] It is assumed that other approximations of the trigonometric functions from equations 16 and 19 may be used to further ease the computational efficiency.

[0081] Here, the direction quaternion from equation 17 is the approximate ΔQ approx T [n] is used to first calculate the entity Q' TF [n] can be calculated by calculating as follows: [Number] This is generally not normalized. The direction quaternion Q' TF [n] may be normalized according to the following equation: [Number] Alternatively, the direction quaternion Q'TF[n] may be normalized according to the following formula to avoid the costly computation of the square root from Equation 36:

number

[0082] In one implementation, the current direction, Q' of the direction tracker 100 TF [n], and a quaternion Q representing the future direction of the direction tracker pred TF [n+L] may be determined in pseudocode segment 2.

number

[0083] In another implementation, the scaling factor β[n] is based on a previous scaling factor, such as β[n-1]. The portion of the pseudocode related to calculating β[n] may be replaced in PSEUDOCODE SEGMENT 3, where the previous value of the scaling factor β[n], denoted as prior_beta, is added as a saved state in PSEUDOCODE SEGMENT 2 above.

number

[0084] Where the constant A is selected to provide an appropriate incremental adjustment of β[n], for example, A may be selected as A=0.01. It is further contemplated that two or more than three incremental adjustment levels may be used, and that the coefficients of each level of incremental adjustment (4, 2, and 1 in the above example) may be selected differently. For example, four incremental adjustment levels may be used, with coefficients of each level being 12, 9, 6, and 3, respectively.

[0085] Furthermore, the operator "c?x:y" is short for "if c is true then x else y", where c is a boolean value, x is a result statement (evaluated when c is true), and y is an alternative statement (evaluated when c is false). For example, at the first increment level of the "else if" clause in the above pseudo-code segment 3, when "-2*beta*dotp < omega_squared" is true, β[n] increases by 4A.

[0086] Therefore, the value of β[n] is incrementally corrected in each sample period so that the correct value is approximated over several sample intervals. That is, by performing the above incremental adjustment of β[n], a sufficiently accurate β[n] can be obtained in a computationally more efficient way than directly calculating r[n] using Equation 33 and then directly calculating β[n] using Equation 29 or Equation 30.

[0087] Figures 5a and 5b show a user 10 listening to a spatial audio representation including at least one spatial audio object 20a, 20b (e.g., using headphones 11). The user 10 is facing a first direction D in Figure 5a and is presented with a related audio presentation. Here, for example, the spatial audio objects 20a, 20b are recognized as being located in front of the user 10. When the user 10 changes the orientation of the head to face a second direction D' different from the first direction D, the audio presentation changes accordingly, and although the user 10 has changed their orientation, the user 10 perceives the audio scene as remaining stationary. For example, the spatial audio object 20a, which was perceived to be located in front of and to the left of the user 10 when the user 10 was facing the first direction D, may be presented to the user 10 so as to appear in front of the user 10 when the user faces the new direction D'. In the illustrated scenario, the user 10's head is rotated left in the horizontal plane at an angle α, but it is understood that the direction of the user 10's head may change by rotation about any of the three axes when the user 10 turns their neck, tilts, or nods.

[0088] In some implementations, the orientation of the user 10 tracked by a direction tracker 100 (e.g., provided on the headphones 11), here the orientation of the user 10 determined by the direction tracker 100, is used to modify the audio presentation based on the orientation of the user 10. It takes some time (e.g., up to 200 milliseconds or more) for the audio processing system to modify the presentation according to the current orientation of the user 10. Thus, if at a first time t1 it is determined that the user 10 is facing in a first direction D, the audio processing system starts a process of modifying the audio presentation to correspond to the user 10 facing in the first direction. This process takes some time and is completed at a second time t2 when the audio presentation corresponding to the user 10 facing in the first direction D is presented to the user 10. However, between the first time t1 and the second time t2, the user 10 has moved to face in the second direction D', which means that an incorrect presentation is presented which causes a distracting delay effect to the user.

[0089] To this end, the method for predicting the orientation of the direction tracker 100 according to some embodiments can be used to predict the direction of the user 10 and initiate a process of modifying the audio presentation to correspond to the predicted direction of the user 10. Most likely, the user 10 will be facing the predicted direction when the audio presentation associated with the predicted direction is presented, which means that distracting delay effects are reduced or completely eliminated. In particular, if the time required to modify the audio presentation after the direction is known is a certain amount of time (e.g., a certain number of milliseconds or samples), the predicted time L can be adjusted accordingly. For example, if the audio processing requires x samples, the predicted time may also correspond to x samples.

[0090] Unless otherwise indicated, and as will be apparent from the discussion that follows, discussions throughout this disclosure using terms such as "processing," "computing," "calculating," "determining," "analyzing," and the like refer to the operations and / or processing of computer hardware or computing systems or similar electronic computing devices that manipulate and / or convert data represented as physical, e.g., electronic, quantities into other data similarly represented as physical quantities.

[0091] It should be understood that in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the present invention and aiding in understanding one or more of the various aspects of the present invention. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, aspects of the present invention lie in fewer than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the present invention. Furthermore, although some embodiments described herein include some features and not other features included in other embodiments, as will be understood by those skilled in the art, combinations of features of different embodiments are meant to be within the scope of the present invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

[0092] Furthermore, some of the embodiments have been described as methods or combinations of method elements that can be implemented by a processor of a computer system or by other means for performing the functions. Thus, a processor with the necessary instructions for implementing such a method or method elements forms a means for performing the method or method elements. It should be noted that when a method includes several elements, e.g. several steps, no order of these elements is implied unless otherwise specified. Furthermore, the elements described herein of the device embodiments are examples of means for performing the functions performed by the elements for implementing the embodiments of the present invention. In the description provided herein, numerous specific details have been described. However, it is understood that the embodiments of the present invention may be practiced without these specific details. In other instances, well-known methods, structures, and techniques have not been shown in order not to obscure an understanding of the description of the present invention.

[0093] Those skilled in the art will appreciate that aspects of the present invention are not limited to the above-described embodiments. Rather, many modifications and variations are possible within the scope of the appended claims. For example, it is envisioned that the prediction of future direction uses computationally efficient quaternions, but the prediction of future direction may also be based on a calculated rotation matrix.

Claims

1. 1. A computer-implemented method for predicting a future direction of a direction tracker for audio processing, the method being executed by one or more processors and comprising: acquiring a sequence of angular velocity samples, each angular velocity sample indicative of an angular velocity of the direction tracker at a point in time; acquiring a sequence of angular acceleration samples, each angular acceleration sample indicating an acceleration or deceleration of the angular velocity of the direction tracker at a respective time; for each time point during which the angular velocity is accelerating, determining a predicted direction of the direction tracker after a predetermined prediction period based on a first-order prediction of a cumulative rotation of the direction tracker, the first-order prediction of the cumulative rotation being based on a cumulative rotation of a current angular velocity occurring during a prediction time; for each time point during which the angular velocity is decelerating, determining a predicted direction of the direction tracker after a predetermined prediction period based on a secondary prediction of a cumulative rotation of the direction tracker, the secondary prediction of the cumulative rotation being based on a current deceleration and a cumulative rotation of a current angular velocity occurring during a prediction time; obtaining audio content; processing the audio content based on the predicted future direction of the direction tracker to obtain a rotated representation of the audio content; rendering the rotated representation of the audio content on at least one loudspeaker; A method comprising:

2. for each time point when the angular velocity is below a predetermined velocity threshold, determining a predicted orientation of the direction tracker after a predetermined prediction period based on the first order prediction of the cumulative rotation of the direction tracker; The method of claim 1 further comprising:

3. for each time point when the angular velocity is decelerating at a deceleration that exceeds a deceleration threshold, the predicted orientation of the direction tracker after the predetermined prediction time is based on a truncated second order prediction of the accumulated rotation of the direction tracker; 2. The method of claim 1, wherein the shortened secondary prediction of the cumulative rotation is based on the cumulative rotation of the current angular velocity and the current deceleration that occurred during a second prediction time, the prediction time being longer than the second prediction time.

4. The method of claim 3 , wherein the deceleration threshold is based on the current angular velocity and the predetermined prediction time.

5. 4. The method of claim 3, wherein the second predicted time is based on the period it takes the direction tracker to reach zero angular velocity at the angular velocity and angular deceleration of a current sample.

6. determining a scaling factor for each time point when the angular velocity is decelerating; The method of claim 3 , wherein the secondary prediction of the accumulated rotation of the orientation tracker is based on the primary prediction scaled by the scaling factor.

7. the scaling factor is assigned a first value each time the angular velocity is decelerating at a deceleration greater than the deceleration threshold, the first value being based on a first function of a current angular velocity sample and a current angular acceleration sample; 7. The method of claim 6, wherein the scaling factor is assigned a second value for each time point when the angular velocity is decelerating at a deceleration below the deceleration threshold, the second value being based on a second function of the current angular velocity sample and the current angular acceleration sample.

8. The method of claim 6 , wherein the scaling factor is further based on a scaling factor from at least one previous time point.

9. The method of claim 1 , further comprising obtaining the sequence of angular acceleration samples by determining a sequence of angular acceleration samples based on the obtained sequence of angular velocity samples.

10. for each said time instant, obtaining a direction sample indicative of a current direction of the direction tracker; The method of claim 1 , wherein the future direction of the direction tracker is further based on the current direction of the direction tracker.

11. The future orientation of the orientation tracker is relative to a floating reference frame, and the method comprises: obtaining an angular shift of the floating frame of reference relative to a global frame of reference; adjusting the predicted orientation of the direction tracker by the angular shift of the floating reference frame relative to a global reference frame to obtain a predicted orientation of the direction tracker relative to the global reference frame; The method of claim 1 further comprising:

12. The method of claim 1 , wherein the cumulative rotation of the orientation tracker is represented by a quaternion.

13. The method of claim 1 , wherein the angular velocity samples and angular acceleration samples indicate the angular velocity and rotation about one, two, or three axes.

14. 1. A direction prediction system for predicting a future direction of a direction tracker for audio processing, the direction prediction system including one or more processors, the one or more processors comprising: acquiring a sequence of angular velocity samples, each angular velocity sample indicative of an angular velocity of the direction tracker at a point in time; acquiring a sequence of angular acceleration samples, each angular acceleration sample indicative of an acceleration or deceleration of the angular velocity of the direction tracker at a respective time; for each time point during which the angular velocity is accelerating, determining a predicted orientation of the direction tracker after a predetermined prediction period based on a first-order prediction of a cumulative rotation of the direction tracker, the first-order prediction of the cumulative rotation being based on a cumulative rotation of the current angular velocity occurring during a prediction time; for each time point during which the angular velocity is decelerating, determining a predicted orientation of the direction tracker after a predetermined prediction period based on a secondary prediction of a cumulative rotation of the direction tracker, the secondary prediction of the cumulative rotation being based on a cumulative rotation of a current angular velocity and a current deceleration that has occurred during the prediction time; Get audio content, processing the audio content based on the predicted future direction of the direction tracker to obtain a rotated representation of the audio content; rendering the rotated representation of the audio content to at least one loudspeaker; A direction prediction system configured to:

15. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1 to 13.

Images