Spatial audio for wearable devices
By receiving and generating posture estimation in the wearable device and refining spatial audio data, the problem of delay between motion detection and spatial audio playback in the wireless head-mounted device is solved, and the reproduction accuracy and user experience of the audio source are improved .
Patent Information
- Application Number
- CN202411104829.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2019-10-22
- Filing Date
- 2020-10-21
- Publication Date
- 2024-12-03
AI Technical Summary
When using remote devices, existing wireless head-mounted devices have a delay of about 400 milliseconds between motion detection and spatial audio playback, affecting the user experience, especially when using sound to guide the user to move.
By receiving and generating posture estimates in a wearable device, refining spatial audio data to reduce delays, updating posture estimates with metadata captured by motion sensors and cameras, spatial audio transformation based on global and local coordinate reference frames, ensuring audio sources accurate reproduction.
Effectively reduces the motion to audio output delay of wearable user devices, improves the accuracy and user experience of spatial audio reproduction, and ensures that the audio source is projected in the correct position and orientation.
Smart Images

Figure CN119065500A_ABST
Abstract
Description
[0001] Description of the case
[0002] This application is a divisional application of Chinese invention patent application No. 202080047687.8, with a filing date of October 21, 2020. Background Art
[0003] To provide a more accurate, immersive, or enjoyable user experience, some head mounted display devices include spatial audio features that utilize speakers that generate sound for the user. Spatial audio refers to the sound reproduced by the device so that the listener perceives the sound as coming from a specific or near direction. Spatial audio rendering techniques have various applications, such as in virtual reality (VR) or augmented reality (AR) systems, navigation systems or other travel aids, and real-time aviation systems.
[0004] Without any mitigation, glasses or other head-mounted accessory devices that support wireless audio and include motion sensors and speakers can have latency in the range of 400 milliseconds (ms) between motion detection and spatial audio playback when utilizing a remote device for the computational operations associated with rendering spatial audio. This can negatively impact the user experience, particularly for implementations that use sound to guide the user or respond to user movement. Summary of the invention
[0005] According to one aspect, a method, the method comprising:
[0006] receiving, with the wearable device, spatial audio data and a first pose estimate corresponding to the wearable device;
[0007] generating, using the wearable device, a second pose estimate corresponding to the wearable device;
[0008] refining the spatial audio data based on the second pose estimate using the wearable device; and
[0009] Utilizing wearable devices to generate sounds based on refined spatial audio data.
[0010] According to some aspects, the method can include one or more (eg, all) of the following features (or any combination thereof).
[0011] The method may include: generating first posture metadata using a motion sensor of the wearable device during a first time period, wherein a first posture estimate is generated based on the first posture metadata. The method may further include: generating second posture metadata using the motion sensor during a second time period, wherein the wearable device generates a second posture estimate based on the second posture metadata. The method may further include: generating first posture metadata using the motion sensor of the wearable device during the first time period; and generating second posture metadata using a camera of a mobile device during the first time period, wherein the mobile device generates the first posture estimate based on the first posture metadata and the second posture metadata. The method may further include: generating third posture metadata using the motion sensor during the second time period, wherein the wearable device generates the second posture estimate based on the third posture metadata. In addition, refining the spatial audio data may include: calculating a local spatial audio transform having a local coordinate reference frame based on a global spatial audio transform having a global coordinate reference frame and the second posture estimate, wherein the global spatial audio transform may indicate a position and orientation of an audio source to be simulated when reproducing the spatial audio data in the world space.
[0012] According to one aspect, a system comprises:
[0013] A wearable device, comprising:
[0014] A processor configured to execute computer readable instructions that, when executed, cause the processor to:
[0015] receiving spatial audio data corresponding to a first pose estimate of the wearable device;
[0016] generating refined spatial audio data by modifying the spatial audio data based on a second pose estimate of the wearable device; and
[0017] Sound is generated based on the refined spatial audio data.
[0018] According to another aspect, a system comprises:
[0019] A wearable device, comprising:
[0020] A processor configured to execute computer readable instructions that, when executed, cause the processor to:
[0021] receiving spatial audio data and a first pose estimate corresponding to a wearable device;
[0022] generating a second pose estimate corresponding to the wearable device;
[0023] refining the spatial audio data based on the second pose estimate; and
[0024] Sound is generated based on the refined spatial audio data.
[0025] According to some aspects, one or both of the two aforementioned systems may include one or more (e.g., all) of the following features (or any combination thereof). The wearable device may include a motion sensor configured to generate first posture metadata during a first time period, wherein a first posture estimate is generated based on the first posture metadata. The motion sensor may be further configured to generate second posture metadata during a second time period, wherein the processor generates a second posture estimate based on the second posture metadata. The system may further include: a companion device, the companion device including a camera, wherein the companion device is configured to: generate second posture metadata during the first time period, the second posture metadata including image data captured by the camera during the first time period; and generate the first posture estimate based on the first posture metadata and the second posture metadata. The motion sensor may be further configured to generate third posture metadata during the second time period, wherein the processor generates the second posture estimate based on the third posture metadata. In addition, the second time period may start immediately after the first time period.
[0026] According to one aspect, a system comprises:
[0027] A first device, the first device comprising:
[0028] A first processor, the first processor is configured to execute computer readable instructions, the computer readable instructions causing the first processor to perform the following operations when executed:
[0029] generating a first pose estimate; and
[0030] rendering spatial audio data based on the first pose estimate;
[0031] A second device, the second device comprising:
[0032] a second processor configured to execute computer-readable instructions that, when executed, cause the second processor to:
[0033] generating first refined spatial audio data based on the second pose estimate, wherein the first pose estimate and the second pose estimate each correspond to at least one pose of the second device; and
[0034] Sound is generated based on the first refined spatial audio data.
[0035] According to some aspects, the system may include one or more (e.g., all) of the following features (or any combination thereof). The system may further include: a motion sensor configured to generate first posture metadata during a first time period, wherein the first processor is configured to generate a first posture estimate based on the first posture metadata. The motion sensor may be further configured to generate second posture metadata during a second time period, wherein the second processor is configured to generate a second posture estimate based on the second posture metadata. The system may further include: a third device, the third device including: a third processor, the third processor configured to generate computer-readable instructions, which when executed cause the third processor to perform the following operations: generate second refined spatial audio data by refining the spatial audio data generated by the first processor based on a third posture estimate corresponding to the second device, wherein the first refined spatial audio data is generated by the second processor by refining the second refined spatial audio data. The motion sensor may be further configured to generate third posture metadata during a third time period, wherein the third processor is configured to generate a third posture estimate based on the third posture metadata. The third time period may occur between the first time period and the second time period. Additionally, the first device may be a wearable device, the second device may be a server, the third device may be a mobile device, and the wearable device may be communicatively coupled to the server and the mobile device.
[0036] According to one aspect, a wearable device comprises:
[0037] Speakers; and
[0038] A processor configured to execute computer readable instructions that, when executed, cause the processor to:
[0039] receiving a sound identifier, a spatial position, and a first pose estimate corresponding to a wearable device;
[0040] updating the first pose estimate to generate a second pose estimate;
[0041] rendering spatial audio data based on the sound identifier, the spatial position, and the second pose estimate; and
[0042] The speaker is caused to produce sound corresponding to the spatial audio data.
[0043] According to another aspect, a wearable device comprises:
[0044] Speakers; and
[0045] A processor configured to execute computer readable instructions that, when executed, cause the processor to:
[0046] receiving spatial audio data and a first pose estimate corresponding to a wearable device;
[0047] generating a second pose estimate corresponding to the wearable device;
[0048] refining the spatial audio data based on the second pose estimate; and
[0049] The speakers are caused to produce sound based on the refined spatial audio data.
[0050] According to some aspects, one or both of the two aforementioned systems can include one or more (eg, all) of the following features (or any combination thereof).
[0051] The wearable device may further include: a motion sensor configured to: generate first posture metadata during a first time period, wherein the first posture metadata indicates movement of the wearable device during the first time period, and wherein a first posture estimate is generated based on the first posture metadata; and generate second posture metadata during a second time period after the first time period, wherein the second posture metadata indicates movement of the wearable device during the second time period, and wherein a second posture estimate is generated based on the second posture metadata. The sound identifier may identify audio data stored at the wearable device, and wherein rendering the spatial audio data may include: spatializing the identified audio data based on the spatial position and the second posture estimate. The spatial audio data may cause the speaker to simulate the projection of the sound at the spatial position when generating the sound corresponding to the spatial audio data, wherein the posture indicated by the second posture estimate with respect to the wearable device defines the spatial position. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] The present disclosure may be better understood, and its several features and advantages may become apparent to those skilled in the art by referencing the accompanying drawings.The use of the same reference numerals in different drawings indicates similar or identical items.
[0053] Figure 1 is a block diagram of a distributed processing environment including a wearable device, a companion device, and a remote server according to some embodiments.
[0054] Figure 2 According to some embodiments Figure 1 Block diagram of a wearable device.
[0055] Figure 3 is a graph depicting changes in pose estimates over time while generating real head pose changes and corresponding pose metadata, according to some embodiments.
[0056] Figure 4is a graph depicting the difference between simulated spatial positions for sound reproduction with and without local refinement of corresponding spatial audio data and pose estimates, according to some embodiments.
[0057] Figure 5 is a flow chart of a method for performing pose estimation and spatial audio refinement at a wearable device after initial pose estimation and spatial audio rendering at a companion device in accordance with some embodiments.
[0058] Figure 6 is a flowchart of a method for performing pose estimation and spatial audio refinement at a wearable device after performing initial pose estimation and spatial audio rendering at a companion device according to some embodiments, wherein the initial pose estimation is based in part on pose metadata captured using a camera and / or other sensors of the companion device.
[0059] Figure 7 is a flow chart of a method for performing pose estimation and spatial audio refinement at a wearable device after initial pose estimation and spatial position determination at a companion device using audio data pre-stored on the wearable device according to some embodiments.
[0060] Figure 8 is a flow chart of a method of multi-stage pose estimation and spatial audio refinement at a wearable device and a companion device after initial pose estimation and spatial audio rendering at a remote server according to some embodiments. DETAILED DESCRIPTION
[0061] In this article, we combine Figures 1 to 8 The described techniques relate to mechanisms for reducing motion-to-audio output latency for wearable user devices employing spatial audio, particularly where rendering and transmission of the spatial audio by wireless or other means introduces significant latency (e.g., latency on the order of hundreds of milliseconds or more), since the accuracy of spatial audio reproduction will degrade rapidly between the time the spatial audio data is rendered and the time the spatial audio data is reproduced due to potential changes in user or device posture during the latency period. The spatial audio rendering techniques described herein may include binaural rendering techniques, such as stereo panning or three-dimensional (3D) sound synthesis techniques, which may be based on a head-related transfer function (HRTF) model.
[0062] Typically, in order to render spatial audio to simulate a sound source at a specific location with respect to a wearable device, the device rendering the spatial audio needs to estimate the posture (i.e., orientation in 3D space) of the user or wearable device, and the posture may be estimated based on posture metadata, which may include image data, motion data, or acceleration data indicating changes in the posture of the user or wearable device during a given time period, or one or more derivations thereof. In some embodiments, the posture metadata may also include timestamps and time synchronization information between the wearable device and one or more other devices (e.g., a mobile device and / or a remote server) that perform spatial audio rendering or refinement processes for the wearable device. For example, each entry of image data, motion data, audio data, or acceleration data may include a timestamp indicating the time when the associated data entry was sampled by the corresponding sensor of the device generating the posture metadata. The less delay between the capture of posture metadata, the rendering of spatial audio data, and the reproduction of spatial audio data improves the accuracy of the reproduction of spatial audio data. This is because the accurate reproduction of spatial audio depends on the accuracy of the posture used to render the spatial audio data with respect to the actual posture when the spatial audio is reproduced. For example, in some embodiments, the accuracy of the reproduction of spatial audio data depends on the perceived location at which the corresponding sound is reproduced with respect to the wearable device or user's posture when reproduced, compared to the expected location at which the corresponding sound is intended to be reproduced with respect to the wearable device or user's posture and / or with respect to a world space posture (e.g., cardinal directions or real-world landmarks).
[0063] To reduce latency and improve spatial audio reproduction accuracy, the pose estimate and spatial audio data are updated and refined by the wearable device, respectively, prior to spatial audio reproduction via the speakers of the wearable device. For example, an initial pose estimate and spatial audio data rendering may be performed at a companion device, which may be a smart phone or other mobile electronic device, based on pose metadata indicating the pose of the wearable device generated by the wearable device during a first time period. The wearable device may receive the initial pose estimate and spatial audio data from the companion device, and may then update the initial pose estimate based on additional pose metadata generated during a second time period following the first time period. The wearable device may then refine the spatial audio data based on the additional local sensor data and reproduce the refined spatial audio via the speakers of the wearable device.
[0064] For example, a "dead reckoning" algorithm may be used by the wearable device (or in some embodiments, a companion device) to update the initial pose estimate based on the additional pose metadata. In some embodiments, the dead reckoning algorithm is implemented using one or more Kalman filters.
[0065] The spatial audio data may be refined by the wearable device based on the updated pose estimate by computing a new local spatial audio transform based on the global spatial audio transform relative to the updated pose estimate. Equation 1 illustrates the global spatial audio transform , Wearable device posture change and local spatial audio transform a relationship between the global spatial audio transform corresponding to the location and orientation of a simulated audio source as defined by an application executed at the wearable device and / or a companion device, the wearable device pose transform corresponding to an updated pose estimate of the wearable device, and the local spatial audio transform being a transform applied to audio included in the spatial audio data when reproduced at the wearable device to simulate the audio as being output by an audio source at a specific spatial location.
[0066]
[0067] As shown in Equation 1, by transforming the wearable device posture into The inverse of the global spatial audio transform Generate local spatial audio transforms when refining spatial audio data , the local spatial audio transform is to be applied to the audio data to simulate the projection of the audio data from the sound at the defined location and orientation. The global spatial audio transform is included in the audio gesture metadata included in the spatial audio data, or the global spatial audio transform can be derived from the audio gesture metadata For example, when rendering spatial audio data, the companion device may determine the position and orientation in world space of an audio source to be simulated during reproduction of the spatial audio data, and may store the position and orientation as audio gesture metadata included in the spatial audio data. In some embodiments, the audio gesture metadata may include a timestamp corresponding to the time when the spatial audio data was rendered. Global Spatial Audio Transform Posture transformation with wearable devices can be expressed in a global coordinate reference frame (i.e., defined with respect to world space). Modifying the world-space audio transform , changing the coordinate reference frame from a global coordinate reference frame defined with respect to world space to a local coordinate reference frame defined with respect to the posture of the wearable device.
[0068] By utilizing posture metadata from a second time period to update the posture estimate and to refine the spatial audio data based on the updated posture estimate, perceived latency in reproduced spatial audio (e.g., with respect to position and rotation of the spatial audio) caused by a difference between an actual posture of the wearable device during spatial audio reproduction and an estimated posture of the wearable device used to render the spatial audio data is desirably reduced when compared to techniques in which posture estimation and spatial audio data rendering are performed solely at a companion device without subsequent modification.
[0069] Figure 1 A system 100 for distributed data processing, in particular, for distributed processing of spatial audio data to be output via speakers of a wearable device is shown. The system 100 includes an audio rendering device 102 (also referred to as a "companion device 102") coupled to a head mounted device 104 (also referred to as a "wearable device 104") via a wired connection or a wireless connection. One or both of the audio rendering device 102 and the wearable device 104 are communicatively coupled to a server 110 via a wired and / or wireless connection and via a gateway or router 106 and / or a network 108. The network 108 may be a local area network (LAN) or a wide area network (WAN), such as the Internet. In some embodiments, the audio rendering device 102 is communicatively connected to the wearable device 104 via a personal area network (PAN), a wireless local area network (WLAN), or the like. As an example used below, the audio rendering device 102 is a mobile device, such as a smart phone, and the wearable device 104 is a pair of smart glasses. In some embodiments, the wearable device 104 and / or the audio rendering device 102 may be disposed within the room 112 and thereby have inherent acoustic characteristics that may be detected by the wearable device 104 or the audio rendering device 102 and used to create an acoustic room model. When mixed with the spatial audio data, the acoustic room model modifies the spatial audio data to include non-ideal factors (e.g., echo, reverberation, attenuation, etc.) that would be introduced into the corresponding sound if produced by an audio source disposed at a given location within the room 112 rather than at the wearable device 104.
[0070] In some embodiments, the audio rendering device 102 generates a pose estimate and renders corresponding spatial audio data based on first pose metadata indicating a pose of the wearable device 104, and then sends the pose estimate and the spatial audio data to the wearable device 104. For example, the spatial audio data may include audio data to be reproduced by the wearable device 104 and may include audio pose metadata that defines the position and orientation (e.g., in world space) of an audio source to be simulated during the reproduction of the spatial audio data by the wearable device. The audio rendering device 102 may generate the audio pose metadata according to instructions received from an application executed at the audio rendering device 102 and / or the wearable device 104. The wearable device 104 then updates the pose estimate based on second pose metadata sampled after the first pose metadata, and then refines the spatial audio data based on the updated pose estimate. For example, the wearable device 104 may refine the spatial audio data by calculating a new local spatial audio transform, as previously described in conjunction with Equation 1. By updating the pose estimate and refining the spatial audio data at the wearable device 104, the latency between the pose estimate and the reproduction of the spatial audio data at the wearable device 104 is reduced.
[0071] Figure 2 An illustrative block diagram of a wearable device 104 is shown in accordance with some embodiments. Figure 2 In an example of the wearable device 104, the wearable device 104 includes a transceiver 202, a processor 204, a motion sensor 206, a memory 208, a speaker 210, a camera 212, and a display 214, some or all of which may be communicatively connected via a communication bus. In some embodiments, the motion sensor 206 is an inertial measurement unit (IMU), and is therefore sometimes referred to herein as "IMU 206". The motion sensor 206 generates gesture metadata, which may include motion data or acceleration data corresponding to the detected movement of the wearable device 104 and corresponding timestamps and / or time synchronization data between the wearable device 104 and the companion device 102 or the server 110 (depending on the embodiment). In some embodiments, the motion sensor 206 may detect movement of the wearable device 104 in three or six degrees of freedom. In some embodiments, the processor 204 generates gesture metadata based on the motion data or acceleration data generated by the motion sensor 206. In some embodiments, the camera 212 generates image data corresponding to the face of the wearer of the wearable device 104 or the environment in one or more directions around the wearer. Image data generated by camera 212 may be included in the gesture metadata, or used by processor 204 to derive at least a portion of the gesture metadata.
[0072] The transceiver 202 may include one or more transceiver circuits that are each configured to communicate according to a corresponding protocol, such as a wireless LAN protocol (e.g., Wi-Fi), a PAN protocol (e.g., Bluetooth, Zigbee), or a cellular communication protocol (e.g., 4G, 4G LTE, 5G). The transceiver 202 transmits the posture metadata to the companion device 102 or the server 110 directly or via one or more intermediate network devices for posture estimation generation and spatial audio rendering. The transceiver 202 then receives the posture estimation and spatial audio data from the companion device 102 or the server 110. In some embodiments, the motion sensor 206 and / or the processor 104 generates additional (“second”) posture metadata during a time period spanning from the time when the transceiver 202 outputs the initial (“first”) posture metadata to the time when the transceiver 202 receives the posture estimation and spatial audio data and stores the second posture metadata in the memory 208, such that the second posture metadata indicates the motion of the wearable device 104 during the time period. The processor 204 updates the posture estimation and refines the spatial audio data based on the second posture metadata. When refining the spatial audio data, the processor 204 may change the left and right speaker intensities of the spatial audio data based on the change in the posture of the wearable device 104 during the time period corresponding to the second posture metadata (e.g., the respective volumes of the sound produced by the left and right speakers of the speaker 210 of the wearable device 104, which the processor 204 may modify by controlling the amplitude of the audio signal provided to the left and right speakers or controlling the gain applied thereto), but for two-channel audio, a more complex algorithm may be required to correctly modify the sound in all 6 degrees of freedom that takes into account the movement of the user. For example, the wearable device 104 may refine the spatial audio data by calculating a new local spatial audio transform, as previously described in conjunction with Equation 1. By refining the spatial audio data based on the updated posture estimate generated based on the second posture metadata, the spatial audio data to be reproduced via the speaker 210 is refined to account for the movement of the wearable device 104 that occurred when the posture estimate and the spatial audio data were generated by the companion device 102 and / or the server 110, thereby reducing latency in the spatial audio data and improving the accuracy of the spatial audio reproduction.
[0073] In some embodiments, the processor 204 mixes the spatial audio data with ambient or environmental audio characteristics that the processor may generate based on an acoustic room model. The acoustic room model simulates the acoustic characteristics of a room, such as room 112, in which the wearable device 104 and / or the companion device 102 is disposed. For example, the processor 204 may generate the acoustic room model based on image data captured by the camera 212. In some embodiments, instead, the acoustic room model is generated by the companion device 102 or server 110 based on the image data captured by the camera 212 and then provided to the wearable device 104 to be mixed with the spatial audio data. In some embodiments, the acoustic room model may be retrieved from a local or remote database of pre-generated room or environment models included in or communicatively coupled to the companion device 102 or server 110.
[0074] Once the processor 204 updates the pose estimate and refines the spatial audio data, the processor 204 causes the speaker 210 configured for spatial sound reproduction to output corresponding sounds. The speaker 210 simulates the sounds of the spatial audio data to be perceived as originating from a specific spatial location with respect to the estimated pose of the wearable device 104, as defined in the spatial audio data.
[0075] Return to view Figure 1 In some embodiments, the wearable device 104 sends the generated gesture metadata to the audio rendering device 102 in response to a determination made by an application executing at the wearable device 104 or the audio rendering device 102 that the spatial audio data should be rendered and reproduced. In some embodiments, the wearable device 104 sends instructions to the audio rendering device 102 indicating which audio data should be rendered. In some embodiments, the audio rendering device 102 renders the spatial audio in conjunction with the AR / VR visual content for transmission to and display at the wearable device 104.
[0076] Upon receiving the pose metadata from the wearable device 104, the audio rendering device 102 estimates the pose (i.e., location and / or orientation) of the wearable device 104, and renders spatial audio data based on the estimated pose, and transmits the rendered spatial audio data to the wearable device. For example, the audio rendering device 102 can render the location sound in several ways from simple panning (left and right channel intensity) to two-channel audio.
[0077] In some embodiments, the audio data rendered as spatial audio data by the audio rendering device 102 is stored on a local memory device of the audio rendering device 102, while in other embodiments, the audio rendering device 102 receives (e.g., streams) the audio data from the server 110. In some embodiments, the audio rendering device 102 communicates the pose metadata from the wearable device 104 to the server 110, and the server 110 renders the spatial audio data and generates an initial pose estimate, and then sends the initial pose estimate and the spatial audio data to the wearable device 104 via the audio rendering device 102. In some embodiments, the wearable device 104 sends the pose metadata directly to the server 110 via the gateway or router 106 and / or via the network 108 without an intermediary audio rendering device 102, and the server 110 renders the spatial audio data, generates an initial pose estimate, and sends both to the wearable device 104 via the gateway or router 106 and / or the network 108.
[0078] In any of the above embodiments, the delay in gesture detection and corresponding gesture metadata generation for rendering spatial audio data to spatial audio reproduction by speakers of wearable device 104 can result in inaccurate audio data being reproduced at inaccurate spatial locations, but this inaccuracy can be reduced by refining the spatial audio data and gesture estimation at wearable device 104 using additional gesture metadata generated after wearable device 104 sends initial gesture metadata to audio rendering device 102 or server 110. Due to this improvement in spatial audio reproduction accuracy, the delay perceived by the user in spatial audio reproduction is advantageously reduced. For example, because the user can turn his head or otherwise change the posture of the wearable device 104 after the wearable device 104 sends the initial gesture metadata to the audio rendering device 102 or server 110 and before the spatial audio is output by the wearable device 104, the estimated posture of the wearable device 104 as represented in the spatial audio can be incorrectly aligned with the actual posture of the wearable device 104 in spatial audio reproduction. Thus, in various embodiments, the system 100 provides improved spatial audio reproduction in a head mounted device by using pose estimation logic by the wearable device 104 to refine pose estimates and spatial audio data generated by the companion device 102 or server 110. For example, while feature extraction is typically performed on the companion device 102 based on image data captured by the camera 212 of the wearable device 104, the wearable device 104 may locally perform “dead reckoning” using motion or acceleration data generated by the IMU 206 to generate a more accurate pose estimate, as described above.
[0079] To illustrate, Figure 3A diagram 300 is shown depicting how, in some embodiments, the wearable device 104 can avoid pose estimation inaccuracies introduced by a delay between the time when the wearable device 104 transmits the first pose metadata to the companion device 102 and the time when the pose estimate is provided from the companion device 102 to the wearable device 104 by updating the pose estimate based on second pose metadata acquired after the first pose metadata is sent to the companion device 102. It should be understood that Figure 3 The examples of also apply to embodiments in which the initial pose estimate is generated by server 110 rather than companion device 102 .
[0080] As shown, the chart 300 includes a plurality of rows 302, 304, 306, and 308. Row 308 shows posture metadata generated by the motion sensor 206 of the wearable device 104 over time. Row 306 shows the actual posture of the wearable device 104 at discrete times T1 and T2. Row 304 shows a posture estimate generated by the companion device 102 at time T1 based on the first posture metadata 310. Row 302 shows an updated posture estimate generated by the wearable device 104 based on the posture estimate generated by the companion device 102 and further based on the second posture metadata 312.
[0081] In this example, the motion sensor 206 of the wearable device 104 generates first posture metadata 310 during a time period from time T0 to time T1 ("first time period"). It should be understood that the first posture metadata includes at least two samples of motion, acceleration, or image data obtained during the first time period, rather than simply reflecting an initial set of conditions. At time T1, the wearable device 104 transmits the first posture metadata 310 to the companion device 102, and the companion device 102 then generates a posture estimate based on the first posture metadata 310. As shown in rows 304 and 306, the actual posture at time T1 matches the posture estimate generated by the companion device 102. It should be noted that there may be some marginal delay between transmitting the first posture metadata 310 to the companion device 102 and generating the posture estimate by the companion device 102 to account for processing time.
[0082] After transmitting the first posture metadata 310 to the companion device 102, the wearable device 104 continues to generate posture metadata, specifically second posture metadata 312, using the motion sensor 206 from time T1 to time T2. The period between time T1 and time T2 (the "second period") corresponds to the time that has passed since the first posture metadata 310 was sent from the wearable device 104 to the companion device 102 and the companion device 102 sent the posture estimate and corresponding spatial audio data to the wearable device 104. In this example, from time T1 to time T2, the posture of the wearable device 104 changes (e.g., due to the wearer turning or moving his head) by approximately 45 degrees. This change in posture is not indicated in the first posture metadata 310 or the corresponding posture estimate generated by the companion device 102. After receiving the posture estimate from the companion device 102 at time T2, the wearable device 104 generates an updated posture estimate based on the second posture metadata 312, which takes into account the change in the actual posture of the wearable device 104 that occurred from time T1 to time T2. The wearable device 104 then refines the spatial audio data based on the updated pose estimate, as shown in row 302, to match the real pose shown in row 306 at or around time T2. In some embodiments, the wearable device 104 updates the pose estimate by integrating the second pose metadata for the second time period to determine a net change in pose during the second time period, and then adding the net change in pose to the initial pose estimate generated by the companion device 102. In this way, the accuracy of reproducing the spatial audio (particularly with respect to the spatial location of the simulated audio) is improved compared to a scene where the spatial audio data is rendered using only the initial pose estimate and the first metadata 310.
[0083] Figure 4 The effect of latency on the perceived location of a sound is illustrated. Two top-down views 400 and 410 illustrate a situation where a user wearing the wearable device 104 turns his head to the right between the time when the first pose metadata is generated at time T1 and the time when the wearable device 104 receives the first pose estimate and spatial audio data from the companion device 102 or server 110 at time T2.
[0084] The diagram 400 shows a real posture 406 of the wearable device 104 at the end of a first time period from time T0 to time T1, during which the motion sensor 206 of the wearable device 104 generates first posture metadata (eg, Figure 3 Marker 404 shows a location along perimeter 402 where initial spatial audio data rendered by companion device 102 or server 110 will simulate the gesture of wearable device 104 when reproduced via speaker 210 of wearable device 104.
[0085] View 400 shows the actual posture 416 of the wearable device 104 at the end of a second time period from time T1 to time T2, during which the motion sensor 206 of the wearable device 104 generates second posture metadata (eg, Figure 3 The second posture metadata 312 is provided for the companion device 102 or server 110 to generate the first posture estimate and render the spatial audio data. The second time period typically spans the time taken for the companion device 102 or server 110 to generate the first posture estimate and render the spatial audio data. The real posture 416 is rotated about the real posture 406, indicating that the user turned his head or body during the second time period. If the spatial audio data remains unchanged, then the audio reproduced by the wearable device 104 based on the spatial audio data will be simulated to the right of the user at the location of the marker 418, because the spatial location used to simulate the spatial audio data is defined with respect to the posture of the wearable device 104 and will shift as the posture shifts. This will undesirably provide the user with a perception that the audio source is "floating" or "floating" without having a fixed spatial location. In some cases, this effect is perceived by the user as if the audio source moves with the user or follows the user's movements with a certain delay, rather than being under a fixed spatial projection. Instead, by updating the pose estimate provided by companion device 102 or server 110 based on the second pose metadata, and then refining the spatial audio data based on the updated pose estimate, wearable device 104 reproduces the spatial audio data at the correct spatial location indicated by marker 414, which corresponds to the location of marker 404 in view 400. By refining the spatial audio data in this manner, the reproduced audio provides the user with the perception that the audio source is located at a fixed spatial location that does not change as the user's head or body moves.
[0086] Figure 5 An illustrative process flow of a method 500 for rendering and reproducing spatial audio data is shown. In method 500, a pose estimate of the wearable device and spatial audio data generated at a companion device based on first pose metadata are updated by the wearable device based on second pose metadata, the second pose metadata being generated while the companion device is generating the pose estimate and the spatial audio data. In this example, method 500 is performed by Figure 1 However, in some embodiments, other applicable devices (such as server 110) may perform some or all of the functions attributed to companion device 102 in this example. Figure 2 The figure describes the functions of the wearable device 104.
[0087] In block 502, the wearable device 104 generates first gesture metadata. In some embodiments, the first gesture metadata includes motion data or acceleration data generated by a motion sensor 206 (which may be an IMU) of the wearable device 104. In some embodiments, in addition to or in lieu of the motion data or acceleration data generated by the motion sensor 206, the wearable device 104 may generate the first gesture metadata based on image data captured by a camera 212 of the wearable device 104. The wearable device 104 samples the first gesture metadata during a first time period that ends immediately before the wearable device 104 sends the first gesture metadata to the companion device 102.
[0088] In block 504, the wearable device 104 sends the first gesture metadata to the companion device 102. For example, the wearable device 104 may wirelessly transmit the first gesture metadata to the companion device 102 using the transceiver 202.
[0089] In block 514, companion device 102 generates a posture estimate based on the first posture metadata received from wearable device 104. For example, companion device 102 may analyze motion data, acceleration data, and / or image data included in the first posture metadata to determine, in three or six degrees of freedom, how the posture of wearable device 104 changed with respect to an initial posture during a first time period in which the first posture metadata was sampled.
[0090] In box 516, the companion device 102 renders the spatial audio data based on the posture estimation. In some embodiments, the companion device 102 retrieves the audio data based on an instruction received from the wearable device 104, which may be generated by a software application executed by the wearable device 104. For example, the software application may be a navigation application, a VR application, an AR application, etc. For example, the software application may be an AR or VR safety system that warns a user of a potential collision with a wall, an object, or a person by playing a sound from the direction and location of a detected danger (i.e., a potential collision). As another example, the software application may be a real-time audio translation application that plays back a synthesized translated speech based on the posture of a real person who is talking to a user wearing the wearable device. As another example, the software application may be configured to make the sound appear to be emitted from a speaker of a physical device (such as a TV, monitor, tablet computer, phone, etc.) without actually causing the sound to be emitted from the speaker of the physical device, but by virtually playing the sound at the speaker of the wearable device. As another example, the software application may be configured to emit a virtual sound simulated from the location of an IoT device (such as an appliance, smart light bulb, etc.) in response to a change in state of the IoT device or in response to a user moving within a defined proximity of the IoT device.
[0091] The companion device 102 may retrieve the audio data from a local memory of the companion device 102 or from a remote memory, such as a remote memory of the server 110. After obtaining the audio data, the companion device 102 spatializes the audio data based on the pose estimation to render into spatial audio data. For example, the companion device 102 may determine a spatial location to simulate the audio data based on instructions received from the wearable device 104 and based on the pose estimation. In some embodiments, the companion device 102 may render the spatial audio data using a head-related transfer function (HRFT) model for simulated reproduction at a specific spatial location based on the pose estimation.
[0092] In block 518 , the companion device 102 sends the spatial audio data and the pose estimate to the wearable device 104 .
[0093] In block 506, in parallel with blocks 514, 516, and 518, wearable device 104 generates second gesture metadata, the second gesture metadata comprising motion data or acceleration data generated by motion sensor 206 and / or image data generated by camera 212. The second gesture metadata is sampled during a second time period immediately following the first time period. In some embodiments, the second time period begins when wearable device 104 sends the first gesture metadata to companion device 102 and ends when wearable device 104 receives spatial audio data and gesture estimates from companion device 102.
[0094] In block 508, after blocks 506 and 518, the wearable device 104 generates an updated posture estimate based on the second posture metadata. For example, the wearable device 104 may analyze the motion data, acceleration data, and / or image data included in the second posture metadata to determine how the posture of the wearable device 104 changes during the second time period in which the second posture metadata is sampled in three or six degrees of freedom. The wearable device 104 may then update the posture estimate based on how the posture changes during the second time period to produce an updated posture estimate.
[0095] In block 510, the wearable device 104 generates refined spatial audio data based on the spatial audio data received from the companion device 102 and based on one or both of the pose estimate received from the companion device 102 and the updated pose estimate. In some embodiments, the wearable device 104 refines the spatial audio data by adjusting the spatial location (e.g., with respect to the pose of the wearable device, the user's pose, and / or the pose in world space) at which the simulated audio is projected based on the updated pose estimate. In some embodiments, the wearable device 104 refines the spatial audio data by adjusting the spatial location based on a difference between the pose estimate and the updated pose estimate. In some embodiments, the wearable device 104 refines the spatial audio data based on the updated pose estimate and audio pose metadata about the world space pose (e.g., cardinal directions or real-world landmarks), as previously described in conjunction with Equation 1.
[0096] In block 512, the wearable device 104 outputs audio via the speaker 110 based on the refined spatial audio data. In some embodiments, the wearable device 104 may delay the output of the audio until the user is in a particular predefined orientation. In some embodiments, the wearable device 104 may delay the output of the audio until the posture remains substantially unchanged for longer than a predefined amount of time. In some embodiments, the wearable device 104 may attenuate the audio output by the speaker 110 (e.g., by reducing the volume of the audio to zero over a predefined period of time) in response to determining that the user's posture has changed by more than a predetermined threshold amount during audio playback to provide feedback to the user that the spatial audio output by the wearable device 104 may no longer be appropriate for the user's current posture.
[0097] Figure 6 An illustrative process flow of a method 600 for rendering and reproducing spatial audio data is shown. In method 600, a pose estimate of a wearable device and spatial audio data generated at a companion device based on first pose metadata generated by a motion sensor of the wearable device and based on second pose metadata generated by a camera of the companion device are updated by the wearable device based on third pose metadata that is generated while the companion device is generating the pose estimate and the spatial audio data. In this example, method 600 is performed by Figure 1 However, in some embodiments, other applicable devices (such as server 110) may perform at least some of the functions attributed to companion device 102 in this example. Figure 2 The figure describes the functions of the wearable device 104.
[0098] In block 602, the wearable device 104 generates first gesture metadata. In some embodiments, the first gesture metadata includes motion data or acceleration data generated by a motion sensor 206 (which may be an IMU) of the wearable device 104. In some embodiments, in addition to or in lieu of the motion data or acceleration data generated by the motion sensor 206, the wearable device 104 may generate the first gesture metadata based on image data captured by a camera 212 of the wearable device 104. The wearable device 104 samples the first gesture metadata during a first time period that ends immediately before the wearable device 104 sends the first gesture metadata to the companion device 102.
[0099] In block 614, in parallel with block 602, companion device 102 generates second gesture metadata using a camera of companion device 102. For example, the second gesture metadata may include image data captured by the camera, wherein the image data includes an image from wearable device 104. Companion device 102 samples the image data corresponding to the second gesture metadata during a first time period. In some embodiments, one or more other sensors may be used to generate the second gesture metadata instead of or in addition to the camera of companion device 102. For example, such sensors may include an ultra-wideband chip sensor, an infrared LED sensor, a tracking marker sensor, and / or a three-dimensional laser tracking system.
[0100] In block 604, the wearable device 104 sends the first gesture metadata to the companion device 102. For example, the wearable device 104 may wirelessly transmit the first gesture metadata to the companion device 102 using the transceiver 202.
[0101] In block 616, following blocks 604 and 614, companion device 102 generates a pose estimate based on the first pose metadata and the second pose metadata received from wearable device 104. For example, companion device 102 may analyze motion data, acceleration data, and / or image data included in the first pose metadata and image data included in the second pose metadata to determine, in three or six degrees of freedom, how the pose of wearable device 104 changed with respect to an initial pose during a first time period in which the first pose metadata and the second pose metadata were sampled.
[0102] In block 618, the companion device 102 renders the spatial audio data based on the pose estimation. In some embodiments, the companion device 102 retrieves the audio data based on instructions received from the wearable device 104, which may be generated by a software application executed by the wearable device 104. The companion device 102 may retrieve the audio data from a local memory of the companion device 102 or from a remote memory, such as a remote memory of the server 110. After obtaining the audio data, the companion device 102 spatializes the audio data based on the pose estimation to render into the spatial audio data. For example, the companion device 102 may determine a spatial location at which to simulate the audio data based on the instructions received from the wearable device 104 and based on the pose estimation. In some embodiments, the companion device 102 may render the spatial audio data using a HRFT model for simulated reproduction at a particular spatial location based on the pose estimation.
[0103] In block 620 , the companion device 102 sends the spatial audio data and the pose estimate to the wearable device 104 .
[0104] In block 606, in parallel with blocks 616, 618, and 620, wearable device 104 generates third gesture metadata, the third gesture metadata comprising motion data or acceleration data generated by motion sensor 206 and / or image data generated by camera 212. The third gesture metadata is sampled during a second time period immediately following the first time period. In some embodiments, the second time period begins when wearable device 104 sends the first gesture metadata to companion device 102 and ends when wearable device 104 receives spatial audio data and gesture estimates from companion device 102.
[0105] In block 608, after blocks 606 and 620, the wearable device 104 generates an updated posture estimate based on the third posture metadata. For example, the wearable device 104 may analyze the motion data, acceleration data, and / or image data included in the third posture metadata to determine how the posture of the wearable device 104 changed during the second time period in which the third posture metadata was sampled in three or six degrees of freedom. The wearable device 104 may then update the posture estimate based on how the posture changed during the second time period to produce an updated posture estimate.
[0106] In block 610, the wearable device 104 generates refined spatial audio data based on the spatial audio data received from the companion device 102 and one or both of the pose estimate and the updated pose estimate received from the companion device 102. In some embodiments, the wearable device 104 refines the spatial audio data by adjusting the spatial location at which the simulated audio is projected based on the updated pose estimate. In some embodiments, the wearable device 104 refines the spatial audio data by adjusting the spatial location based on a difference between the pose estimate and the updated pose estimate. In some embodiments, the wearable device 104 refines the spatial audio data based on the updated pose estimate and audio pose metadata about a world space pose (e.g., cardinal directions or real-world landmarks), as previously described in conjunction with Equation 1.
[0107] In block 612, the wearable device 104 outputs audio via the speaker 110 based on the refined spatial audio data. In some embodiments, the wearable device 104 may delay output of the audio until the user is in a particular predefined orientation. In some embodiments, the wearable device 104 may delay output of the audio until the posture remains substantially unchanged for longer than a predefined amount of time. In some embodiments, the wearable device 104 may attenuate the audio output by the speaker 110 (e.g., by reducing the volume of the audio to zero over a predefined period of time) in response to determining that the user's posture has changed by more than a predetermined threshold amount during audio playback to provide feedback to the user that the spatial audio output by the wearable device 104 may no longer be appropriate for the user's current posture.
[0108] Figure 7 An illustrative process flow of a method 700 for rendering and reproducing spatial audio data is shown. In method 700, a pose estimate of a wearable device generated at a companion device based on first pose metadata generated by a motion sensor of the wearable device is updated by the wearable device based on second pose metadata that was generated while the companion device was generating the pose estimate. Rather than rendering the spatial audio data using the companion device, the wearable device stores preloaded sounds identified by the companion device for reproduction. In some embodiments, the preloaded sounds are provided to the wearable device by the companion device prior to performing method 700. By using the preloaded audio at the local wearable device, rather than rendering the audio data at the remote device, latency is advantageously reduced.
[0109] In this example, method 700 consists of Figure 1 However, in some embodiments, other applicable devices (such as server 110) may perform at least some of the functions attributed to companion device 102 in this example. Figure 2The figure describes the functions of the wearable device 104.
[0110] In block 702, the wearable device 104 generates first gesture metadata. In some embodiments, the first gesture metadata includes motion data or acceleration data generated by a motion sensor 206 (which may be an IMU) of the wearable device 104. In some embodiments, in addition to or in lieu of the motion data or acceleration data generated by the motion sensor 206, the wearable device 104 may generate the first gesture metadata based on image data captured by a camera 212 of the wearable device 104. The wearable device 104 samples the first gesture metadata during a first time period that ends immediately before the wearable device 104 sends the first gesture metadata to the companion device 102.
[0111] In block 704, the wearable device 104 sends the first gesture metadata to the companion device 102. For example, the wearable device 104 may wirelessly transmit the first gesture metadata to the companion device 102 using the transceiver 202.
[0112] In block 714, the companion device 102 generates a pose estimate based on the first pose metadata received from the wearable device 104, determines a sound identifier that identifies a sound to be reproduced at the wearable device 104, and determines a spatial location of the reproduced sound to be simulated. For example, the companion device 102 may analyze the motion data, acceleration data, and / or image data included in the first pose metadata to determine, in three or six degrees of freedom, how the pose of the wearable device 104 changes with respect to an initial pose during a first time period in which the first pose metadata is sampled.
[0113] The various preloaded sounds may be stored at the memory 208 of the wearable device 104. Based on instructions received from the wearable device 104 or originating from the companion device 102 (which may be generated by the wearable device 104 or a software application executed by the companion device 102, respectively), the companion device 102 determines which preloaded sounds should be output by the wearable device 104 and selects corresponding sound identifiers.
[0114] The instructions generated by the software application may also indicate the posture-related locations where the preloaded sounds should be reproduced by the wearable device 104. In generating the posture estimate, the companion device 102 determines the spatial location based on the posture-related locations indicated in the instructions and based on the posture estimate.
[0115] In box 716, the companion device 102 sends the pose estimate, sound identifier, and spatial location to the wearable device 104.
[0116] In block 706, in parallel with blocks 714 and 716, wearable device 104 generates second gesture metadata, the second gesture metadata comprising motion data or acceleration data generated by motion sensor 206 and / or image data generated by camera 212. The second gesture metadata is sampled during a second time period immediately following the first time period. In some embodiments, the second time period begins when wearable device 104 sends the first gesture metadata to companion device 102 and ends when wearable device 104 receives spatial audio data and a gesture estimate from companion device 102.
[0117] In block 708, after blocks 706 and 716, the wearable device 104 generates an updated posture estimate based on the second posture metadata. For example, the wearable device 104 may analyze the motion data, acceleration data, and / or image data included in the second posture metadata to determine how the posture of the wearable device 104 changes during the second time period in which the second posture metadata is sampled in three or six degrees of freedom. The wearable device 104 may then update the posture estimate based on how the posture changes during the second time period to produce an updated posture estimate.
[0118] In block 710, the wearable device 104 generates spatial audio data based on the sound identifier and the spatial position received from the companion device 102 and based on the updated pose estimate. For example, the wearable device 104 may retrieve audio data from the memory 208 based on the sound identifier. The wearable device 104 may then spatialize the retrieved audio data based on the spatial position and the updated pose estimate (e.g., using a binaural rendering technique) to generate the spatial audio data.
[0119] In box 712, the wearable device 104 outputs audio via the speaker 110 based on the spatial audio data. In some embodiments, the wearable device 104 can delay the output of the audio until the user is in a specific predefined orientation or until another predefined condition is satisfied. In some embodiments, the wearable device 104 can delay the output of the audio until the posture remains substantially unchanged for longer than a predefined amount of time. In some embodiments, the wearable device 104 can attenuate the audio output by the speaker 110 (e.g., by reducing the volume of the audio to zero for a predefined period of time) in response to determining that the user's posture has changed by more than a predetermined threshold amount during audio playback to provide feedback to the user that the spatial audio output by the wearable device 104 may no longer be applicable to the user's current posture.
[0120] Figure 8 An illustrative process flow of a method 800 for rendering and reproducing spatial audio data using multi-stage pose estimation and spatial audio refinement is shown. In this example, the method 800 consists of Figure 1The wearable device 104, the companion device 102 and the server 110 are performed. In this example, sometimes Figure 2 The figure describes the functions of the wearable device 104.
[0121] In block 802, the wearable device 104 generates first gesture metadata. In some embodiments, the first gesture metadata includes motion data or acceleration data generated by a motion sensor 206 (which may be an IMU) of the wearable device 104. In some embodiments, in addition to or in lieu of the motion data or acceleration data generated by the motion sensor 206, the wearable device 104 may generate the first gesture metadata based on image data captured by a camera 212 of the wearable device 104. The wearable device 104 samples the first gesture metadata during a first time period that ends immediately before the wearable device 104 sends the first gesture metadata to the server 110.
[0122] In block 804, the wearable device 104 sends the first gesture metadata to the server 110. For example, the wearable device 104 may use the transceiver 202 to wirelessly transmit the first gesture metadata to the companion device 102, which may then send the first gesture metadata to the server 110 via the gateway or router 106, the network 108, the cellular network, or some combination thereof. Alternatively, the wearable device 104 may use the transceiver 202 to send the first gesture metadata to the server 110 via the gateway or router 106, the network 108, the cellular network, or some combination thereof without including the companion device 102 in the communication path.
[0123] In block 822, the server 110 generates a first pose estimate based on the first pose metadata received from the wearable device 104. For example, the server 110 may analyze the motion data, acceleration data, and / or image data included in the first pose metadata to determine, in three or six degrees of freedom, how the pose of the wearable device 104 changes with respect to the initial pose during the period in which the first pose metadata is sampled.
[0124] In box 824, the server 110 renders the spatial audio data based on the first posture estimate. In some embodiments, the server 110 retrieves the audio data based on an instruction received from the wearable device 104, which can be generated by a software application executed by the wearable device 104. For example, the server 110 can retrieve the audio data from a local memory. After obtaining the audio data, the server 110 spatializes the audio data based on the first posture estimate to render it into the spatial audio data. For example, the server 110 can determine the spatial location of the simulated audio data based on the instruction received from the wearable device 104 and based on the first posture estimate. In some embodiments, the server 110 can use the HRFT model to render the spatial audio data for simulated reproduction at a specific spatial location based on the first posture estimate.
[0125] In block 826, the server 110 sends the spatial audio data and the first pose estimate to the companion device 102. According to various embodiments, the server 110 may send the spatial audio data and the first pose estimate.
[0126] In block 806, in parallel with blocks 822, 824, and 826, wearable device 104 generates second gesture metadata, the second gesture metadata comprising motion data or acceleration data generated by motion sensor 206 and / or image data generated by camera 212, wherein the second gesture metadata is sampled during a second time period immediately following the first time period. In some embodiments, the second time period begins at the time wearable device 104 sends the first gesture metadata to server 110 and ends at the time companion device 102 receives the spatial audio data and the first gesture estimate from server 110.
[0127] In block 816, following blocks 806 and 826, companion device 102 generates a second pose estimate based on the second pose metadata received from wearable device 104. For example, companion device 102 may analyze motion data, acceleration data, and / or image data included in the second pose metadata to determine how the pose of wearable device 104 changed in three or six degrees of freedom during the second time period in which the second pose metadata was sampled. Wearable device 104 may then update the first pose estimate based on how the pose changed during the second time period to produce a second pose estimate.
[0128] In block 818, the companion device 102 generates first refined spatial audio data based on the spatial audio data received from the server 110 and one or both of the first pose estimate and the second pose estimate received from the server 110. In some embodiments, the companion device 102 refines the spatial audio data by adjusting the spatial position at which the simulated audio is projected based on the second pose estimate. In some embodiments, the companion device 102 refines the spatial audio data by adjusting the spatial position based on a difference between the first pose estimate and the second pose estimate. In some embodiments, the companion device 102 refines the spatial audio data based on the second pose estimate and audio pose metadata about a world space pose (e.g., cardinal directions or real-world landmarks), as previously described in conjunction with Equation 1.
[0129] In block 820 , the companion device 102 sends the first refined spatial audio data and the second pose estimate to the wearable device 104 .
[0130] In block 808, in parallel with blocks 816, 818, and 820, wearable device 104 generates third pose metadata, the third pose metadata comprising motion data or acceleration data generated by motion sensor 206 and / or image data generated by camera 212, wherein the third pose metadata is sampled during a third time period immediately following the second time period. In some embodiments, the third time period begins when wearable device 104 sends the second pose metadata to companion device 102 and ends when wearable device 104 receives the first refined spatial audio data and the second pose estimate from companion device 102.
[0131] In block 810, after blocks 808 and 820, the wearable device 104 generates a third pose estimate based on the third pose metadata. For example, the wearable device 104 may analyze the motion data, acceleration data, and / or image data included in the third pose metadata to determine how the pose of the wearable device 104 changes during a third time period in which the third pose metadata is sampled in three or six degrees of freedom. The wearable device 104 may then update the second pose estimate based on how the pose changes during the third time period to produce a third pose estimate.
[0132] In block 812, the wearable device 104 generates second refined spatial audio data based on the first refined spatial audio data received from the companion device 102 and one or both of the second pose estimate and the third pose estimate received from the companion device 102. In some embodiments, the wearable device 104 refines the first refined spatial audio data by adjusting the spatial position at which the simulated audio is projected based on the third pose estimate. In some embodiments, the wearable device 104 refines the first refined spatial audio data by adjusting the spatial position based on a difference between the second pose estimate and the third pose estimate. In some embodiments, the wearable device 104 further refines the first refined spatial audio data based on the third pose estimate and audio pose metadata about a world space pose (e.g., cardinal directions or real-world landmarks), as previously described in conjunction with Equation 1.
[0133] In block 814, the wearable device 104 outputs audio via the speaker 110 based on the second refined spatial audio data. In some embodiments, the wearable device 104 may delay output of the audio until the user is in a particular predefined orientation. In some embodiments, the wearable device 104 may delay output of the audio until the posture remains substantially unchanged for longer than a predefined amount of time. In some embodiments, the wearable device 104 may attenuate the audio output by the speaker 110 (e.g., by reducing the volume of the audio to zero over a predefined period of time) in response to determining that the user's posture has changed by more than a predetermined threshold amount during audio playback to provide feedback to the user that the spatial audio output by the wearable device 104 may no longer be appropriate for the user's current posture.
[0134] In some embodiments, method 800 may be selectively performed by wearable device 104, companion device 102, and server 110 based on a determined processing workload of companion device 102 or device 104. For example, method 800 may be performed in response to wearable device 104 or companion device 102 determining that the processing workload of companion device 102 exceeds a predetermined threshold (e.g., 80% of maximum processor utilization). In some such embodiments, if the processing workload is not exceeded, another spatial audio rendering method may be performed instead, such as Figures 5 to 7 As another example, method 800 may be performed in response to the wearable device 104 or the companion device 102 determining that the network latency between the wearable device 104 and the server 110 is less than the network latency between the wearable device 104 and the companion device 102. However, if the network latency between the wearable device 104 and the server 110 is higher, another spatial audio rendering method may be performed instead, such as Figures 5 to 7 One of the methods.
[0135] exist Figures 1 to 8 In any of the examples of , acoustic modeling may be applied to refine the spatial audio data to improve the realism of the sound in the mixed reality environment and take into account the correct physical acoustic properties (reverberation, etc.) of the user's room or environment. For example, the companion device 102 and the wearable device 104 may create an acoustic model of the physical room in which the wearable device is located, including the location of walls and objects and materials that may affect the sound quality. In some embodiments, the acoustic model may be generated based at least in part on 3D modeling of the environment obtained from an imaging camera on the companion device 102 and / or the wearable device 104. The acoustic room model is then stored at one or both of the companion device 102 and the wearable device 104 for subsequent mixing with the refined spatial audio data. In some embodiments, the acoustic model has a location and orientation calibrated to the real world, and the wearable device 104 adjusts the acoustic model with respect to the final pose estimate immediately prior to audio playback.
[0136] In some embodiments, certain aspects of the techniques described above may be implemented by one or more processors in a processing system that executes software. Software includes one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. Software may include instructions and specific data that manipulate one or more processors to perform one or more aspects of the techniques described above when executed by one or more processors. Non-transitory computer-readable storage media may include, for example, magnetic disk storage devices or optical disk storage devices, solid-state storage devices (such as flash memory), caches, random access memories (RAM), or one or more other non-volatile memory devices, etc. Executable instructions stored on a non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats that may be interpreted or otherwise executed by one or more processors.
[0137] Computer-readable storage media may include any storage media or combination of storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. These storage media may include, but are not limited to, optical media (e.g., compact disks (CDs), digital versatile disks (DVDs), Blu-ray disks), magnetic media (e.g., floppy disks, tapes, or magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or micro-electromechanical system (MEMS)-based storage media. Computer-readable storage media may be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard drive), removably attached to a computing system (e.g., an optical disk or flash memory based on a universal serial bus (USB)), or coupled to a computer system via a wired network or a wireless network (e.g., a network accessible storage device (NAS)).
[0138] It should be noted that not all activities or elements described above in the general description are required, part of a specific activity or device may not be required, and one or more other activities may be performed, or elements other than those described may be included. Further, the order in which the activities are listed is not necessarily the order in which they are performed. In addition, concepts have been described with reference to specific embodiments. However, those of ordinary skill in the art will recognize that various modifications and changes may be made without departing from the scope of the present disclosure as set forth in the appended claims below. Therefore, the specification and drawings should be regarded as illustrative, rather than restrictive, and all such modifications are intended to be included within the scope of the present disclosure.
[0139] The benefits, other advantages, and solutions to the problems have been described above for specific embodiments. However, the benefits, advantages, solutions to the problems, and any features that can make any benefit, advantage, or solution appear or become more obvious should not be interpreted as key features, required features, or essential features of any or all claims. Moreover, the specific embodiments disclosed above are merely illustrative, since it is obvious to those skilled in the art who benefit from the teachings of this article that the disclosed subject matter can be modified and practiced in different but equivalent ways. Except as described in the claims below, the details of the construction or design shown in this article are not intended to be limiting. Therefore, it is obvious that the specific embodiments disclosed above can be changed or modified, and all these changes are considered to be within the scope of the disclosed subject matter. Therefore, the protection sought herein is as set forth in the appended claims.
Claims
1. A wearable device, comprising: speaker; as well as A processor configured to execute computer readable instructions which, when executed, cause the processor to: receiving a sound identifier, a spatial position, and a first pose estimate corresponding to the wearable device; updating the first pose estimate to generate a second pose estimate; rendering spatial audio data based on the sound identifier, the spatial position, and the second pose estimate; as well as The speaker is caused to produce sound corresponding to the spatial audio data.
2. The wearable device according to claim 1, further comprising: A motion sensor, the motion sensor being configured to: generating first posture metadata during a first time period, wherein the first posture metadata indicates movement of the wearable device during the first time period, and wherein the first posture estimate is generated based on the first posture metadata; as well as Second posture metadata is generated during a second time period after the first time period, wherein the second posture metadata indicates movement of the wearable device during the second time period, and wherein the second posture estimate is generated based on the second posture metadata.
3. The wearable device according to claim 1, further comprising: A camera, the camera being configured to: generating first posture metadata during a first time period, wherein the first posture metadata indicates movement of the wearable device during the first time period, and wherein the first posture estimate is generated based on the first posture metadata; as well as Second posture metadata is generated during a second time period after the first time period, wherein the second posture metadata indicates movement of the wearable device during the second time period, and wherein the second posture estimate is generated based on the second posture metadata.
4. The wearable device according to claim 2 or 3, wherein: The processor is configured to send the first posture metadata to a companion device separate from the wearable device, and wherein the processor is configured to receive the sound identifier, the spatial position, and the first posture estimate from the companion device in response to sending the first posture metadata to the companion device.
5. The wearable device according to any one of claims 1 to 3, wherein: The sound identifier identifies audio data stored at the wearable device, and wherein rendering the spatial audio data comprises spatializing the identified audio data based on the spatial position and the second pose estimate.
6. The wearable device according to claim 5, wherein: The spatial audio data causes the speaker to simulate a projection of the sound at the spatial position when producing the sound corresponding to the spatial audio data, wherein the spatial position is defined with respect to the posture of the wearable device indicated by the second posture estimate.
7. A system comprising: The wearable device according to claim 1; as well as a companion device comprising a processor for generating the first pose estimate based on first pose metadata generated by the wearable device during a first time period, Wherein, the wearable device is configured to generate second posture metadata during a second time period after the first time period, wherein the second posture metadata indicates movement of the wearable device during the second time period, and wherein the second posture estimate is generated based on the second posture metadata.
8. The system according to claim 7, wherein: The sound identifier identifies audio data stored at the wearable device, and wherein the spatial location is defined with respect to a posture of the wearable device indicated by the second posture estimate.
9. A method for pose estimation and spatial audio refinement, comprising: receiving, with the wearable device, from a companion device, a sound identifier, a spatial location, and a first pose estimate corresponding to the wearable device, wherein the first pose estimate is generated by the companion device based on first pose metadata generated during a first time period; updating, with the wearable device, the first pose estimate to generate a second pose estimate based on second pose metadata generated during a second time period after the first time period; rendering, using the wearable device, spatial audio data based on the sound identifier, the spatial position, and the second pose estimate; and Cause a speaker at the wearable device to produce sound corresponding to the spatial audio data.
10. The method according to claim 9, further comprising: generating, with the wearable device, the first gesture metadata during the first time period and transmitting the first gesture metadata to the companion device, and The second posture metadata is generated using the wearable device.
