Low latency wireless audio streaming

Abstract

An exemplary process for wirelessly streaming audio data to a head-mounted display (HMD) with low latency and reduced auditory distortion is described. This includes determining one or more statistics indicating the latency associated with audio data wirelessly received by the HMD and stored in a buffer within the HMD, and determining, at least in part, to adjust the size of the buffer based on these statistics. To reduce the buffer size, segments of audio data can be replaced with synthesized audio data segments to obtain modified audio data in the buffer. To increase the buffer size, synthesized audio data segments can be added to the audio data to obtain modified audio data in the buffer. Audio content can then be output via one or more speakers of the HMD, at least in part, based on the modified audio data.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Patent Application No. 18 / 523,379, filed November 29, 2023, the entire contents of which are incorporated herein by reference. Background Technology

[0002] Wireless streaming technology is used both inside and outside the video game industry. Within the video game industry, virtual reality (VR) gaming systems can utilize wireless streaming technology to leverage the high computing power of host computers to execute video games, while providing greater mobility for wearers of wireless VR headsets compared to tethered (non-wireless) head-mounted displays.

[0003] Despite these advantages, developing VR gaming systems that reliably stream data over consumer-grade wireless LANs (LANs) such as home Wi-Fi networks can be challenging. For example, the inherent unpredictability of consumer-grade LAN performance, combined with the relatively low latency tolerance of VR gaming applications, makes it difficult to ensure that every data packet arrives at the VR headset in time to output the corresponding VR content at a constant rate without distortion. Since this involves audio data, if the data packet carrying the audio data is lost or arrives late at the VR headset, the user may hear auditory distortions in the audio content, such as "bangs," "scratches," and / or sudden silences.

[0004] This provides technical solutions for improving and enhancing these and other systems. Attached Figure Description

[0005] The accompanying drawings are provided for detailed description. In the drawings, the leftmost number of the reference numerals indicates the drawing in which that reference numeral first appears. The same reference numerals are used in different drawings to denote similar or identical parts or features.

[0006] Figure 1 An exemplary environment is illustrated for wirelessly streaming audio data to a head-mounted display (HMD) with low latency.

[0007] Figure 2 This is a schematic diagram illustrating a technique for reducing the size of the audio data buffer in an HMD.

[0008] Figure 3 This is a schematic diagram illustrating a technique for increasing the size of the audio data buffer in an HMD.

[0009] Figure 4This is a schematic diagram illustrating an exemplary threshold that can be used to maintain the audio latency within a target latency range.

[0010] Figure 5 This is a schematic diagram illustrating a technique for mitigating auditory distortion that would otherwise be caused by lost data packets in an audio data stream.

[0011] Figure 6 This is a flowchart of an exemplary process for wirelessly streaming audio data to an HMD with low latency.

[0012] Figure 7 This is a flowchart of an exemplary process for adjusting audio latency by modifying buffered audio data using synthesized audio data segments.

[0013] Figure 8 This is a flowchart of an exemplary process for using a threshold to maintain the audio latency within a target latency range and / or to reduce auditory distortion in audio content output via a speaker from an HMD.

[0014] Figure 9 This is a flowchart of an exemplary process for mitigating auditory distortion that would otherwise be caused by a stream of audio data with missing data packets.

[0015] Figure 10 Exemplary components of an HMD system that can implement the techniques disclosed herein are illustrated according to embodiments disclosed herein. Detailed Implementation

[0016] As mentioned above, VR gaming applications have relatively low latency tolerance, which makes it challenging to develop VR gaming systems that reliably stream audio data over consumer-grade wireless LANs such as home Wi-Fi networks. For example, if there is network congestion and / or interference spikes in the wireless LAN during a gaming session, the audio data carried by the wirelessly transmitted packets may arrive at the HMD later than expected, or may not arrive at all, which could result in auditory distortion in the audio content output via the HMD's speakers.

[0017] This document describes techniques, devices, and systems for wirelessly streaming audio data to an HMD with low latency and reduced auditory distortion in the audio content output to the wearer of the HMD. As used herein, "latency" can refer to the time interval between the time when audio data is received by the HMD (or an earlier time associated with the generation of audio data or transmission of audio data to the HMD) and the time when the corresponding audio content is output via the HMD's speakers. In some instances, the HMD is configured to store the audio data wirelessly received (e.g., from a host computer) in a buffer before outputting the corresponding audio content via one or more speakers of the HMD. In some instances, the buffer is a jitter buffer that is at least partially used to eliminate jitter by queuing the audio data in a sequence of audio data segments to absorb the delay difference between packets of wireless transmission carrying the audio data. In these instances, the processor (e.g., the HMD's processor) is configured to dynamically adjust the size of the buffer, thereby dynamically adjusting the latency associated with the audio data. As used herein, “dynamically” adjusting the size of the buffer can mean using an automated process to increase or decrease the size of the buffer in real time or near real time without user intervention. Furthermore, the size of the buffer can be adjusted by modifying the buffered audio data in a way that mitigates auditory distortion that would otherwise be present in the audio content output via the HMD’s speakers based on the modification of the buffered audio data.

[0018] For illustration, if the processor determines to reduce the size of the buffer based at least in part on one or more latency statistics associated with the buffered audio data, the processor can replace the buffered audio data segments with synthesized audio data segments to obtain modified audio data in the buffer. In some instances, the processor can create synthesized audio data segments based at least in part on combinations of non-sequential segments of the buffered audio data, and the processor can replace the non-sequential segments and one or more intermediate segments of audio data between the non-sequential segments with synthesized audio data segments to obtain modified audio data in the buffer. In some instances, the synthesized audio data segments are created by overlapping non-sequential segments and cross-fading the corresponding audio signal waveforms of the non-sequential segments, thereby “smoothing” the transition between the combined non-sequential segments to mitigate (e.g., hide, conceal, etc.) auditory distortion that would otherwise be caused by modifications to the buffered audio data. Conversely, if the processor determines to increase the size of the buffer based at least in part on latency statistics, the processor can add synthesized audio data segments to the buffered audio data to obtain modified audio data in the buffer. In some instances, the processor can create synthesized audio data segments, at least in part, based on a combination of discontinuous segments of buffered audio data (e.g., by overlapping discontinuous segments and crossfading the corresponding audio signal waveforms of the discontinuous segments). In this way, the buffered audio data can be modified (e.g., adjusted on a millisecond (ms) scale) to remain within a target latency range while mitigating auditory distortion in the corresponding audio content output via the HMD's speakers. Therefore, when network conditions fluctuate, the processor can execute instructions to maintain the audio latency within a specified latency tolerance, and the modifications made to the buffered audio data are imperceptible to the HMD wearer when the corresponding audio content is output via the HMD's speakers.

[0019] While many of the examples described herein relate to gaming systems, the descriptions to come are not limited to gaming systems, or even VR systems, as the techniques described herein can be implemented in any suitable HMD system, including a host computer and a wireless HMD communicatively coupled thereto. For example, depending on the circumstances, the HMD can be worn by a user (or wearer) to immerse the user in VR, augmented reality (AR), and / or mixed reality (MR) environments, which may or may not be game-related. One or more display panels of the HMD are configured to present images based on video data generated by an executing application (e.g., a video game), and one or more speakers of the HMD are configured to output audio content based on audio data generated by the executing application. The application can execute on the host computer and can generate pixel data and audio data, the pixel data being processed to present corresponding video content (e.g., images) via the HMD, and the audio data being processed to output audio content via the HMD. The video content is viewed by the user through optics included in the HMD, and the audio content is heard by the user through the HMD's speakers, thus allowing the user to perceive the content as if they were immersed in a VR, AR, and / or MR environment.

[0020] This document also discloses techniques, apparatus, and systems for using thresholds to maintain audio latency within a target latency range and / or mitigate auditory distortion in audio content output via a speaker from an HMD. For example, if the latency associated with audio data wirelessly received by the HMD and stored in a buffer within the HMD is within a target latency range, the processor can suppress adjusting the buffer size; however, if the audio latency exceeds a low threshold less than the target latency, the processor can dynamically increase the buffer size to maintain the audio latency within the target latency range. Conversely, if the audio latency exceeds a high threshold greater than the target latency, the processor can decrease the buffer size to maintain the audio latency within the target latency range. In some instances, if the audio latency exceeds a critical threshold less than the aforementioned low threshold, the processor can attempt to allow the audio latency to increase above the critical threshold by iteratively outputting audio content via the HMD's speaker a predefined number of times to allow the audio latency to be "replayed," and the processor can then attempt to increase the audio latency above the low threshold to bring the audio latency back within the target latency range. In some instances, if network conditions deteriorate and / or if a system failure causes the HMD to fail to receive additional audio data after the audio latency exceeds a critical threshold, the processor can implement a "last resort" technique of fading out the remaining audio content by gradually decreasing the volume of the audio content as it is output through the HMD's speakers. After this fade-out, normal operation of the HMD system can resume once the HMD receives audio data again. Therefore, under normal operating conditions, the audio latency can be maintained within a target latency range, and auditory distortion and / or sudden silence can be mitigated when network conditions deteriorate beyond a certain point.

[0021] This document also discloses techniques, apparatus, and systems for mitigating auditory distortion that would otherwise be caused by lost data packets within an audio data stream. For example, a processor can determine, at least in part, that a lost data packet was not received by the HMD based on analysis of audio data packets wirelessly received by the HMD. If a data packet is dropped or lost after being wirelessly transmitted by the host computer, it may be lost from the audio data stream. In this scenario, the processor can create a synthesized audio data segment, at least in part, based on a combination of discontinuous segments of audio data surrounding the lost data packet (e.g., by overlapping discontinuous segments and crossfading the corresponding audio signal waveforms of the discontinuous segments), and the processor can replace the discontinuous segments with the synthesized audio data segment to obtain modified audio data in a buffer. Any auditory distortion that would otherwise be caused by the lost data packet is mitigated when the modified audio data is processed for outputting corresponding audio content via the HMD's speakers.

[0022] The technologies, devices, and systems described herein can provide an improved user experience (e.g., an improved gaming experience for video game players). This is at least because buffered audio data can be moderately modified (e.g., fine-tuned) to control audio latency (e.g., on the scale of milliseconds) while mitigating auditory distortion in the audio content output via the HMD. Therefore, the HMD wearer is able to hear the audio content, and any distortion in the audio content due to modifications to the buffered audio data and / or discontinuities in the buffered audio data is imperceptible to the wearer. Additionally or alternatively, the technologies, devices, and systems described herein improve the audio output functionality of HMDs and / or HMD systems (e.g., VR gaming systems) by mitigating auditory distortion in the audio content while maintaining a low latency tolerance for streaming audio data. Additionally or alternatively, the technologies, devices, and systems described herein allow one or more devices to conserve resources regarding processing resources, memory resources, networking resources, etc., in the various ways described herein. For example, memory resources can be saved by controlling the audio latency within a target latency limit (e.g., by dynamically reducing the size of the HMD's buffer), thus suppressing the storage of audio data exceeding a threshold amount in the buffer at any given time. As another example, network resources are saved by compensating for lost packets from the audio data stream using techniques that avoid retransmission; alternatively, modifying the buffered audio data without retransmitting lost packets mitigates auditory distortion in the audio content that would otherwise be caused by the lost packets. These are merely examples of computational resources that can be saved by implementing the techniques described herein.

[0023] While many of the examples provided herein relate to streaming audio data over a wireless LAN, such as a home Wi-Fi network, it should be understood that the techniques described herein can be implemented in geographically distributed systems and / or networks, such as systems streaming audio data over a wide area network (e.g., the Internet). For example, an HMD streaming data from a remote server (e.g., the “cloud”) can implement the techniques described herein to stream audio data with low latency and by mitigating auditory distortion in the audio content. Furthermore, as noted above, the techniques described herein can be implemented in gaming (e.g., VR gaming) systems and / or non-gaming and / or non-VR systems.

[0024] Figure 1 An exemplary environment 100 is illustrated for wirelessly streaming audio data to an HMD with low latency. Figure 1 The image depicts an HMD 102 worn by user 104, and a host computer 106. Figure 1An exemplary embodiment of a host computer 106 in the form of a personal computer (PC) is depicted, which may be located, for example, in the home 108 of user 104. However, it should be understood that this exemplary type of host computer 106 is not limited to this disclosure. For example, host computer 106 can be implemented as any type of computing device and / or any number of computing devices, including but not limited to PCs, laptops, desktop computers, portable digital assistants (PDAs), mobile phones, tablets, set-top boxes, game consoles, server computers, wearable computers (e.g., smartwatches), or any other electronic device capable of sending data to and receiving data from other devices. Host computer 106 may be co-located in the same environment as HMD 102, such as in the home 108 of user 104 wearing HMD 102. Alternatively, host computer 106 may be remotely located relative to HMD 102, such as a host computer 106 in the form of a server computer located in a remote geographical location relative to HMD 102. In a specific implementation of remote host computer 106, host computer 106 can be communicatively coupled to HMD 102 via a wide area network such as the Internet. In a specific implementation of local host computer 106, host computer 106 can be co-located with HMD 102 in an environment (e.g., home 108), whereby host computer 106 and HMD 102 can be communicatively coupled together via a LAN, either directly or via one or more intermediate network devices 109. Figure 1 In one example, the network device 109 used to wirelessly transmit data between host computer 106 and HMD 102 can represent a wireless router.

[0025] exist Figure 1 In this example, HMD 102 and host computer 106 are communicatively coupled together and configured to work collaboratively to generate data and output corresponding content (e.g., images, audio content, etc.) via one or more output devices of HMD 102. This collaboration allows content (e.g., video and audio content for VR games) to be rendered iteratively at HMD 102. In the illustrated specific implementation, HMD 102 includes one or more processors 110, a communication interface 112, memory 114 (e.g., non-transitory computer-readable medium), and / or one or more speakers 116. Reference will be made below. Figure 10The components of HMD 102 and / or the HMD system are described in more detail below. Typically, HMD 102 may include logical components (e.g., software, hardware, and / or firmware, etc.) configured to implement the technologies, functionalities, and / or operations described herein. Memory 114 may include various modules, such as instructions, data repositories, etc., which may be configured to execute on processor 110 to perform the technologies, functionalities, and / or operations described herein. Exemplary functional modules in the form of audio data management components 118 are shown stored in memory 114 and executable on processor 110, although the same functionality may alternatively be implemented in hardware, firmware, or as a system-on-a-chip (SOC) and / or other logic. Furthermore, additional or different functional modules may be stored in memory 114 and executable on processor 110. Figure 1 A further example is a buffer 120 (e.g., a volatile memory buffer, such as a dynamic random access memory (DRAM) buffer), which may be part of memory 114 and is used to temporarily store audio data until it is used, transferred, deleted, and / or permanently stored.

[0026] In some instances, HMD 102 may refer to a VR head-mounted display for use in VR systems, such as for use with VR gaming systems. In these instances, host computer 106 may execute one or more VR video game applications and may wirelessly transmit associated data to HMD 102 to render VR video game content via HMD 102. However, HMD 102 may additionally or alternatively be implemented as an AR head-mounted display for AR video game applications, an MR head-mounted display for MR video game applications, or a head-mounted display for VR, AR, and / or MR applications unrelated to gaming (e.g., industrial applications, robotics applications, military / weapon applications, medical applications, etc.).

[0027] Typically, the application running on host computer 106 can represent a graphics-based application (e.g., a video game). The application is configured to generate pixel data for video content (e.g., a series of images) and audio data for audio content, and such data is wirelessly transmitted to HMD 102. Data received by HMD 102 can be processed to output corresponding content (e.g., video and audio content) via HMD 102. In some instances, host computer 106 can iteratively receive data, such as head tracking data, from HMD 102 at any suitable frequency (e.g., approximately 1000 Hz), and the application running on host computer 106 can generate video and audio data, at least in part, based on the data received from HMD 102 (e.g., head tracking data). For example, pose data indicative of the predicted pose of HMD 102 can be received by host computer 106 and provided to the application running on it for generating data for upcoming frames. Figure 1 Audio data 122 is illustrated, which is generated at host computer 106 (e.g., by an application such as a video game) and wirelessly transmitted to HMD 102, although it should be understood that pixel data can also be generated and wirelessly transmitted to HMD 102. Audio data 122 can be wirelessly transmitted to HMD 102 in the form of data packets. Therefore, HMD 102 can be configured to receive data packets carrying audio data 122 via communication interface 112 (e.g., radio).

[0028] In some instances, the data packets carrying audio data 122 can be sequentially numbered, indicating the order of the data packets. In this way, the audio data management component 118 can be configured to analyze the received data packets to determine if any data packets in the audio data stream are out of order or missing. In some instances, individual data packets can carry a single frame of audio data 122, or multiple sequential frames of audio data 122. A “frame” of audio data 122 is a collection of time-consistent audio samples. In some instances, each frame has multiple (e.g., two) audio samples; one audio sample from the left channel and another audio sample from the right channel, and these channels can correspond to the left and right speakers 116 of the HMD 102. In some instances, individual data packets can carry a specific amount of audio data 122. The specific amount of audio data 122 carried in a single packet can be measured using any suitable metric, such as the number of bytes of data (or megabytes, gigabytes, etc.) or the amount of playback time (e.g., 10 milliseconds of audio data 122).

[0029] In response to receiving audio data 122 via the communication interface 112 of HMD 102, processor 110 may execute audio data management unit 118 to process the received audio data 122 for performing the techniques, functions, and / or operations described herein. For example, processor 110 may execute audio data management unit 118 to store audio data 122 in buffer 120 of HMD 102. In some instances, audio data 122 may be stored in buffer 120 as a sequence of audio data segments 124(1), 124(2), ..., 124(N) (collectively referred to as 124). Individual audio data segments 124 may represent audio samples, frames, data packets, or any other suitable segment of the received audio data 122. In some instances, the audio data management component 118 may be configured to divide the received audio data 122 into segments 124, such as by using any suitable segmentation algorithm to divide the audio data 122 into blocks or chunks, such as an algorithm that analyzes the audio signal waveform associated with the audio data 122 and divides the audio data 122 into segments 124 at points in the waveform where the amplitude of the waveform is minimized, in order to mitigate distortion when segments 124 are moved, deleted, and / or combined in the buffer 120.

[0030] Initially, when user 104 powers on HMD 102, buffer 120 may be empty. In some instances, processor 110 may execute audio data management unit 118 to “lock” buffer 120 until a threshold amount of audio data 122 (e.g., a threshold amount of audio data segments 124) is stored in buffer 120. This threshold amount of audio data 122 is sometimes referred to herein as the “fill depth.” In this context, “locking” buffer 120 may mean keeping queued audio data segments 124 in buffer 120 and suppressing processing of audio data segments 124 to prevent audio content from being output via speaker 116 of HMD 102. However, once buffer 120 is “filled” to this fill depth, processor 110 may execute audio data management unit 118 to begin processing the buffered audio data segments 124 so that the corresponding audio content is output (or played) via speaker 116 of HMD 102.

[0031] In some instances, buffer 120 represents a jitter buffer. In these instances, buffer 120 can be used to eliminate jitter by queuing audio data 122 in a sequence of audio data segments 124(1)-(N) to absorb the delay difference between packets of wireless transmission carrying audio data 122 when audio data 122 is delivered to HMD 102. In other words, storing audio data 122 in buffer 120 can provide time for any late-arriving audio packets to “catch up”, so that buffered audio data 122 can be processed to output the corresponding audio content at a constant rate via speaker 116. In some instances, buffer 120 represents a dynamic jitter buffer, and therefore, the size 126 of buffer 120 can be dynamically adjusted. In these instances, processor 110 is configured to execute audio data management unit 118 to dynamically adjust the size 126 of buffer 120, thereby adjusting the latency associated with audio data 122. Furthermore, the size 126 of the buffer 120 can be adjusted to reduce auditory distortion that would otherwise be present in the audio content output via the speaker 116 of the HMD 102, as described in more detail below.

[0032] In some instances, the size 126 of buffer 120 can be increased to a maximum size and decreased to a minimum size. As the size 126 of buffer 120 increases toward the maximum size, the latency associated with the audio data 122 increases, meaning the playback delay of the corresponding audio content becomes longer. As the size 126 of buffer 120 decreases toward the minimum size, the latency associated with the audio data 122 decreases, meaning the playback delay of the audio content becomes shorter. Therefore, one can conceptualize adjusting the size 126 of buffer 120 as "fast-forwarding" or "rewinding" the buffered audio data. Because the audio latency tolerance may be relatively low, the maximum size of buffer 120 may be limited to the amount of audio data 122 corresponding to audio playback of less than 100 milliseconds. In some instances, the maximum size of buffer 120 is in the range of 30 to 50 milliseconds for audio playback. For example, unlike pre-recorded audio data associated with music, movies, etc., storing large amounts of audio data 122 (e.g., audio playback of more than 100 milliseconds) in buffer 120 may be impractical. Alternatively, the audio content can be output within tens of milliseconds (e.g., within 20 to 25 milliseconds) after the HMD 102 receives the corresponding audio data 122. In some instances, the minimum size of the buffer 120 can be a non-zero value to ensure that at least some audio data 122 is stored in the buffer 120 at any given time.

[0033] The process described herein is illustrated as a set of boxes in a logic flowchart, representing a sequence of operations that can be implemented in hardware, software, firmware, or a combination thereof (i.e., logic). In the context of software, a box represents a computer-executable instruction that performs the enumerated operations when executed by one or more processors. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific abstract data type. The order in which the operations are described is not intended to be construed as limiting, and any number of described boxes can be combined in any order and / or in parallel to implement the process.

[0034] Figure 1 A flowchart of an exemplary process 128 for wirelessly transmitting audio data to HMD 102 with low latency is further illustrated. At block 130, processor 110 of HMD 102 may execute audio data management component 118 to determine one or more statistics indicating the latency associated with audio data 122 wirelessly received by HMD 102 and stored in buffer 120 of HMD 102. The latency statistics determined at block 130 may be used to determine whether to adjust the size 126 of buffer 120, whether to increase or decrease the size 126 of buffer 120, and / or by how much to adjust the size 126 of buffer 120.

[0035] In some instances, the latency statistics determined at box 130 may include, but are not limited to, the minimum latency of received audio data 122 (e.g., audio data segment 124 stored in buffer 120), the maximum latency of received audio data 122, the average latency of received audio data 122, and / or the standard deviation associated with received audio data 122. In some instances, the latency statistics determined at box 130 may indicate the number of milliseconds of audio playback associated with audio data 122 (e.g., audio data segment 124) stored in buffer 120. In some instances, the latency statistics may be determined at box 130 at any suitable rate or frequency (such as 2 Hz). There is a trade-off between determining latency statistics at a faster rate versus a slower rate. For example, determining latency statistics at a slower rate may save computational resources, but may require larger modifications to the buffered audio data 122 to adjust the size 126 of buffer 120. On the other hand, if latency statistics are determined at a faster rate, more computational resources are used to determine the latency statistics, but the buffered audio data 122 can be modified at a faster rate to adjust the size 126 of the buffer 120 by a relatively small amount, which may not be noticeable to the wearer of the HMD 102. In any case, at box 130, latency statistics can be determined by analyzing the audio data 122 received since the previous latency statistics were determined (e.g., audio data 122 received in the last 500 milliseconds).

[0036] At box 132, a determination can be made regarding whether to increase the size 12 of buffer 120. In some instances, this determination at box 132 is based at least in part on latency statistics determined at box 130. For example, processor 110 may execute audio data management component 118 at box 132 to compare the latency statistics determined at box 130 with a target latency time and / or with a low threshold less than the target latency time. Based at least in part on this comparison, audio data management component 118 may determine at box 132 whether to increase the size 126 of buffer 120, and / or by how much to increase the size 126 of buffer 120. In other words, an exemplary objective of audio data management component 118 may be to maintain the audio latency within a target latency range, and the latency statistics determined at box 130 may provide information about whether the audio latency is outside or within the target latency range. If the determination at box 132 is to increase the size 126 of buffer 120, then process 128 can follow the "yes" route from box 132 to box 134.

[0037] At block 134, processor 110 may execute audio data management unit 118 to add synthesized audio data segments to buffered audio data 122 (e.g., to a sequence of audio data segments 124 stored in buffer 120) to obtain modified audio data in buffer 120. By adding synthesized audio data segments to buffered audio data 122 without deleting or replacing any existing audio data segments 124, the size 126 of buffer 120 increases, thereby increasing audio latency. See below for further details. Figure 3 Exemplary techniques for creating synthetic audio data segments at box 134 to be added to buffered audio data 122 are discussed in more detail. In some instances, the synthetic audio data segments added to buffered audio data 122 at box 134 are generated by artificial intelligence (AI) (e.g., AI-generated synthetic audio data segments). For example, a trained AI model (e.g., a trained machine learning model) can receive at least some buffered audio data from buffered audio data 122 as input (e.g., a cue), and the trained model can generate synthetic audio data segments to be added to the audio data stream in buffer 120. In some instances, the amount by which the size 126 of buffer 120 is to be increased is determined at box 132, and the number of synthetic audio data segments added to buffered audio data 122 at box 134 is at least partially based on that amount. For example, if it is determined that the audio latency will be increased by 10 milliseconds, a synthetic audio data segment corresponding to 10 milliseconds of audio playback can be added to buffered audio data 122. In another instance, if it is determined that the audio latency will be increased by 20 milliseconds, then two synthesized audio data segments corresponding to a total of 20 milliseconds of audio playback can be added to the buffered audio data 122. After box 134, process 128 can proceed to box 136, where processor 110 can wait for the start of the next statistical calculation period (e.g., after 500 milliseconds have elapsed since the latency statistics were determined at box 130, process 128 can proceed from box 136 to box 130 to iterate over process 128). If the determination at box 132 is to suppress the increase of the size 126 of buffer 120, then process 128 can follow a "no" route from box 132 to box 138.

[0038] At box 138, a determination can be made regarding whether to reduce the size 126 of buffer 120. In some instances, this determination at box 138 is based at least in part on latency statistics determined at box 130. For example, processor 110 may execute audio data management unit 118 at box 138 to compare the latency statistics determined at box 130 with a target latency time and / or with a high threshold greater than the target latency time. Based at least in part on this comparison, audio data management unit 118 may determine at box 138 whether to reduce the size 126 of buffer 120, and / or by how much the size 126 of buffer 120 should be reduced. If the determination at box 138 is to reduce the size 126 of buffer 120, then process 128 may follow a "yes" route from box 138 to box 140.

[0039] At block 140, processor 110 may execute audio data management unit 118 to replace audio data segments 124 stored in buffer 120 with synthesized audio data segments to obtain modified audio data in buffer 120. By replacing multiple audio data segments 124 in buffered audio data 122 with synthesized audio data segments, the size 126 of buffer 120 is reduced, thereby reducing audio latency. The synthesized audio data segments are smaller in size than the replaced multiple audio data segments 124 (or shorter than the audio playback duration). See below for reference. Figure 2Exemplary techniques for creating a synthetic audio data segment at box 140 to replace multiple audio data segments 124 in buffered audio data 122 are discussed in more detail. In some instances, the synthetic audio data segment used at box 140 to replace multiple audio data segments 124 in buffered audio data 122 is an AI-generated synthetic audio data segment. For example, a trained AI model (e.g., a trained machine learning model) may receive at least some of the buffered audio data in buffered audio data 122 as input (e.g., a cue), and the trained model may generate a synthetic audio data segment to replace multiple audio data segments within the audio data stream in buffer 120. In some instances, the amount by which the size 126 of buffer 120 is to be reduced is determined at box 138, and the number of audio data segments 124 replaced at box 140 is at least partially based on that amount. Following box 140, process 128 can proceed to box 136, where processor 110 can wait for the start of the next statistical calculation period (e.g., 500 milliseconds after the wait time statistics were determined at box 130, process 128 can proceed from box 136 to box 130 to iterate process 128). If the determination at box 138 is to suppress the reduction of buffer 120 size 126, then process 128 can follow a "no" route from box 138 to box 136, where processor 110 can wait for the start of the next statistical calculation period before proceeding to box 130 to iterate process 128. Therefore, process 128 can be implemented to dynamically adjust buffer 120 size 126, thereby adjusting the audio wait time on the fly (e.g., during a game session involving HMD 102).

[0040] Figure 2 This is a schematic diagram illustrating a technique for reducing the size 126 of the audio data buffer 120 of the HMD 102. In some instances, Figure 2 The illustrated technique can be performed at box 140 of process 128 described above. For example... Figure 2 As shown, the audio data 122 stored in buffer 120 at any given time may include a sequence of audio data segments 124(1), 124(2), 124(3), 124(4), and 124(5). As indicated by the ellipses on the left and right sides of the sequence of segments 124(1)-(5), in some instances, additional segments 124 may be stored in buffer 120. If each segment 124 corresponds to 10 milliseconds of audio playback, then the size 126 of buffer 120 may correspond to 50 milliseconds of audio latency, assuming that no other segments 124 besides segments 124(1)-(5) are stored in buffer 120. Figure 2As shown, a synthesized audio data segment 200 can be created at least in part based on a combination of non-contiguous segments 124 of the audio data 122 stored in buffer 120. For example, segments 124(2) and 124(4) are non-contiguous segments 124 because, in the sequence of segments 124(1)-(5), segment 124(4) does not immediately follow segment 124(2); instead, the intermediate segment 124(3) lies between non-contiguous segments 124(2) and 124(4). Any suitable selection algorithm can be used to determine which non-contiguous segments 124 of the audio data 122 stored in buffer 120 should be selected. In some instances, the most recently stored non-contiguous segment 124 is selected to allow time for modifying the buffered audio data 122 while processing the earlier stored segment 124. In some instances, a non-contiguous segment 124 near the middle of the sequence of buffered segments 124 is selected.

[0041] Figure 2 Further examples illustrate that discontinuous segments 124(2) and 124(4), as well as intermediate segment 124(3), can be replaced with synthesized audio data segment 200 to obtain modified audio data 202 in buffer 120. Figure 2 In one instance, the modified audio data 202 in buffer 120 may include a new sequence of audio data segments comprising an original audio data segment 124(1), followed by a synthesized audio data segment 200, followed by another original audio data segment 124(5). In other words, a gap in the audio data stream can be created by removing an intermediate segment 124(3), and this gap can be closed by combining discontinuous segments 124(2) and 124(4) surrounding the intermediate segment 124(3) to create a synthesized audio data segment 200 used to replace... Figure 2 In the example, all three segments 124(2)-124(4). To make a larger adjustment to the size 126 of the buffer 120, more segments 124 can be replaced. For example, non-contiguous segments 124(1) and 124(5) are combined to create a synthesized audio data segment that can replace all five segments 124(1)-(5), which includes three intermediate segments 124(2)-(4). In any case, in Figure 2 In an example, if each segment 124 corresponds to 10 milliseconds of audio playback, and if the synthesized audio data segment 200 corresponds to, say, 15 milliseconds of audio playback, then the size 126 of buffer 120 can be reduced by 15 milliseconds (e.g., from 50 milliseconds of audio latency to 35 milliseconds of audio latency). In some instances, a 35-millisecond audio latency is within the target latency range, and therefore, Figure 2 The illustrated latency reduction technique can be used to maintain audio latency within a target latency range.

[0042] In some instances, the synthesized audio data segment 200 can be created by overlapping discontinuous segments 124(2) and 124(4) and crossfading the corresponding audio signal waveforms of the discontinuous segments 124(2) and 124(4). For example, Figure 2 The amount by which audio data segments 124(2) and 124(4) can overlap by 204 is shown. In some instances, the amount of overlap 204 is predetermined. In some instances, the amount of overlap 204 is dynamically determined. For example, the amount of overlap 204 may be based at least in part on the amount of audio data 122 (e.g., the number of audio data segments 124 currently stored in buffer 120). In some instances, the amount of overlap 204 may be determined at least in part on the amount of audio data 112 stored in buffer 120 before, after, and / or including the segments 124(2)-(4) to be replaced (e.g., the number of audio data segments 124). To crossfade the corresponding audio signal waveforms of the discontinuous segments 124(2) and 124(4), a crossfade function 206 is initiated at a value of 1 for one of the audio signal waveforms and scaled to 0, and a crossfade function 208 is initiated at a value of zero for the other of the audio signal waveforms and scaled to 1. Metrics associated with the audio signal waveforms (e.g., voltage, power, etc.) can be maintained at constant values ​​across overlap 204. In some instances, windowing functions, such as the Hann windowing function, can be used to combine the discontinuous audio data segments 124(2) and 124(4) to create a composite audio data segment 200. In some instances, the Pitch Synchronous Overlap and Addition (PSOLA) algorithm can be used to combine the discontinuous audio data segments 124(2) and 124(4) to create a composite audio data segment 200. In some instances, the processor 110 analyzes the corresponding audio signal waveforms to identify the zero-crossing points at the start of the crossfade functions 206 and 208. These techniques can help to “smooth” the transition between discontinuous segments 124(2) and 124(4) of the combination, in order to mitigate (e.g., hide, conceal, etc.) auditory distortion that would otherwise result from modifications to the buffered audio data 122.

[0043] Figure 3 This is a schematic diagram illustrating a technique for increasing the size of the audio data buffer 120 of the HMD 102. In some instances, Figure 3 The illustrated technique can be performed at box 134 of process 128 described above. For example... Figure 3As shown, the audio data 122 stored in buffer 120 at any given time may include a sequence of audio data segments 124(6), 124(7), and 124(8). As indicated by the ellipses on the left and right sides of the sequence of segments 124(6)-(8), in some instances, additional segments 124 may be stored in buffer 120. Figure 3 As shown, a synthesized audio data segment 300 can be created, at least in part, based on a combination of non-contiguous segments 124 of the audio data 122 stored in buffer 120. For example, segments 124(6) and 124(8) are non-contiguous segments 124 because, in the sequence of segments 124(6)-(8), segment 124(8) does not immediately follow segment 124(6); instead, the intermediate segment 124(7) lies between non-contiguous segments 124(6) and 124(8). Any suitable selection algorithm can be used to determine which non-contiguous segments 124 of the audio data 122 stored in buffer 120 should be selected. In some instances, the most recently stored non-contiguous segment 124 is selected to allow time for modifying the buffered audio data 122 while processing earlier stored segments 124. In some instances, a non-contiguous segment 124 near the middle of the sequence of buffered segments 124 is selected.

[0044] Figure 3 Further example illustrates that synthesized audio data segment 300 can be added to buffered audio data 122 (e.g., to the sequence of segments 124(6)-(8)) to obtain modified audio data 302 in buffer 120. Figure 3 In one instance, the modified audio data 302 in buffer 120 may include a new sequence of audio data segments comprising original audio data segment 124(6), followed by another original audio data segment 124(7), followed by synthesized audio data segment 300, followed by another original audio data segment 124(8). In other words, synthesized audio data segment 300 may be inserted or appended to the sequence of segments 124(6)-(8) to obtain modified audio data 302 in buffer 120. For larger adjustments to the size 126 of buffer 120, a larger number of synthesized audio data segments (e.g., multiple synthesized audio data segments) may be created and added to the buffered audio data 122. In some instances, synthesized audio data segment 300 is inserted into the buffered audio data 122 at a position adjacent (e.g., immediately before or after) one of the non-contiguous segments 124(6) and 124(8) used to create synthesized audio data segment 300. For example, in Figure 3 In this instance, the synthesized audio data segment 300 is positioned adjacent to segment 124(8). For example, the synthesized audio data segment 300 precedes one of the non-contiguous segments 124 within the modified audio data 302 (i.e., segment 124(8)).

[0045] In some instances, the synthesized audio data segment 300 can be created by overlapping discontinuous segments 124(6) and 124(8) and crossfading the corresponding audio signal waveforms of the discontinuous segments 124(6) and 124(8). This technique can be similar to the one described above. Figure 2 The techniques described for discontinuous segments 124(2) and 124(4) are as follows. For example, discontinuous segments 124(6) and 124(8) can overlap by a predetermined or dynamically determined amount of overlap 304. Furthermore, in order to crossfade the corresponding audio signal waveforms of discontinuous segments 124(6) and 124(8), by starting the crossfade function 306 at a value of 1 for one of the audio signal waveforms and sloping it to 0, and by starting the crossfade function 308 at a value of zero for the other of the audio signal waveforms and sloping it to 1, the metrics associated with the audio signal waveforms (e.g., voltage, power, etc.) can be maintained at constant values ​​across overlap 304. In some instances, a windowing function such as the Hann windowing function can be used to combine discontinuous audio data segments 124(6) and 124(8) to create a synthetic audio data segment 300. In some instances, the PSOLA algorithm can be used to combine discontinuous audio data segments 124(6) and 124(8) to create a synthetic audio data segment 300. In some instances, processor 110 analyzes the corresponding audio signal waveform to identify the zero-crossing points at the start of crossfade functions 306, 308. These techniques can help “smooth” the transition between discontinuous segments 124(6) and 124(8) of the combination to mitigate (e.g., hide, conceal, etc.) auditory distortion that would otherwise be caused by modifications to the buffered audio data 122.

[0046] Figure 4 This is a schematic diagram illustrating an instance-like threshold that can be used to maintain audio latency within a target latency range. Figure 4 In some instances, the target latency 400 can represent the ideal audio latency of an HMD system under normal conditions (e.g., where network traffic flows with little or no network congestion and / or little or no interference, etc.). In some instances, the target latency 400 is predetermined. In one instance, the target latency 400 can be set to a value in the range of approximately 20 to 25 milliseconds during audio playback. In some instances, the target latency 400 can be dynamically determined, at least in part, based on periodically determined latency statistics (e.g., in...). Figure 1 (See box 130 of process 128). An instance of the purpose of the audio data management component 118 may be to maintain the audio latency within a target latency of 400 by iteratively adjusting the size 126 of the buffer 120, as described herein.

[0047] Figure 4The low threshold 402 and high threshold 404 are further described to provide a margin around the target latency 400. Audio latency within these soft limits (e.g., equal to or greater than the low threshold 402 and less than or equal to the high threshold 404) can be considered to be within the “range” of the target latency 400. Figure 4 As illustrated, a low threshold 402 is less than a target wait time 400, and a high threshold 404 is greater than a target wait time 400. In some instances, the low threshold 402 is approximately 10 milliseconds less than the target wait time 400 in terms of audio playback amount, and the high threshold 404 is approximately 10 milliseconds less than the target wait time 400 in terms of audio playback amount. In some instances, the low threshold 402 and / or the high threshold 404 are predetermined and are set at a fixed distance from the target wait time 400. In some instances, the low threshold 402 and / or the high threshold 404 are at least partially based on periodically determined wait time statistics (e.g., in...). Figure 1 (In process 128, at box 130) it is dynamically adjusted relative to or in coordination with the target waiting time 400. Figure 4A critical threshold 406, less than the low threshold 402, is further described. In some instances, these exemplary thresholds 402, 404, 406 can be used to maintain audio latency within a target latency 400 and / or mitigate auditory distortion in the audio content output via the speaker 116 of the HMD 102. For example, if the latency associated with audio data 122 wirelessly received by the HMD 102 and stored in buffer 120 is within the target latency 400, the audio data management unit 118 can suppress adjustments to the size 126 of buffer 120. However, if the audio latency crosses the low threshold 402, the audio data management unit 118 can dynamically increase the size 126 of buffer 120 to keep the audio latency within the target latency 400. Conversely, if the audio latency exceeds the high threshold 404, the audio data management unit 118 can decrease the size 126 of buffer 120 to keep the audio latency within the target latency 400. In some instances, if the audio latency exceeds a critical threshold 406, the audio data management unit 118 may attempt to allow the audio latency to increase above the critical threshold 406 by iteratively outputting audio content via the speaker 116 of the HMD 102 for a predetermined number of times (e.g., twice) to allow the audio latency to be "replayed." The audio data management unit 118 may then attempt to increase the audio latency above a low threshold 402, bringing the audio latency back within the target latency range 400. In some instances, if network conditions deteriorate and / or if a system failure occurs, causing the HMD 102 to not receive additional audio data 122 after the audio latency exceeds the critical threshold 406, the audio data management unit 118 may implement a "last resort" technique of fading out the remaining audio content by gradually decreasing the volume of the remaining audio content while outputting it via the speaker 116 of the HMD 102. After this fade-out, once the HMD 102 receives audio data 122 again and the buffer 120 begins to fill up again, normal operation of the HMD system can resume. Therefore, under normal operating conditions, the audio latency can be maintained within the target latency of 400, and when network conditions deteriorate beyond a certain point, auditory distortion and / or sudden silence can be mitigated.

[0048] Figure 5This is a schematic diagram illustrating a technique for mitigating auditory distortion that would otherwise be caused by lost data packets in an audio data stream. As described above, in some instances, the data packets carrying audio data 122 received by HMD 102 can be sequentially numbered, which can indicate the sequence of data packets. In this way, audio data management component 118 can be configured to analyze the received data packets to determine whether any data packets in the audio data stream are out of order or missing. For example, if audio data management component 118 determines, at least in part, that HMD 102 has not received lost data packets based on analysis of the audio data packets wirelessly received by HMD 102, then audio data management component 118 can create a synthetic audio data segment 500 based at least in part on the combination of discontinuous segments 124 of audio data 122 surrounding the lost data packets. Figure 5 In this instance, the lost data packet may have already carried audio data segments labeled #108 and #109 within a sequence of segments labeled #106-111. These lost audio data segments may represent audio data samples from the lost data packet. Therefore, the sequence of audio data segments 124(9), 124(10), 124(13), and 124(14) in buffer 120 may include segment 124 labeled #106, #107, #110, #111, etc. In this instance, segments 124(10) and 124(13) are non-contiguous and they enclose the lost data packet. Therefore, the non-contiguous segments 124(10) and 124(13) can be combined to create a composite audio data segment 500. Figure 5 Further examples illustrate that discontinuous segments 124(10) and 124(13) can be replaced with synthesized audio data segment 500 to obtain modified audio data 502 in buffer 120. Figure 5 In one instance, the modified audio data 502 in buffer 120 may include a new sequence of audio data segments comprising an original audio data segment 124(9), followed by a synthesized audio data segment 500, followed by another original audio data segment 124(14). In other words, gaps in the audio data stream created by the lost data packet can be closed by combining discontinuous segments 124(10) and 124(13) surrounding the lost data packet to create a synthesized audio data segment 500 used to replace Figure 5 The non-continuous segments 124(10) and 124(13) in the examples.

[0049] In some instances, the synthesized audio data segment 500 can be created by overlapping discontinuous segments 124(10) and 124(13) and crossfading the corresponding audio signal waveforms of the discontinuous segments 124(10) and 124(13). This technique can be similar to the one described above. Figure 2The techniques described in discontinuous segments 124(2) and 124(4) are as follows. For example, discontinuous segments 124(10) and 124(13) can overlap by a predetermined or dynamically determined amount of overlap 504. Furthermore, in order to crossfade the corresponding audio signal waveforms of discontinuous segments 124(10) and 124(13), by starting the crossfade function 506 at a value of 1 for one of the audio signal waveforms and sloping it to 0, and by starting the crossfade function 508 at a value of zero for the other of the audio signal waveforms and sloping it to 1, the metrics associated with the audio signal waveforms (e.g., voltage, power, etc.) can be maintained at constant values ​​across overlap 504. In some instances, windowing functions such as the Hann windowing function can be used to combine discontinuous audio data segments 124(10) and 124(13) to create a composite audio data segment 500. In some instances, the PSOLA algorithm can be used to combine discontinuous audio data segments 124(10) and 124(13) to create a synthesized audio data segment 500. In some instances, the processor 110 analyzes the corresponding audio signal waveforms to identify the zero-crossing points at the start of the crossfade functions 506 and 508. These techniques can help to “smooth” the transition between the combined discontinuous segments 124(10) and 124(13). Thus, any auditory distortion that would otherwise be caused by missing data packets is mitigated when the modified audio data 502 is processed for outputting the corresponding audio content via the speaker 116 of the HMD 102.

[0050] Figure 6 This is a flowchart of an exemplary process 600 for wirelessly streaming audio data to an HMD with low latency. For the purposes of discussion, process 600 is described with reference to the preceding figures.

[0051] At box 602, HMD 102 can receive data packets carrying audio data 122 from host computer 106. At box 602, HMD 102 wirelessly receives the audio data 122 within the data packets. For example, at box 602, HMD 102 can wirelessly receive the data packets via communication interface 112 (e.g., radio). In some instances, the data packets received at box 602 are sequentially numbered, which can indicate the sequence of data packets. In this way, audio data management component 118 can be configured to analyze the received data packets to determine if any data packets in the audio data stream are out of order or missing. In some instances, individual data packets received at box 602 can carry a single frame of audio data 122, or multiple consecutive frames of audio data 122. In some instances, as described herein, individual data packets can carry a specific amount of audio data 122.

[0052] At block 604, and in response to receiving a data packet at block 602, a processor (e.g., processor 110 of HMD 102) may execute audio data management component 118 to store audio data 122 in buffer 120 of HMD 102. In some instances, audio data 122 may be stored in buffer 120 as a sequence of audio data segments 124(1), 124(2), ..., 124(N) (collectively referred to as 124). Individual audio data segments 124 may represent audio samples, frames, data packets, or any other suitable segment of the received audio data 122. In some instances, audio data management component 118 may be configured to divide the received audio data 122 into segments 124, such as by dividing the audio data 122 into blocks or chunks using any suitable segmentation algorithm, as described herein.

[0053] At box 606, a determination is made regarding whether to adjust the latency associated with the received audio data 122 stored in buffer 120. In some instances, this determination at box 606 is made at least in part based on one or more latency statistics associated with the buffered audio data 122, as described herein. In some instances, the above references... Figure 4 The described thresholds 402, 404, and / or 406 can be used at box 606 to determine whether to adjust the audio wait time. If the audio data management unit 118 determines to adjust the audio wait time at box 606, then process 600 can follow a "yes" route from box 606 to box 608.

[0054] At block 608, a processor (e.g., processor 110) may execute audio data management unit 118 to modify audio data 122 stored in buffer 120 with synthesized audio data segments. For example, if the determination at block 606 is to reduce audio latency, audio data management unit 118 may use the above-referenced... Figure 2 The described technique uses synthesized audio data segment 200 to modify buffered audio data 122 to obtain modified audio data 202 in buffer 120. Conversely, if the determination at box 606 is to increase the audio latency, the audio data management component 118 can use the above reference. Figure 3 The described technique uses a synthesized audio data segment 300 (or multiple synthesized audio data segments) to modify buffered audio data 122 to obtain modified audio data 302 in buffer 120.

[0055] At block 610, the processor (e.g., processor 110) may output audio content via one or more speakers 116 of HMD 102, at least in part, based on modified audio data (e.g., modified audio data 202, 302). For example, at block 610, a series of audio data segments 124, including synthesized audio data segments 200, 300, may be processed from buffer 120 to output corresponding audio content via speakers 116. If the audio data management unit 118 determines at block 606 to suppress adjustment audio latency, process 600 may follow a "No" route from block 606 to block 612, at block 612, where the processor (e.g., processor 110) may output audio content via one or more speakers 116 of HMD 102, at least in part, based on unmodified audio data 122 stored in buffer 120. Therefore, during runtime, during a session involving HMD 102 (e.g., during a game session), procedure 600 can be executed to maintain the audio wait time within the target wait time of 400 by making a relatively small modification to the buffered audio data at box 608.

[0056] Figure 7 This is a flowchart of an exemplary process 700 for adjusting audio latency by modifying buffered audio data using synthesized audio data segments. For the purposes of discussion, process 700 is described with reference to the preceding figures.

[0057] At box 702, a processor (e.g., processor 110 of HMD 102) may execute audio data management component 118 to determine one or more statistics indicating the latency associated with audio data 122 wirelessly received by HMD 102 and stored in buffer 120 of HMD 102. The latency statistics determined at box 702 may be similar to those described above with respect to box 130 of process 128, and the latency statistics may be used to determine whether to adjust the size 126 of buffer 120, whether to increase or decrease the size 126 of buffer 120, and / or by how much to adjust the size 126 of buffer 120.

[0058] Therefore, at box 704, a determination is made regarding whether to adjust the latency associated with the received audio data 122 stored in buffer 120, based at least in part on the latency statistics determined at box 702. For example, a processor (e.g., processor 110) may execute audio data management unit 118 at box 704 to compare the latency statistics determined at box 702 with a target latency 400 and / or with thresholds 402, 404 that are less than or greater than the target latency 400. Based at least in part on this comparison, audio data management unit 118 may determine at box 704 whether to adjust the audio latency. If audio data management unit 118 determines at box 704 to suppress the adjustment of the audio latency, process 700 may follow a "No" route from box 704 to box 706, where the processor may wait for the start of the next statistical calculation period before re-determining the latency statistics at box 702. For example, if the latency statistics are determined at a rate of 2 Hz, the processor can wait at box 706 until the next 500-millisecond period begins, and then return to box 702. If the audio data management unit 118 determines to adjust the audio latency at box 704, then process 700 can follow a "yes" route from box 704 to box 708.

[0059] At box 708, a determination is made regarding whether to increase or decrease the audio latency. In some instances, a comparison of the latency statistic determined at box 702 with a target latency 400 and / or with thresholds 402, 404 that are less than or greater than the target latency 400 can indicate that the audio latency is “too low” or “too high” (e.g., outside the range of the target latency 400). Therefore, if the audio data management component 118 determines at box 708 to decrease the audio latency, process 700 can follow a “reduction” path from box 708 to box 710.

[0060] At box 710, a processor (e.g., processor 110) may execute audio data management unit 118 to create a synthesized audio data segment 200 based at least in part on a combination of discontinuous segments 124 of audio data 122 stored in buffer 120. For example, as Figure 2As illustrated, the audio data management component 118 can create a synthetic audio data segment 200 based at least in part on a combination of discontinuous segments 124(2) and 124(4). In some instances, the synthetic audio data segment 200 can be created by overlapping the discontinuous segments 124(2) and 124(4) and crossfading the corresponding audio signal waveforms of the discontinuous segments 124(2) and 124(4), as described above. For example, the discontinuous segments 124(2) and 124(4) can overlap by a predetermined or dynamically determined amount of overlap 204. Furthermore, in order to crossfade the corresponding audio signal waveforms of the discontinuous segments 124(2) and 124(4), by starting a crossfade function 206 at a value of 1 and sloping to 0 for one of the audio signal waveforms, and by starting a crossfade function 208 at a value of zero and sloping to 1 for the other of the audio signal waveforms, metrics (e.g., voltage, power, etc.) associated with the audio signal waveforms can be maintained at constant values ​​across overlap 204. In some instances, at box 710, a windowing function such as the Hann windowing function can be used to combine discontinuous audio data segments 124(2) and 124(4) to create a synthesized audio data segment 200. In some instances, at box 710, the PSOLA algorithm can be used to combine discontinuous audio data segments 124(2) and 124(4) to create a synthesized audio data segment 200. In some instances, the processor analyzes the corresponding audio signal waveforms at box 710 to identify the zero-crossing points at the start of the crossfade functions 206, 208. These techniques can help to “smooth” the transition between the combined discontinuous segments 124(2) and 124(4) to mitigate (e.g., hide, conceal, etc.) auditory distortion that would otherwise be caused by the modification of the buffered audio data 122 that occurs at box 712.

[0061] At block 712, the processor (e.g., processor 110) may execute audio data management unit 118 to replace one or more intermediate segments 124(3) of the audio data 122 between discontinuous segments 124(2) and 124(4) with synthesized audio data segment 200, to obtain modified audio data 202 in buffer 120. After block 712, process 700 may proceed to block 706, in which the processor may wait for the start of the next statistical calculation period before re-determining the wait time statistics at block 702. If audio data management unit 118 determines to increase the audio wait time at block 708, process 700 may follow an "increase" route from block 708 to block 714.

[0062] At box 714, a processor (e.g., processor 110) may execute audio data management unit 118 to create a synthesized audio data segment 300 based at least in part on a combination of discontinuous segments 124 of audio data 122 stored in buffer 120. For example, as Figure 3 As illustrated, the audio data management component 118 can create a synthesized audio data segment 300 based at least in part on a combination of discontinuous segments 124(6) and 124(8). In some instances, multiple synthesized audio data segments can be created at block 714 for larger-scale modifications to the buffered audio data 122. In some instances, the synthesized audio data segment 300 can be created by overlapping discontinuous segments 124(6) and 124(8) and crossfading the corresponding audio signal waveforms of the discontinuous segments 124(6) and 124(8), as described above. For example, the discontinuous segments 124(6) and 124(8) can overlap by a predetermined or dynamically determined amount of overlap 304. Furthermore, to crossfade the corresponding audio signal waveforms of the discontinuous segments 124(6) and 124(8), by starting the crossfade function 306 at a value of 1 and sloping it to 0 for one of the audio signal waveforms, and by starting the crossfade function 308 at a value of zero and sloping it to 1 for the other of the audio signal waveforms, the metrics associated with the audio signal waveforms (e.g., voltage, power, etc.) can be maintained at constant values ​​across overlap 304. In some instances, at block 714, a windowing function such as the Hann windowing function can be used to combine the discontinuous audio data segments 124(6) and 124(8) to create a composite audio data segment 300. In some instances, at block 714, the PSOLA algorithm can be used to combine the discontinuous audio data segments 124(6) and 124(8) to create a composite audio data segment 300. In some instances, the processor analyzes the corresponding audio signal waveforms at block 714 to identify the zero-crossing points at the start of the crossfade functions 306, 308. These techniques can help to “smooth” the transition between discontinuous segments 124(6) and 124(8) of the combination to mitigate (e.g., hide, conceal, etc.) auditory distortion that would otherwise be caused by the modification of the buffered audio data 122 that occurs at box 716.

[0063] At box 716, the processor (e.g., processor 110) may execute audio data management unit 118 to add synthesized audio data segment 300 to buffered audio data 122 (e.g., add to Figure 3In the sequence of segments 124(6)-(8) shown, modified audio data 302 is obtained in buffer 120. After block 716, process 700 can proceed to block 706, in which the processor can wait for the start of the next statistical calculation period before re-determining the wait time statistics at block 702. Therefore, process 700 can iterate to continuously and dynamically adjust the audio wait time to maintain the audio wait time within a specified wait time tolerance through iterative modification of the buffered audio data 122, which adjusts the size 126 of buffer 120.

[0064] Figure 8 This is a flowchart of an exemplary process 800 for maintaining audio latency within a target latency range and / or mitigating auditory distortion in audio content output via a speaker from an HMD using a threshold. For the purposes of discussion, process 800 is described with reference to the preceding figures.

[0065] At block 802, a processor (e.g., processor 110 of HMD 102) may execute audio data management component 118 to "fill" buffer 120 of HMD 102 with audio data 122 wirelessly received from host computer 106 up to the fill depth. For example, buffer 120 may initially be empty when user 104 powers on HMD 102. In some instances, at block 802, the processor (e.g., processor 110) may execute audio data management component 118 to "lock" buffer 120 until a threshold amount of audio data 122 (e.g., a threshold amount of audio data segments 124) is stored in buffer 120 (up to the fill depth). Once buffer 120 is "filled" to that fill depth, process 800 can proceed from block 802 to block 804.

[0066] At block 804, a processor (e.g., processor 110 of HMD 102) can at least partially base audio content on audio data 122 stored in buffer 120 to cause audio content to be output via one or more speakers 116 of HMD 102. For example, the processor can execute audio data management unit 118 to begin processing buffered audio data segments 124 in buffer 120 so that the corresponding audio content is output (or played) via speakers 116 of HMD 102.

[0067] At box 806, the question concerns whether the latency associated with the received audio data 122 stored in buffer 120 is outside the range of the target latency 400 (e.g., less than or equal to a low threshold 402, or greater than or equal to a high threshold 404, such as...). Figure 4The determination is made at box 806, as described herein, based at least in part on one or more latency statistics associated with the buffered audio data 122. If the audio data management component 118 determines that the audio latency is not outside the range of the target latency 400, process 800 may follow a "No" route from box 806 to box 804, in which the processor (e.g., processor 110) may continue to output audio content via one or more speakers 116 of the HMD 102, based at least in part on the audio data 122 stored in the buffer 120. It is worth noting that the buffer 120 is replenished with audio data 122 as more audio data 122 is wirelessly received by the HMD 102 and as audio content is played via the speakers 116. If the audio data management component 118 determines that the audio latency is outside the range of the target latency 400, process 800 may follow a "Yes" route from box 806 to box 808.

[0068] At block 808, a processor (e.g., processor 110) may execute audio data management component 118 to modify audio data 122 stored in buffer 120, thereby adjusting the audio latency up or down as needed. In some instances, at block 808, synthesized audio data segments (e.g., synthesized audio data segments 200, 300 described above) may be used to modify the buffered audio data 122.

[0069] At box 810, after modifying the buffered audio data 122 to adjust the audio latency up or down, a determination is made regarding whether the audio latency is equal to or greater than a critical threshold 406 that is less than or equal to the low threshold 402. In some instances, this determination at box 810 is made at least in part based on one or more latency statistics associated with the buffered audio data 122, as described herein. If the audio data management component 118 determines at box 810 that the audio latency is equal to or greater than the critical threshold 406, then process 800 may follow a "yes" route from box 810 to box 806, in which the determination is made regarding whether the audio latency is outside the range of the target latency 400 (e.g., less than or equal to the low threshold 402, or greater than or equal to the high threshold 404, such as...). Figure 4 (As depicted), a determination is made again. If the modification of the audio data 122 stored in buffer 120 works as expected to adjust the audio latency to within the range of the target latency 400, then process 800 can iterate over blocks 804 and 806. If the audio data management unit 118 determines at block 810 that the audio latency is less than the critical threshold 406, then process 800 can follow the "No" route from block 810 to block 812.

[0070] At block 812, a processor (e.g., processor 110) may attempt to allow the audio content to be “replayed” by iteratively outputting audio content via speaker 116 of HMD 102 for a predetermined number of times (e.g., twice, three times, etc.). For example, at least a portion of any remaining audio data 122 in buffer 120 may be processed to replay the audio content for the predetermined number of times. Additionally or alternatively, recently played audio data segment 124 may be maintained (e.g., in volatile memory, non-volatile memory, etc.) for a period of time and retrieved at block 812, such that the retrieved audio data segment 124 may be iteratively processed to replay the audio data for the predetermined number of times. If audio data 122 is received by HMD 102 during playback at block 812, buffer 120 may be “refilled” to a level that satisfies or exceeds the critical threshold 406. On the other hand, if network conditions have deteriorated or a system failure has occurred, causing HMD 102 to not receive additional audio data 122 during playback at box 812, the audio latency can be kept below the critical threshold 406.

[0071] Therefore, at box 814, after the audio content has been replayed a predetermined number of times at box 812, a determination is made again regarding whether the audio wait time is equal to or greater than the critical threshold 406. In some instances, as described herein, this determination at box 814 is based at least in part on one or more wait time statistics associated with the buffered audio data 122 (e.g., the remaining audio data 122 stored in buffer 120). If the audio data management component 118 determines at box 814 that the audio wait time is equal to or greater than the critical threshold 406, then process 800 may follow a "yes" route from box 814 to box 806, in which the determination is made regarding whether the audio wait time is outside the range of the target wait time 400 (e.g., less than or equal to the low threshold 402, or greater than or equal to the high threshold 404, such as...). Figure 4 (As depicted), a determination is made again. In some instances, process 800 may iterate over boxes 806-810 until the audio latency is equal to or greater than the low threshold 402 (i.e., within the range of the target latency 400), and in this scenario, process 800 may iterate over boxes 804 and 806. If the audio data management unit 118 determines at box 814 that the audio latency is less than the critical threshold 406, then process 800 may follow a "no" route from box 814 to box 816.

[0072] At block 816, the processor (e.g., processor 110) can implement a "last resort" technique of fading out the remaining audio content by gradually decreasing the volume of the audio content as it is output via speaker 116 of HMD 102. After this fade-out, process 800 can return to block 802, where, once the audio data 122 is wirelessly received again from host computer 106, the processor can execute audio data management unit 118 to "fill" buffer 120 with audio data 122 until the fill depth is reached. Thus, process 800 can be executed such that, under normal operating conditions, the audio latency can be maintained within the range of a target latency 400, and that auditory distortion and / or sudden silence can be mitigated when network conditions deteriorate beyond a certain point.

[0073] Figure 9 This is a flowchart of an exemplary process 900 for mitigating auditory distortion that would otherwise be caused by a stream of audio data with missing data packets. For the purposes of discussion, process 900 is described with reference to the preceding figures.

[0074] At box 902, HMD 102 can receive data packets carrying audio data 122 from host computer 106. At box 902, HMD 102 wirelessly receives the audio data 122 within the data packets. For example, at box 902, HMD 102 can wirelessly receive the data packets via communication interface 112 (e.g., radio). In some instances, the data packets received at box 902 are sequentially numbered, which can indicate the sequence of data packets. In some instances, a single data packet received at box 902 can carry a single frame of audio data 122, or multiple consecutive frames of audio data 122. In some instances, as described herein, a single data packet can carry a specific amount of audio data 122.

[0075] At box 904, and in response to receiving a data packet at box 902, a processor (e.g., processor 110 of HMD 102) may execute audio data management component 118 to store audio data 122 in buffer 120 of HMD 102. In some instances, audio data 122 may be stored in buffer 120 as a sequence of audio data segments 124(1), 124(2), ..., 124(N) (collectively referred to as 124). Individual audio data segments 124 may represent audio samples, frames, data packets, or any other suitable segment of the received audio data 122. In some instances, audio data management component 118 may be configured to divide the received audio data 122 into segments 124, such as by dividing the audio data 122 into blocks or chunks using any suitable segmentation algorithm, as described herein.

[0076] At block 906, a processor (e.g., processor 110) may execute audio data management unit 118 to analyze received data packets to determine if any data packets have been lost in the audio data stream. For example, if the data packets received at block 902 are sequentially numbered, and if sequentially received data packets skip the number of data packets in the sequence assigned to them, then audio data management unit 118 may determine that a data packet is lost. In this scenario, process 900 may follow a "yes" route from block 906 to block 908.

[0077] At block 908, a processor (e.g., processor 110) may execute audio data management unit 118 to create a synthesized audio data segment 500 based at least in part on a combination of discontinuous segments 124 of audio data 122 surrounding the lost data packet. Refer to the description above. Figure 5 In one instance, the lost data packet might have already carried audio data segments labeled #108 and #109 within a sequence of segments labeled #106-111. These lost audio data segments could represent audio data samples from the lost data packet. Therefore, the sequence of audio data segments 124(9), 124(10), 124(13), and 124(14) in buffer 120 could include segment 124 labeled #106, #107, #110, and #111, as... Figure 5As shown. In this example, segments 124(10) and 124(13) are discontinuous, and they enclose the missing data packet. Therefore, at box 908, the discontinuous segments 124(10) and 124(13) can be combined to create a synthetic audio data segment 500. In some instances, the synthetic audio data segment 500 can be created by overlapping the discontinuous segments 124(10) and 124(13) and crossfading the corresponding audio signal waveforms of the discontinuous segments 124(10) and 124(13), as described above. For example, the discontinuous segments 124(10) and 124(13) can be overlapped by a predetermined or dynamically determined amount of overlap 504. Furthermore, to crossfade the corresponding audio signal waveforms of the discontinuous segments 124(10) and 124(13), a crossfade function 506 is started at a value of 1 for one of the audio signal waveforms and scaled to 0, and a crossfade function 508 is started at a value of zero for the other of the audio signal waveforms and scaled to 1. This allows metrics associated with the audio signal waveforms (e.g., voltage, power, etc.) to be maintained at constant values ​​across overlap 504. In some instances, at box 908, a windowing function such as the Hann windowing function can be used to combine the discontinuous audio data segments 124(10) and 124(13) to create a composite audio data segment 500. In some instances, at box 908, the PSOLA algorithm can be used to combine the discontinuous audio data segments 124(10) and 124(13) to create a composite audio data segment 500. In some instances, the processor analyzes the corresponding audio signal waveform at block 908 to identify the zero-crossing points at the start of the crossfade functions 506, 508. These techniques can help “smooth” the transition between the discontinuous segments 124(10) and 124(13) of the combination to mitigate (e.g., hide, conceal, etc.) auditory distortion that would otherwise be caused by missing data packets.

[0078] At block 910, a processor (e.g., processor 110) may execute audio data management unit 118 to replace discontinuous segments 124(10) and 124(13) with synthesized audio data segment 500 to obtain modified audio data 502 in buffer 120. Following block 910, process 900 may proceed to block 912, in which the processor (e.g., processor 110) may, at least in part, based on the modified audio data 502, cause audio content to be output via one or more speakers 116 of HMD 102. For example, at block 912, a series of audio data segments 124, including synthesized audio data segment 500, may be processed from buffer 120 to output corresponding audio content via speakers 116. If the audio data management unit 118 determines at block 906 that no data packets are lost within the audio data stream, then process 900 may follow a "No" route from block 906 to block 914, in which the processor (e.g., processor 110) may, at least in part, output audio content via one or more speakers 116 of the HMD 102 based on unmodified audio data 122 stored in buffer 120. Therefore, process 900 may be executed at runtime, during a session involving the HMD 102 (e.g., during a gaming session), to mitigate auditory distortion that would otherwise be caused by lost (e.g., dropped, missing, etc.) data packets within the audio data stream.

[0079] Figure 10 Exemplary components of an HMD system 1000, which may implement the techniques disclosed herein, are illustrated according to embodiments thereof. In some instances, the HMD system 1000 may be a distributed system including an HMD 102 and one or more additional computers 1002 communicatively coupled to the HMD 102. Figure 10 In this context, the auxiliary computer 1002 may represent or otherwise include Figure 1 The host computer is 106. (For example...) Figure 1 As shown. Additionally or alternatively, the additional computer 1002 may represent or otherwise include a computer different from the host computer 106 but separate from the HMD 102. In some instances, the HMD system 1000 may be a standalone HMD 102, in which case, Figure 10 The components shown may be components of the HMD 102 itself.

[0080] HMD 102 may be implemented as a device to be worn by a user (e.g., on the head of user 104). In some embodiments, HMD 102 may be head-mounted, such as by allowing user 104 to secure HMD 102 to his / her head using a fixation mechanism (e.g., an adjustable strap) sized to surround user 104's head. In some embodiments, HMD 102 includes a VR, AR, or MR head-mounted display, which includes a near-eye or near-to-eye display. Therefore, the terms "wearable device," "wearable electronic device," "VR head-mounted display," "AR head-mounted display," and "head-mounted display (HMD)" are used interchangeably herein to refer to HMD 102. However, it should be understood that these types of devices are merely examples of HMD 102, and it should be understood that HMD 102 may be implemented with a variety of other form parameters.

[0081] In the illustrated specific implementation, the HMD system 1000 includes one or more processors 1004 and memory 1006 (e.g., a non-transitory computer-readable medium). The processor 1004 may be connected to... Figure 1 The processor 110 described herein is the same as or similar to that described herein, and / or the memory 1006 may be the same as that described herein. Figure 1 The memory 114 described herein is the same as or similar to that described elsewhere. For example... Figure 1 As shown. In some specific embodiments, processor 1004 may include CPU 1008, GPU 1010, both CPU 1008 and GPU 1010, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or in addition, the functions described herein may be executed at least in part by one or more hardware logic components. Examples of possible hardware logic components include, but not limited to, Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), System-on-a-Chip Systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc. Furthermore, each processor in processor 1004 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0082] Memory 1006 may include volatile and non-volatile memory, removable and non-removable media implemented by any method or technology for storing information, such as computer-readable instructions, data structures, program modules or other data. Such memory includes, but is not limited to, Random Access Memory RAM, Read-Only Memory ROM, Electrically Erasable Programmable ROM, Flash memory or other memory technologies, Compact Disk ROM (CD-ROM), Digital Versatile Disk DVD or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, Redundant Array of Independent Disks (RAID) storage systems, or any other medium that can be used to store desired information and can be accessed by a computing device. Memory 1006 may be implemented as Computer-Readable Storage Media (CRSM), which may be any available physical medium that the processor 1004 can access to execute instructions stored on memory 1006. In one basic implementation, the CRSM may include RAM and flash memory. In other implementations, the CRSM may include, but is not limited to, ROM, EEPROM, or any other non-transitory and / or tangible medium that can be used to store desired information and can be accessed by the processor 1004.

[0083] Typically, the HMD system 1000 may include logical components (e.g., software, hardware, and / or firmware, etc.) configured to implement the technologies, functionalities, and / or operations described herein. The memory 1006 is shown as including a variety of modules, such as instructions, data repositories, etc., which may be configured to execute on the processor 1004 to perform the technologies, functionalities, and / or operations described herein. Several exemplary functional modules are shown stored in the computer-readable medium 1006 and capable of executing on the processor 1004; however, the same functionality may alternatively be implemented in hardware, firmware, or as a system-on-a-chip (SoC) and / or other logical components.

[0084] For the benefit of other modules, the operating system module 1012 may be configured to manage the hardware within and coupled to the HMD 102 and / or HMD system 1000. Furthermore, in some cases, the HMD system 1000 may include one or more applications 1014 stored in memory 1006 or otherwise accessible by the HMD system 1000. In this specific embodiment, application 1014 includes game applications (e.g., VR game applications). However, the HMD system 1000 may include any number or type of applications and is not limited to the specific instance shown herein. Figure 1 The audio data management component 118 described herein may be stored in memory 1006 and configured to perform the techniques, functions, and / or operations described herein, such as by implementing one or more steps of processes 128, 600, 700, 800, and / or 900 described in detail above. Memory 1006 may further store buffer 120 described above, which may be used to implement the techniques, functions, and / or operations described herein.

[0085] Typically, the HMD system 1000 has an input device 1016 and an output device 1018. The input device 1016 may include control buttons. In some embodiments, one or more microphones may be used as the input device 1016 to receive audio input, such as user voice input. In some embodiments, one or more cameras or other types of sensors, such as inertial measurement units (IMUs), may be used as the input device 1016. For example, the IMU may be configured to detect head movements of the user 104, including for gesture input purposes. Other sensors, such as gyroscopes, accelerometers, magnetometers, color sensors or other motion sensors, positioning sensors, and orientation sensors, may be used as the input device 1016 to generate motion, positioning, and orientation data. Sensors may also include sub-parts of the sensor, such as a series of active or passive markers that can be viewed externally by a camera or color sensor to generate motion, positioning, and orientation data. For example, a VR headset may include multiple markers, such as reflectors or light (e.g., infrared or visible light), on its exterior. These markers can provide one or more reference points for software interpretation when viewed by an external camera or illuminated by light (e.g., infrared or visible light) to generate motion, positioning, and orientation data. Other sensors may include light sensors sensitive to light (e.g., infrared or visible light) projected or broadcast by base stations in the environment of the HMD 102. The IMU may be an electronic device that generates calibration data based on measurement signals received from accelerometers, gyroscopes, magnetometers, and / or other sensors suitable for detecting motion, correcting errors associated with the IMU, or some combination thereof. Based on the measurement signals, such motion-based sensors, such as IMUs, can generate calibration data indicating an estimated positioning of the HMD 102 relative to its initial position. For example, multiple accelerometers may measure translational motion (forward / backward, up / down, left / right), and multiple gyroscopes may measure rotational motion (e.g., pitch, yaw, and roll). For example, the IMU can rapidly sample measurement signals and calculate the estimated location of HMD 102 based on the sampled data. For instance, the IMU can integrate the measurement signals received from the accelerometer over time to estimate the velocity vector, and integrate the velocity vector over time to determine the estimated location of a reference point on HMD 102. A reference point is a point that can be used to describe the location of HMD 102. While a reference point is typically defined as a point in space, in a variety of embodiments, the reference point is defined as a point within HMD 102 (e.g., the center of the IMU). Alternatively, the IMU provides the sampled measurement signals to an external console (or other computing device) that determines calibration data.

[0086] The sensor used as input device 1016 can operate at a relatively high frequency to provide sensor data at a high rate. For example, sensor data can be generated at a rate of 1000 Hz (or one sensor reading per 1 millisecond). This results in one thousand readings per second. When the sensor generates so much data at this rate (or even higher), the dataset used to predict motion is quite large, even over relatively short time periods on the order of tens of milliseconds. As mentioned, in some embodiments, for the purpose of tracking the location and / or orientation, attitude, etc., of the HMD 102 in three-dimensional (3D) space, the sensor may include a light sensor sensitive to light emitted by a base station in the environment of the HMD 102. The calculation of location and / or orientation may be based on the timing characteristics of the light pulses and the presence or absence of light detected by the sensor.

[0087] In some embodiments, the additional input device 1016 may be provided in the form of a keyboard, keypad, mouse, touchscreen, joystick, etc. In other embodiments, the HMD system 1000 may omit the keyboard, keypad, or other similar mechanical input. In some instances, the input device 1016 may include control mechanisms such as basic volume control buttons for increasing / decreasing volume, as well as power and reset buttons.

[0088] Output device 1018 may include a display or display panel 1020 (e.g., a pair of stereoscopic display panels). Display panel 1020 may utilize any suitable type of display technology, such as an emitting display that uses light-emitting elements (e.g., light-emitting diodes (LEDs)) to present light emitted during frames on display panel 1020. As examples, display panel 1020 may include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an inorganic light-emitting diode (ILED) display, or any other suitable type of display technology for HMD applications. Output device 1018 may further include, but is not limited to, light elements (e.g., LEDs), a vibrator for generating tactile sensations, and the aforementioned speaker 119 of HMD 102. In some instances, speaker 116 is a 37.5 mm external balanced mode radiator (BMR) with a frequency response in the range of approximately 40 Hz and 24 kHz, an impedance of 6 ohms, and a sound pressure level (SPL) of 98.96 dBSPL at 1 cm. In some instances, the external speaker 116 is implemented as a sound bar, a speaker embedded in the main HMD body, and / or a speaker embedded in the headband of the HMD 102.

[0089] HMD system 1000 may include power supply 1022, such as one or more batteries. Additionally or alternatively, HMD 102 may include a power cable port for connection to an external power source via a wired device such as a cable.

[0090] The HMD system 1000 may further include a communication interface 1024, which can communicate with... Figure 1 The communication interface 112 described herein is the same as or similar to that described elsewhere. The communication interface 1024 may be or include a wireless unit coupled to an antenna to facilitate wireless connection to a network. Such a wireless unit may implement one or more of a wide variety of wireless technologies, such as Wi-Fi, Bluetooth, Radio Frequency (RF), etc. It should be understood that the HMD 102 may further include physical ports to facilitate wired connections to a network, connected peripheral devices (including computer 1002, such as a host computer that may be a PC, game console, etc.), or plug-in network devices communicating with other wireless networks.

[0091] HMD 102 may further include an optical subsystem 1026 that uses one or more optical elements to direct light from the electronic display panel 1020 to the user's eyes. The optical subsystem 1026 may include various types and combinations of different optical elements, including but not limited to apertures, lenses (e.g., Fresnel lenses, convex lenses, concave lenses, etc.), filters, etc. In some embodiments, one or more optical elements in the optical subsystem 1026 may have one or more coatings, such as an anti-reflective coating. The amplification of image light by the optical subsystem 1026 makes the electronic display panel 1020 physically smaller, lighter, and consumes less power compared to a larger display. Additionally, the amplification of image light increases the field of view (FOV) of the displayed content (e.g., an image). For example, the FOV of the displayed content allows the displayed content to be presented using almost the entire user's FOV (e.g., 120 to 150 degrees diagonally) and, in some cases, the entire user's FOV. AR applications may have a narrower FOV (e.g., approximately 40 degrees FOV). The optical subsystem 1026 may be designed to correct one or more optical errors, such as, but not limited to, barrel distortion, pincushion distortion, longitudinal chromatic aberration, lateral chromatic aberration, spherical aberration, coma, field curvature, astigmatism, etc. In some embodiments, the content provided to the electronic display panel 1020 for display is pre-distorted, and the optical subsystem 1026 corrects the distortion when it receives image light generated based on the content from the electronic display panel 1020.

[0092] HMD system 1000 may further include eye-tracking system 1028. A camera or other optical sensor within HMD 102 can capture image information of the user's eyes, and eye-tracking system 1028 can use the captured information to determine interpupillary distance, interocular distance, 3D positioning of each eye relative to HMD 102 (e.g., for distortion correction purposes), including the magnitude of torsion and rotation (i.e., roll, pitch, and yaw), and the gaze direction of each eye. In one example, infrared light is emitted within HMD 102 and reflected from each eye. The reflected light is received or detected by a camera in HMD 102 and analyzed to extract eye rotation from changes in the infrared light reflected by each eye. Many methods for tracking the eyes of user 104 can be used by eye-tracking system 1028. Therefore, the eye-tracking system 1028 can track up to six degrees of freedom for each eye (i.e., 3D positioning, roll, pitch, and yaw), and at least a subset of the tracked quantities can be combined from the user 104's two eyes to estimate the gaze point (i.e., the 3D position or orientation of the virtual scene the user is viewing). For example, the eye-tracking system 1028 may integrate information from past measurements, measurements identifying the positioning of the user 104's head, and 3D information describing the scene presented by the display panel 1020. Thus, information about the positioning and orientation of the user 104's eyes is used to determine the gaze point in the virtual scene presented by the HMD 102 that the user 104 is viewing.

[0093] HMD system 1000 may further include head tracking system 1030. Head tracking system 1030 may utilize one or more sensors to track the head movements of user 104, including head rotation, as described above. For example, head tracking system 1030 may track up to six degrees of freedom (i.e., 3D positioning, roll, pitch, and yaw) of HMD 102. These calculations may be performed at each frame in a series of frames, allowing application 1014 (e.g., a video game) to determine how to render the scene in the next frame and the audio data 122 to be generated based on head positioning and orientation. In some embodiments, head tracking system 1030 is configured to predict the future positioning and / or orientation of HMD 102 based on current and / or past data. This is because application 1014 is required to render a frame before user 104 actually sees the light on display 1020 (and thus sees the image). Therefore, the next frame may be rendered based on this future prediction of head positioning and / or orientation made at an earlier point in time, such as approximately 25-30 milliseconds before the frame is rendered. In a distributed system where a host computer 106 is communicatively (e.g., wirelessly) coupled to the HMD 102, future predictions of head pose can be made 30 milliseconds or more before the illumination time of a frame to account for network latency, compression operations, etc. Rotation data provided by the head tracking system 1030 can be used to determine both the direction of rotation of the HMD 102 and the amount of rotation of the HMD 102 in any suitable unit of measurement. For example, the direction of rotation can be simplified and output as a positive or negative horizontal direction and a positive or negative vertical direction, corresponding to left, right, up, and down. The amount of rotation can be expressed in degrees, radians, etc. Angular velocity can be calculated to determine the rotation rate of the HMD 102.

[0094] HMD system 1000 may further include controller tracking system 1032. Controller tracking system 1032 may utilize one or more sensors to track controller movement. For example, controller tracking system 1032 may track up to six degrees of freedom (i.e., 3D positioning, roll, pitch, and yaw) of a controller held in his / her hand by user 104. These calculations may be performed at each frame in a series of frames, allowing application 1014 (e.g., a video game) to determine how to render a virtual controller and / or virtual hand in the scene in the next frame based on the controller's positioning and orientation. In some embodiments, controller tracking system 1032 is configured to predict the future positioning and / or orientation of a handheld controller based on current and / or past data, similar to the description above regarding head tracking system 1030.

[0095] Although the subject matter has been described in structurally specific language, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features described. Rather, the specific features are disclosed as illustrative forms for implementing the claims.

Claims

1. A method comprising: The processor determines one or more statistics indicating the latency associated with audio data wirelessly received by the head-mounted display (HMD) and stored in the buffer of the HMD; The processor determines to reduce the size of the buffer based at least in part on one or more of the statistics. The processor replaces segments of the audio data with synthesized audio data segments to obtain modified audio data in the buffer; as well as The processor causes audio content to be output via one or more speakers of the HMD, based at least in part on the modified audio data.

2. The method of claim 1, further comprising the processor creating the synthesized audio data segment based at least in part on a combination of discontinuous segments of the audio data.

3. The method of claim 2, wherein replacing the segment comprises replacing one or more intermediate segments of the audio data between the discontinuous segments with the synthesized audio data segment to obtain the modified audio data in the buffer.

4. The method of claim 2, wherein the creation includes overlapping the discontinuous segments and cross-fading the corresponding audio signal waveforms of the discontinuous segments.

5. The method of claim 1, further comprising: The processor determines one or more second statistics indicating the latency associated with the second audio data received wirelessly by the HMD and stored in the buffer; The processor determines to increase the size of the buffer based at least in part on the one or more second statistics; The processor adds a second synthesized audio data segment to the second audio data to obtain second modified audio data in the buffer; as well as The processor, at least in part, bases the second modified audio data on the output of the second audio content via the one or more speakers.

6. The method of claim 5, further comprising the processor creating the second synthesized audio data segment based at least in part on a combination of discontinuous segments of the second audio data.

7. The method of claim 1, further comprising: The processor determines, at least in part, that the waiting time is greater than or equal to a threshold waiting time based on one or more statistics, wherein the threshold waiting time is greater than a target waiting time. The determination to reduce the size of the buffer is based at least in part on the waiting time being greater than or equal to the threshold waiting time.

8. The method of claim 1, further comprising: The processor determines one or more second statistics indicating the latency associated with the second audio data received wirelessly by the HMD and stored in the buffer; The processor determines, at least in part, based on the one or more second statistics, that the waiting time associated with the second audio data is less than a threshold waiting time; as well as The processor, based at least in part on at least a portion of the second audio data, causes the second audio content to be played back via the one or more speakers a predetermined number of times.

9. The method of claim 8, further comprising, after causing the second audio content to be replayed continuously a predetermined number of times: The processor determines one or more third statistics indicating the waiting time associated with the remaining audio data stored in the buffer; The processor determines, at least in part, based on the one or more third statistics, that the waiting time associated with the remaining audio data is less than the threshold waiting time; and The processor causes the remaining audio content to be output via the one or more speakers, at least in part, based on the remaining audio data, while gradually decreasing the volume of the remaining audio content as it is output.

10. A head-mounted display (HMD) system, comprising: HMD, the HMD including one or more speakers; One or more processors; as well as The memory stores computer-executable instructions that, when executed by the one or more processors, cause the HMD system to: Determine one or more statistics indicating the latency associated with audio data wirelessly received by the HMD and stored in the buffer of the HMD; The reduction in the size of the buffer is determined at least in part based on one or more of the aforementioned statistics; Replace the segments of the audio data with the segments of the synthesized audio data to obtain the modified audio data in the buffer; as well as Based at least in part on the modified audio data, the audio content is output via the one or more speakers.

11. The HMD system of claim 10, wherein the computer-executable instructions, when executed by the one or more processors, further cause the HMD system to create the synthesized audio data segment based at least in part on a combination of discontinuous segments of the audio data.

12. The HMD system of claim 11, wherein replacing the segment comprises replacing one or more intermediate segments of the audio data between the discontinuous segments with the synthesized audio data segment to obtain the modified audio data in the buffer.

13. The HMD system of claim 11, wherein creating the synthesized audio data segment includes overlapping the discontinuous segments and cross-fading the corresponding audio signal waveforms of the discontinuous segments.

14. The HMD system of claim 10, wherein the computer-executable instructions, when executed by the one or more processors, further cause the HMD system to: Determine one or more second statistics indicating the latency associated with the second audio data wirelessly received by the HMD and stored in the buffer; Based at least in part on the one or more second statistics, it is determined that the waiting time associated with the second audio data is less than a threshold waiting time; as well as Based at least in part on at least a portion of the second audio data, the second audio content is played back via the one or more speakers a predetermined number of times.

15. The HMD system of claim 14, wherein the computer-executable instructions, when executed by the one or more processors, further cause the HMD system to, after causing the second audio content to be played back a predetermined number of times: Determine one or more third statistics that indicate the waiting time associated with the remaining audio data stored in the buffer; Based at least in part on the one or more third statistics, it is determined that the waiting time associated with the remaining audio data is less than a threshold waiting time; as well as The remaining audio content is output through the one or more speakers, at least in part based on the remaining audio data, while the volume of the remaining audio content is gradually reduced as it is output.

16. A method comprising: The processor determines one or more statistics indicating the latency associated with audio data wirelessly received by the head-mounted display (HMD) and stored in the buffer of the HMD; The processor determines to increase the size of the buffer based at least in part on one or more of the statistics. The processor adds synthesized audio data segments to the audio data to obtain modified audio data in the buffer; as well as The processor causes audio content to be output via one or more speakers of the HMD, based at least in part on the modified audio data.

17. The method of claim 16, further comprising creating the synthesized audio data segment by the processor based at least in part on a combination of discontinuous segments of the audio data.

18. The method of claim 17, wherein the creation includes overlapping the discontinuous segments and cross-fading the corresponding audio signal waveforms of the discontinuous segments.

19. The method of claim 16, further comprising: The processor determines, at least in part, that the waiting time is less than a threshold waiting time, wherein the threshold waiting time is less than a target waiting time, based on one or more statistics. The determination to increase the size of the buffer is based at least in part on the waiting time being less than the threshold waiting time.

20. The method of claim 16, further comprising: The HMD receives a data packet carrying second audio data; The processor stores the second audio data in the buffer; The processor determines, at least in part, based on analysis of the data packets, that the HMD did not receive any lost data packets; The processor replaces the discontinuous segments surrounding the lost data packet in the second audio data with a second synthesized audio data segment to obtain the second modified audio data in the buffer; as well as The processor, at least in part, bases the second modified audio data on the output of the second audio content via the one or more speakers.

Images