Active fan noise reduction in head mounted displays

Abstract

Active noise reduction (ANR) techniques for reducing noise generated by a fan of a head mounted display (HMD) are described herein. An exemplary process may include receiving data indicative of noise generated by the fan, determining one or more audio parameter values based at least in part on the data and using a model, and outputting, via one or more extraaural speakers of the HMD, a sound having one or more audio characteristics based at least in part on the one or more audio parameter values to reduce the noise generated by the fan at an ear location of a user of the HMD.

Description

Technical Field

[0001] This application claims priority to U.S. patent application Ser. No. 17 / 941,363, filed Sep. 9, 2022. Application Ser. No. 17 / 941,363 is incorporated herein by reference in its entirety. Background Art

[0002] Head mounted displays (HMDs), such as virtual reality (VR) headsets, are used both within and outside of the video gaming industry. Some users run their headsets to their maximum performance, such as by throttling the central processing unit (CPU) and graphics processing unit (GPU), to achieve a high fidelity experience. When electronic components are running at or near maximum performance, they tend to get hot. Many VR headsets include fans to cool the electronic components when they get too hot, thereby preventing them from overheating.

[0003] For VR headsets with off-ear speakers, the noise generated by the headset's fan may be unpleasant to the headset user. In addition, the fan noise may interfere with the user's ability to hear the sound of audio content output via the off-ear speaker (such as the sound of a video game played by the user). For example, when the headset's fan is running, the fan may produce a humming, buzzing, or whirring sound that is audible to the headset user due to the headset's off-ear speaker.

[0004] This article provides technical solutions to improve and enhance these and other systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The detailed description is described with reference to the accompanying drawings. In the drawings, the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears. The use of the same reference numbers in different drawings indicates similar or identical components or features.

[0006] Figure 1 An exemplary HMD system is shown in which the techniques disclosed herein may be implemented for actively reducing noise generated by a fan of the HMD in accordance with embodiments disclosed herein.

[0007] Figure 2 is a flow chart of an exemplary process for actively reducing noise generated by a fan of an HMD according to embodiments disclosed herein.

[0008] Figure 3is a diagram illustrating an exemplary technique for using microphones of an HMD to actively reduce a noise-generating fan of the HMD relative to at least one of the microphones.

[0009] Figure 4 is a flow chart of an exemplary process for proactively reducing noise generated by a fan of an HMD in an energy efficient manner according to embodiments disclosed herein.

[0010] Figure 5 Exemplary components of an HMD system in which the techniques disclosed herein may be implemented are shown according to embodiments disclosed herein. DETAILED DESCRIPTION

[0011] When the user of the HMD needs a high-fidelity experience, the user can configure the HMD to run certain electronic components (e.g., CPU, GPU, etc.) at their maximum performance, or the HMD can be a high-performance HMD that always runs the electronic components at their maximum performance by default. Such HMDs can allow users to play "intense" video games with high-fidelity graphics, low latency, and / or other optimized performance parameters. As mentioned above, when the electronic components of the HMD run at or near their maximum performance, they tend to generate heat, which causes the electronic components to heat up. The temperature sensor of the HMD can sense the temperature of the electronic components that become too hot, which can trigger the fan of the HMD to turn on or otherwise start, so that the fan begins to cool the overheated electronic components. The fan can prevent the electronic components from overheating and shutting down and / or causing discomfort to the user. However, the fan also produces noise as a byproduct. For the external ear speakers of the HMD, this fan noise may be unpleasant to the user of the HMD, and it may interfere with the user's experience, such as by making it difficult for the user to hear the desired sound output by the external ear speakers (e.g., the sound of the video game played by the user via the HMD).

[0012] This document describes, among other things, active noise reduction (ANR) technology, and devices and systems that implement the technology, for reducing the noise generated by the fan of the HMD at the location of the ears of the user of the HMD. Although "ANR" is a term used throughout this disclosure, it should be understood that alternative terms such as "active noise control (ANC)", "noise cancellation (NC)", "noise control" and / or "noise reduction" can be used interchangeably with the ANR terms used herein. In addition, it should be understood that, in contrast to completely passive noise reduction techniques that use noise isolation materials (e.g., isolation members) instead of a power source, the ANR techniques described herein are at least partially "active" (e.g., using a power source for noise reduction). However, passive noise reduction techniques can be used in combination with the ANR techniques described herein.

[0013] When implemented on an HMD with one or more external ear speakers, the ANR technology described herein improves the audio experience of the user of the HMD by reducing the fan noise at the user's ear position. For illustration, the HMD can be worn by the user for the purpose of immersing the user in a VR environment or an augmented reality (AR) environment. An application (e.g., a video game) can be executed on the HMD, or on a host computer of an HMD system including the HMD. When the application is executed, the application generates pixel data and audio data for output on the HMD. The pixel data is used to display the video content of the application being executed in the form of a series of frames on the HMD, and the audio data is used to output the sound corresponding to the audio content of the application being executed via the speaker of the HMD.

[0014] The speakers of the HMD include "out-of-ear" speakers (sometimes referred to as "open-ear" speakers), meaning that the speakers do not cover the user's ears. Instead, the out-of-ear speakers leave the user's ears uncovered, such as by spacing the speakers from the ears a distance that allows the user to hear both the sound of the audio content corresponding to the application being executed and the sound of the user's surroundings. Users of VR headsets tend to use out-of-ear speakers to better experience the sound of VR games that reach the ears from different directions. If the speakers are implemented as on-ear speakers or earplugs, this effect will disappear. The out-of-ear speakers of the HMD also allow some depth cues / signals that the human ear is designed to pick up to be retained, thereby providing a more immersive audio experience than can be provided by on-ear speakers or earplugs. However, using out-of-ear speakers also means that users can still hear the sounds around them, including the noise from the fan of the HMD whenever the fan operates to cool the electronic components of the HMD. It is worth noting that the techniques, devices, and systems described herein are customized to reduce the specific noise generated by the fan of the HMD without reducing other noises around the user. In other words, the techniques described herein allow for the preservation of ambient noise at the location of the user's ears while isolating and reducing the specific noise generated by the fan of the HMD.

[0015] In an exemplary ANR process, a processor of the HMD system may receive data indicating noise generated by a fan of the HMD. The data received by the processor may be any suitable type of data that can be used as an agent to determine the characteristics of the fan noise at the ear position of the user of the HMD, as will be described in more detail below. Based at least in part on the received data and using the model, the processor may determine one or more audio parameter values, such as frequency parameter values, phase parameter values, and / or amplitude parameter values, etc. The determined audio parameter values ​​may then be used to output a sound having one or more audio characteristics (e.g., frequency, phase, and / or amplitude, etc.) via an external ear speaker of the HMD to reduce the noise generated by the fan at the ear position of the user of the HMD.

[0016] The ANR techniques described herein provide a more pleasant, higher-fidelity audio experience for users of HMDs with off-ear speakers by reducing (if not eliminating) fan noise at the location of the user's ears. This can allow the user to adjust the performance of their HMD components to their maximum performance without hearing the noise generated by the fans being used to cool the electronic components of the HMD (i.e., the fans are barely audible, if not inaudible, to the user of the HMD).

[0017] Also disclosed herein is a technique for actively reducing fan noise in an energy-saving manner, which saves power resources of an already power-short HMD and / or HMD system. For example, the ANR algorithm described herein can be executed on an as-needed basis, such as in response to determining that the fan is drawing power from a power source (e.g., a battery) of the HMD, and / or in response to determining that the fan is generating a level of noise that guarantees noise reduction. In other words, there may be fan noise at a level that most HMD users can tolerate, such as low-level fan noise. Therefore, even when the fan is operating, as long as the fan operates at or below a level that is tolerable for most users (the level may be a configurable level), power resources can be saved. In other examples, power saving techniques may involve determining whether and when to "turn on" the microphone of the HMD for performing ANR with improved accuracy, and retaining the use of the microphone of the HMD for use in ANR where the benefit of more accurate noise reduction exceeds those instances attributable to the power consumption cost of using the microphone for ANR.

[0018] Also disclosed herein is a system including an HMD configured to implement the techniques and processes disclosed herein, and a non-transitory computer-readable medium storing computer-executable instructions that implement the techniques and processes disclosed herein. Although the techniques and systems disclosed herein are often discussed in the context of video game applications, and in particular VR game applications, by way of example, it should be understood that the techniques and systems described herein can provide benefits for other applications, including but not limited to non-VR applications (e.g., AR applications) and / or non-game applications, such as industrial machine applications, defense applications, robotic applications, etc.

[0019] Figure 1 An exemplary HMD system 100 is shown in which the techniques disclosed herein can be implemented for actively reducing the noise generated by the fan of the HMD 102 according to the embodiments disclosed herein. In some examples, the HMD system 100 can include a stand-alone HMD 102 (sometimes referred to as an "all-in-one" HMD 102) that includes all of the components described herein and can operate without or with minimal assistance from a separate computer. In other examples, the HMD system 100 is a distributed system that includes the HMD 102 and at least one additional computer that is separate from the HMD 102 but communicatively coupled to the HMD. For example, the HMD system 100 can include a host computer 104 that is communicatively coupled to the HMD 102. In some examples, the host computer 104 can be collocated in the same environment as the HMD 102, such as in the home of a user 106 wearing the HMD 102. The host computer 104 and the HMD 102 can be communicatively coupled together wirelessly and / or via a wired connection. For example, the devices 102 / 104 may exchange data using Wi-Fi, Bluetooth, radio frequency (RF), and / or any other suitable wireless protocol. Additionally or alternatively, the devices 102 / 104 may include one or more physical ports to facilitate a wired connection (e.g., a tether, cable, etc.) for data transfer therebetween. In some examples, the HMD system 100 may include a remote system 108 in addition to or in lieu of a host computer 104 located in the environment of the HMD 102. The remote system 108 may be communicatively coupled to the host computer 104 and / or the HMD 102 via a wide area network 110 (such as the Internet). Thus, the remote system 108 may represent one or more server computers located in one or more remote geographic locations relative to the geographic location of the HMD 102. In other examples, the network 110 may represent a local area network (LAN), and although the remote system 108 is considered to be remote from the HMD 102, the remote system 108 may be located, for example, in the same building as the HMD 102.

[0020] The HMD 102 and the host computer 104 and / or the remote system 108 collectively represent a distributed system for executing an application 112 (e.g., a video game) to present associated video content (e.g., a series of images) on a display of the HMD 102 and / or output sounds corresponding to the audio content of the application being executed via one or more off-ear speakers 114 of the HMD 102. By being communicatively coupled together, the HMD 102 and the host computer 104 and / or the remote system 108 can be configured to work together in a collaborative manner to output such video content and / or audio content via the HMD 102. Thus, at least some of the components, programs, and / or data described herein, such as the processor 116, the application 112, etc., can reside on the host computer 104 and / or the remote system 108. Alternatively, as mentioned above, the components, programs, and / or data can reside entirely on the HMD 102, such as in a standalone HMD 102. Host computer 104 and / or remote system 108 may be implemented as any type of computing device and / or any number of computing devices, including but not limited to a personal computer (PC), a laptop computer, a desktop computer, a portable digital assistant (PDA), a mobile phone, a tablet computer, a set-top box, a game console, a server computer, a wearable computer (e.g., a smart watch, etc.), or any other electronic device that can send / receive data.

[0021] In some examples, HMD 102 may represent a VR headset for a VR system (such as for a VR gaming system). However, HMD 102 may additionally or alternatively be implemented as an AR headset for AR applications, or a headset that can be used for VR applications unrelated to gaming and / or AR applications (e.g., industrial applications). In AR, user 106 sees virtual objects overlaid on a real-world environment, while in VR, user 106 typically does not see the real-world environment, but is fully immersed in the virtual environment, as perceived via the display panel and optical device (e.g., lens) of HMD 102. It should be understood that in some VR systems, a pass-through imagery of the real-world environment of user 106 may be displayed in conjunction with a virtual image to create an enhanced VR environment in the VR system, whereby the VR environment is enhanced with real-world images (e.g., overlaid on a virtual world). The examples described herein primarily relate to a VR-based HMD 102, but it should be understood that HMD 102 is not limited to specific implementations in VR applications.

[0022] Figure 1The HMD 102 is depicted as including various components, although as mentioned, at least some components, programs, and / or data may reside elsewhere (e.g., on the host computer 104 and / or the remote system 108). These components may include, but are not limited to, an external speaker 114, one or more fans 118, a drive circuit 120, one or more sensors 122 (such as one or more temperature sensors 124), one or more microphones 126, one or more power sources 128, one or more processors 116 (such as one or more CPUs 130 and / or one or more GPUs 132), a memory 134 storing applications 112, an ANR component 136, and / or a model 138, among others.

[0023] The fan 118 may generate noise 140 during operation of the fan 118. Due to the "out-of-ear" speakers 114 of the HMD 102, the noise 140 may be heard by the user 106 while the application 112 is executing (e.g., while playing a video game). In the examples disclosed herein, the HMD 102 includes a pair of out-of-ear speakers 114, which include a first (e.g., left) out-of-ear speaker 114(1) and a second (e.g., right) out-of-ear speaker 114(2). For example, the first (e.g., left) out-of-ear speaker 114(1) may be located on the left side of the HMD 102, and the second (e.g., right) out-of-ear speaker 114(2) may be located on the right side of the HMD 102 (e.g., from the perspective of the user 106 wearing the HMD 102). Figure 3 Both the external ear speakers 114(1) and 114(2) are schematically shown in FIG. Figure 1An enlarged view of a second (e.g., right) off-ear speaker 114 (2) of the HMD 102 is shown. As the name implies, the "off-ear" speaker 114 does not cover the ear of the user 106. Instead, the off-ear speaker 114 can be spaced a corresponding distance from the ear of the user 106, leaving the ear uncovered. As mentioned, this allows the user 106 to hear both the sounds corresponding to the audio content of the executing application 112 and the sounds of the user's 106 surroundings, and the user 106 is able to adjust the distance that the off-ear speaker 114 is spaced from the ear to obtain a desired audio experience. However, without the ANR technology described herein, the use of the off-ear speaker 114 means that the user 106 can hear the noise 140 from the fan 118 whenever the fan 118 is operating to cool the electronic components of the HMD 102. It should be understood that an "off-ear" speaker as used herein means a speaker (or array of speakers) that does not cover the ear of the user / listener. Thus, the external ear speaker may be spaced (e.g., offset) a distance from the ear so that the speaker is near the ear, but does not touch or cover the ear. As another example, the external ear speaker may be disposed in (or fixed to) a headband of the HMD 102, and / or disposed in or on the main housing of the HMD 102 (e.g., inside the main housing of the HMD 102). Thus, it should be understood that an “external ear” speaker as used herein is not limited to a speaker that is proximate to the ear; such external ear speakers may be positioned at any suitable distance from the ear and / or at any suitable position relative to the ear, as long as the ear is not covered by the speaker and as long as the speaker does not block the ear canal.

[0024] The ANR component 136 may represent computer-executable instructions that, when executed by the processor 116, result in the performance of the operations and techniques described herein, such as executing an ANR algorithm for reducing fan noise 140 at the ear locations of the user 106. For example, the ANR algorithm may cause one or more off-ear speakers 114 to output a sound 142 having one or more audio characteristics (e.g., frequency, phase, amplitude, etc.) to reduce the noise 140 generated by the fan 118 at the ear locations of the user 106 of the HMD 102.

[0025] For example, when the ANR component 136 is executed by the processor 116, the processor 116 may receive data indicating the noise 140 generated by the fan 118. The data received by the processor 116 may be any suitable type of data that can be used as a proxy to determine the characteristics of the fan noise 140 at the ear location of the user 106 of the HMD 102. In some examples, the data received by the processor 116 includes audio data representing the noise 140 captured by the microphone 126 of the HMD 102. In some examples, the data received by the processor 116 may additionally or alternatively include non-audio data, such as temperature data indicating a temperature sensed by the temperature sensor 124 of the HMD 102, drive data indicating a level at which the fan 118 is driven via the drive circuit 120 of the HMD 102, utilization data indicating an amount of utilization of the CPU 130 and / or GPU 132, and / or any other suitable type of data, such as vibration data indicating vibrations generated by the fan detected by an accelerometer, operational data indicating an operating speed of the fan 118 determined by a tachometer, and the like. These are merely exemplary types of data that may be indicative of noise 140 generated by fan 118 .

[0026] For example, when an electronic component (e.g., CPU 130) begins to heat up and temperature sensor 124 senses an increased temperature of the electronic component, this may trigger fan 118 to begin operating to cool the electronic component. Thus, for example, the increased temperature sensed by temperature sensor 124 may indicate that fan 118 is operating, and thus generating noise 140. As another example, if fan 118 is driven at a certain level, and if drive data indicating the drive level is received by processor 116, such drive data indicates that fan 118 is operating, and thus generating noise 140. As yet another example, if the utilization of CPU 130 and / or GPU 132 suddenly increases, this may indicate that CPU 130 and / or GPU 132 is running at or near maximum performance, which may indicate that CPU 130 and / or GPU 132 is overheating or about to overheat, which in turn may indicate that fan 118 is operating to cool the electronic component, and thus generating noise 140.

[0027] The model 138 stored in the memory 134 may be referenced and used at runtime to determine one or more audio parameter values ​​based on the received data, which will be used to output a noise reduction sound 142 having one or more audio characteristics (e.g., frequency, phase, amplitude, etc.) to reduce the noise 140 generated by the fan 118 at the ear location of the user 106 of the HMD 102. Such a model 138 may represent any suitable type of model, such as a mathematical model, a statistical model, a weighted model, a trained machine learning model, etc. Therefore, depending on the type of model 138, the model 138 may be generated in various ways.

[0028] Any given fan 118 may generate noise within a certain tolerance range, depending on the level at which the fan 118 is driven, and due to inherent differences from one fan 118 to another. With this understanding, testing may be performed by operating each fan 118 within its respective operating range (e.g., from minimum input current to maximum input current), capturing the corresponding fan noise 140 using a microphone positioned at a position representing where the user's 106 ear is located when wearing the HMD 102, and recording the audio characteristics of the fan noise 140. Such heuristic testing may be performed repeatedly to calibrate the range of noise generated by the fan 118, as well as a range of head sizes and shapes (because the size and shape of the head varies among different user groups), a range of different positions of the fan 118 relative to the microphone, and / or a range of test environments with different ambient / background noise levels, etc. During such testing, the recorded audio characteristics of the fan noise 140 may be plotted as a function of the level at which the fan 118 is driven (e.g., input current). For example, the X-axis of the graph may represent the drive level of the fan 118 (e.g., milliamperes (mA) of input current), ranging from zero (e.g., the fan 118 is off) to N (e.g., the fan 118 is driven at a maximum level). At the same time, one or more audio characteristics of the fan noise 140 may be plotted on the Y-axis of the graph, where the audio characteristics include, but are not limited to, frequency, amplitude, phase, etc. In some examples, averages and / or other statistical values ​​may be calculated as part of the testing process, such as the average phase of the fan noise 140 recorded at the position of the user 106's ear relative to the fan 118. In some examples, the final audio parameter values ​​derived from the test may compensate for the delay of the microphone used to simulate the user's 106 ear during the test, thereby obtaining the audio characteristics of the fan noise 140 as the noise 140 will be heard by the human ear (as opposed to the noise 140 as "heard" by the microphone). In addition, the application 112 may be executed by the HMD system 100 under test while collecting and recording test data, such test data including, for example, the above-mentioned temperature data, drive data, utilization data, etc. The collected data may also be plotted as a function of the drive level of the fan 118. A model 138 may be generated based on such testing, and the model 138 may be used to determine audio parameter values ​​(e.g., frequency parameter values, phase parameter values, amplitude parameter values, pitch parameter values, pitch parameter values, signal-to-noise ratio (SNR) parameter values, etc.) based on one or more types of input data, which may produce a noise reduction sound 142 for reducing the fan noise 140 at the ear location of the user 106.

[0029] In examples where the model 138 is a trained machine learning model, test data collected during the above-described testing (e.g., by running the fan 118 of the HMD 102 within its operating range and recording various types of test data while the fan 118 is operating) can be used as training data to train the machine learning model 138. Additionally or alternatively, data collected by the HMD 102 when it is used by the end user 106 can be used as training data to train the machine learning model 138. Such a trained machine learning model 138 can be configured to output a classification of a noise reduction signal (e.g., according to its audio parameter values, such as frequency, amplitude, phase, etc.) for generating the noise reduction sound 142. The trained machine learning model 138 for reducing fan noise 140 can represent a single model or an ensemble of base-level machine learning models, and can be implemented as any type of machine learning model. For example, suitable machine learning models for use with the techniques and systems described herein include neural networks (e.g., deep neural networks (DNNs), recursive neural networks (RNNs), etc.), tree-based models, support vector machines (SVMs), kernel methods, random forests, splines (e.g., multivariate adaptive regression splines), hidden Markov models (HMMs), Kalman filters (or enhanced Kalman filters), Bayesian networks (or Bayesian trust networks), multilayer perceptrons (MLPs), expectation maximization, genetic algorithms, linear regression algorithms, nonlinear regression algorithms, classification models based on logistic regression, or sets thereof. An "ensemble" may include a collection of machine learning models whose outputs (predictions) are combined, such as by using a weighted average or voting. The individual machine learning models of the collection may differ in their expertise, and the collection may operate as a committee of individual machine learning models that is, as a whole, "smarter" than any individual machine learning model of the collection. In addition, the machine learning models may be trained using any suitable learning technique (such as supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, etc.).

[0030] Once the model 138 is generated and accessible to the HMD system 100 (e.g., stored in its memory 134), the model 138 can be used during runtime to reduce the fan noise 140. That is, the audio parameter values ​​(based on the received data and determined using the model 138) can be used to generate a noise reduction signal that causes a noise reduction sound 142 to be output via one or more off-ear speakers 114 of the HMD 102, the noise reduction sound 142 causing the fan noise 140 to be reduced (if not eliminated) at the location of the ears of the user 106. For example, using the exemplary HMD 102 described herein, a first (e.g., left) off-ear speaker 114(1) can output a first sound 142 having one or more first audio characteristics (e.g., frequency, amplitude, phase, etc.) based at least in part on the determined audio parameter values, and a second (e.g., right) off-ear speaker 114(2) can output a second sound 142 having one or more second audio characteristics (e.g., frequency, amplitude, phase, etc.) based at least in part on the determined audio parameter values. In this manner, the first sound 142 can reduce the noise 140 generated by the fan 118 at the first (e.g., left) ear location of the user 106, and the second sound 142 can reduce the noise 140 generated by the fan 118 at the second (e.g., right) ear location of the user 106. The noise reduction sound 142 can have the same amplitude as the amplitude of the fan noise 140, but with an opposite phase (e.g., due to a phase shift) and / or a zeroed frequency to reduce the unwanted fan noise 140. In some examples, the noise reduction sound 142 can be represented by a sound wave having an amplitude that is the same or proportional to the sound wave of the fan noise 140 and an opposite polarity (e.g., a waveform of opposite polarity). The noise reduction sound 142 described herein can be represented by a sound wave having any suitable audio characteristics that destructively interferes with the sound wave of the fan noise 140 at the ear location of the user 106 to reduce or attenuate the noise 140 generated by the fan 118 near the user's ear.

[0031] In some examples, the first sound 142 output via the first (e.g., left) extra-ear speaker 114(1) has different audio characteristics (e.g., different phase, different frequency, different amplitude, etc.) than the audio characteristics of the second sound 142 output via the second (e.g., right) extra-ear speaker 114(2). This may be because the fan noise 140 may have different audio characteristics at the corresponding positions of the left and right ears of the user 106 due to, for example, the asymmetry of the HMD 102 (e.g., the fan 118 may be located on one side (e.g., the left or right side) of the HMD 102, the tolerances of the extra-ear speakers 114(1) and 114(2) may be different, etc.), thereby generating a different noise 140 at the left ear than the noise 140 at the right ear (i.e., a noise 140 with different audio characteristics). In other examples, each extra-ear speaker 114 may output the same noise reduction sound 142 with the same audio characteristics.

[0032] It should be understood that the noise reduction sound 142 configured to reduce the fan noise 140 can be output by the external ear speaker 114 while the speaker 114 also outputs the sound corresponding to the audio content of the application 112 being executed. For example, the application 112 can represent a video game, and the user 106 can use the HMD 102 to play the video game, so that the noise reduction sound 142 can be output simultaneously with the sound of the video game. It is worth noting that the noise reduction sound 142 is not intended to reduce the sound corresponding to the audio content of the application 112 being executed. Instead, the noise reduction sound 142 is intended to reduce the specific noise 140 generated by the fan 118 of the HMD 102.

[0033] The processes described herein are illustrated as a collection of boxes in a logical flow diagram, which represent a sequence of operations that can be implemented in hardware, software, firmware, or a combination thereof (i.e., logic). In the context of software, a box represents a computer-executable instruction that, when executed by one or more processors, performs the operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described is not intended to be interpreted as a limitation, and any number of the described boxes can be combined in any order and / or in parallel to implement the process.

[0034] Figure 2 1 is a flow chart of an exemplary process 200 for actively reducing noise 140 generated by a fan 118 of an HMD 102 according to embodiments disclosed herein. For discussion purposes, the process 200 is described with reference to the previous figures. Furthermore, the process 200 may be implemented by the HMD system 100 described herein.

[0035] At 202, the processor 116 of the HMD system 100 may receive data indicative of noise 140 generated by a fan 118 of an HMD 102 of the HMD system 100. In some examples, the data received by the processor 116 at block 202 includes audio data representing the fan noise 140 captured by the microphone 126 of the HMD 102. In such examples, the audio data received at block 202 may be received from the microphone 126 and / or from other components that process the sound captured by the microphone 126. However, it should be understood that the data received at block 202 may include non-audio data, or be combined with audio data received from the microphone 126, or without audio data generated by such microphone. In examples where the data received at block 202 includes non-audio data, the non-audio data may be received from one or more other components of the HMD 102 (or HMD system 100) in addition to the microphone 126. For example, temperature data may be received from a temperature sensor 124 of the HMD 102 at block 202 , drive data may be received from a drive circuit 120 of the HMD 102 , and / or utilization data may be received from the processor 116 .

[0036] There may be costs and benefits to using the microphone 126 of the HMD 102 for ANR. Therefore, whether to use or not use the microphone 126 of the HMD 102 for ANR may depend on whether the cost of using the microphone 126 for ANR exceeds the benefits of doing so. For example, using audio data representing the fan noise 140 captured by the microphone 126 may allow the noise 140 to be reduced with improved accuracy compared to not using the microphone 126 for ANR. For example, using the microphone 126 for actively reducing the fan noise 140 may achieve measurably better noise reduction than performing ANR without using the microphone 126 (e.g., using non-audio data received from one or more other components). However, running the microphone 126 for ANR also requires power. Therefore, in some scenarios, if a marginal (e.g., 10% or less) improvement in noise reduction performance is obtained by using the microphone 126 for ANR compared to using non-audio data received from another component for ANR, it may not be worth spending extra power to run the microphone 126 for ANR. Thus, it should be understood that in some examples, the data received at block 202 may not include audio data. In these examples, the data received at block 202 may include, but is not limited to, the above-described temperature data indicating a temperature sensed by the temperature sensor 124 of the HMD 102, drive data indicating a level at which the fan 118 is driven via the drive circuit 120 of the HMD 102, and / or utilization data indicating an amount of utilization of the CPU 130 and / or GPU 132. These are merely exemplary types of data that may indicate the noise 140 generated by the fan 118, and other types of data indicative of fan noise 140 may be received at block 202.

[0037] At 204, the processor 116 may determine one or more audio parameter values ​​based at least in part on the data received at block 202 and using the model 138. For example, the processor 116 may determine audio parameter values, such as, but not limited to, frequency parameter values, phase parameter values, amplitude parameter values, etc. The model 138 used by the processor 116 to make this determination at block 204 may include, but is not limited to, a mathematical model, a statistical model, a weight model, a trained machine learning model, etc. These types of models 138 may be generated based on previously collected data associated with operating the fan 118 of the HMD 102 while the HMD system 100 is executing the application 112, as described above. For example, the HMD 102 with the fan 118 may be used as a test unit during heuristic testing to determine data (e.g., audio data, temperature data, drive data, utilization data, etc.) indicative of noise 140 generated by the fan 118 within its operating range (or a portion thereof), and the resulting mapping between the input data and the audio characteristics of the fan noise 140 may be reflected in the model 138, so that the processor 116 may determine the audio parameter value of the noise reduction signal from the new data received at block 202 and using the model 138. As another example, the machine learning model 138 may be trained using previously collected data associated with operating the fan 118 of the HMD 102 while the HMD system 100 is executing the application 112, and the trained machine learning model 138 may be used at block 204 to output the audio parameter value as a classification of the noise reduction signal (e.g., according to the classification of the audio parameter value of the noise reduction signal, such as frequency, amplitude, phase, etc.).

[0038] At 206 , sound 142 having one or more audio characteristics based at least in part on the audio parameter values ​​determined at block 204 may be output via the off-ear speaker 114 of the HMD 102 to reduce noise 140 generated by the fan 118 at the ear location of the user 106 of the HMD 102 . In a specific implementation of the HMD 102 having multiple extra-ear speakers 114, such as a first (e.g., left) extra-ear speaker 114(1) and a second (e.g., right) extra-ear speaker 114(2), the output at box 206 may involve, at box 208, outputting a first sound 142 having one or more first audio characteristics via the first (e.g., left) extra-ear speaker 114(1) to reduce the noise 140 generated by the fan 118 at the first (e.g., left) ear location of the user 106, and at box 210, outputting a second sound 142 having one or more second audio characteristics via the second (e.g., right) extra-ear speaker 114(2) to reduce the noise 140 generated by the fan 118 at the second (e.g., right) ear location of the user 106. In some examples, the audio characteristics of the sound 142 output at block 206 may include, but are not limited to, the frequency of the sound 142 (or sound wave), the amplitude of the sound 142 (or sound wave), the phase of the sound 142 (or sound wave), and other possible audio characteristics. In some examples, the first audio characteristics of the first sound 142 output by the first (e.g., left) external ear speaker 114 (1) may be different from the second audio characteristics of the second sound 142 output by the second (e.g., right) external ear speaker 114 (2). In some examples, the noise reduction sound 142 output at block 206 may be represented by a sound wave having any suitable audio characteristics that destructively interferes with the sound waves of the fan noise 140 at the location of the ear of the user 106 to reduce or attenuate the noise 140 generated by the fan 118 near the user's ear, as described herein.

[0039] Figure 3 is a diagram illustrating an exemplary technique for using the microphone 126 of the HMD 102 to actively reduce noise generated by the fan 118 of the HMD 102 (ie, a source of the unwanted noise 140 ). Figure 3A graphical representation of the HMD 102 is depicted as if looking at the front of a user 106 wearing the HMD 102 (i.e., looking at the front of the HMD 102). Thus, the HMD 102 is shown as having a pair of external ear speakers 114, including a first (e.g., left) external ear speaker 114(1) and a second (e.g., right) external ear speaker 114(2). The HMD 102 is also shown as having a pair of microphones 126, including a first (e.g., left) microphone 126(1) and a second (e.g., right) microphone 126(2). The first external ear speaker 114(1) and the first microphone 126(1) may be located on the left side of the HMD 102, and the second external ear speaker 114(2) and the second microphone 126(2) may be located on the right side of the HMD 102. For example, when the user 106 wears the HMD 102, the first microphone 126(1) may be positioned at or near the left cheekbone of the user 106, and when the user 106 wears the HMD 102, the second microphone 126(2) may be positioned at or near the right cheekbone of the user 106. These are merely exemplary locations for the microphones 126(1) and 126(2), and it should be understood that the microphones 126 may be positioned elsewhere on the HMD 102. It should also be understood that for microphone arrays having multiple microphones 126 with more than two microphones 126(1) and 126(2), additional microphones 126 may be included in the HMD 102. Figure 3 The pair of microphones 126(1) and 126(2) shown in the figure can be used as a dual microphone array, or a linear array of microphones 126, configured to capture audio data around them. In some examples, the individual microphones 126 are omnidirectional microphones, which means that the microphones 126 respond equally to sounds coming from any direction.

[0040] While one use of the microphone 126 of the HMD 102 may be to capture the sound of the user's 106 voice (e.g., to allow the user 106 to issue voice commands, and / or converse with other users wearing other HMDs, etc.), the microphone 126 of the HMD 102 may also be configured to generate audio data representative of the noise 140 generated by the fan 118 of the HMD 102 for use in the audio data in the ANR, as described herein. Figure 3In the example of , when the fan 118 generates noise 140, first audio data representing the noise 140 captured by the first microphone 126(1) can be generated. In addition, when the fan 118 generates noise 140, second audio data representing the noise 140 captured by the second microphone 126(2) can be generated. The first audio data (associated with the first microphone 126(1)) and the second audio data (associated with the second microphone 126(2)) can exhibit a phase difference during the time interval in which the microphones 126(1) and 126(2) receive or capture the fan noise 140. The phase difference can be used to find the direction of the noise source, which in this case is the fan 118. Therefore, the microphones 126(1) and 126(2) of the HMD 102 can be used as a phased array to locate the source of the unwanted fan noise 140.

[0041] Figure 3 A first instance 300(1) of noise 140 generated by a fan 118 detected by a first microphone 126(1) and a second instance 300(2) of noise 140 generated by a fan 118 detected by a second microphone 126(2) are shown. Due to the direction and / or position of the fan 118 relative to each microphone 126, the fan noise 140 may cause different audio data to be generated by each microphone 126. In some examples, the direction and / or position of the fan 118 relative to each microphone 126 can be described based on a relative azimuth 302 and a relative elevation 304 from either microphone 126(1) or 126(2), or from another location (e.g., from a midpoint between microphones 126(1) and 126(2)). The relative direction and / or position of the fan 118 to each microphone 126 of the HMD 102 can indicate an audio parameter value of the noise reduction signal, as described herein. That is, the model 138 may include or otherwise reflect the directionality and / or relative position of the fan 118 of the HMD 102 relative to the microphone 126, and the audio parameter values ​​(e.g., frequency parameter values, phase parameter values, and / or amplitude parameter values, etc.) determined at block 204 of the process 200—which may be used to output the noise reduction sound 142 via the off-ear speakers 114(1) and 114(2)—may vary depending on the direction and / or position of the fan 118 relative to the first microphone 126(1) and / or the second microphone 126(2). Thus, in some examples, at block 204 of the process 200, the directionality of the fan 118 may be taken into account in determining the audio parameter values.

[0042] It should also be understood that the HMD 102 may include multiple fans 118, and in such examples, the respective directions of individual fans in the multiple fans 118 may be taken into account in the ANR techniques described herein. In such examples, each fan 118 may be considered an independent source of unwanted noise, and the ANR techniques described herein may be performed independently with respect to the individual fans in the fans 118. Alternatively, the model 138 may be used to output a noise reduction sound 142 as a collaborative effort to reduce the multiple different noises 140 generated by the multiple different fans 118 of the HMD 102. In some examples, the multiple fans 118 may be located on a head strap 107 of the HMD 102, such as behind the head strap 107.

[0043] Figure 3 A third instance 302(1) of the noise 140 generated by the fan 118 when the fan 118 emits sound at a first (e.g., left) ear of the user 106 near the first (e.g., left) external-ear speaker 114(1) and a fourth instance 302(2) of the noise 140 generated by the fan when the fan 118 emits sound at a second (e.g., right) ear of the user 106 near the second (e.g., right) external-ear speaker 114(2) are shown. Notably, the third instance 302(1) of the fan noise 140 is different from the first instance 300(1) of the fan noise 140. This may be due to the first microphone 126(1) being at a different location from the first (e.g., left) ear of the user 106. Similarly, the fourth instance 302(1) of the fan noise 140 is different from the second instance 300(2) of the fan noise 140, which may be due to the second microphone 126(2) being at a different location from the second (e.g., right) ear of the user 106. As described above, unit-by-unit testing may be performed to characterize what fan noise 140 sounds like at the location of the ear based on audio data captured by microphone 126, which may not be located at the same location as the ear, and the results of such testing may be used to generate model 138. At runtime, model 138 may be used to infer audio parameter values ​​of a noise reduction signal based on the audio data captured by microphone 126 to reduce the unwanted fan noise 140 at the location of the user's ear.

[0044] In some examples, HMD 102 may include microphone 126(3), such as microphone 126(3) disposed within HMD 102 (e.g., within a main housing of HMD 102). Figure 3Another example 300(3) of noise 140 generated by fan 118 detected by microphone 126(3) is shown. In some examples, HMD 102 includes microphone 126(3) instead of microphones 126(1) and 126(2). In other examples, HMD 102 includes a third microphone 126(3) in addition to microphones 126(1) and 126(2). Microphone 126(3) can be strategically positioned within HMD 102 for capturing fan noise 140. In some examples, microphone 126(3) is dedicated to capturing fan noise 140, as opposed to having dual functionality as a microphone that can be used to capture both user voice and fan noise 140, as described above with respect to microphones 126(1) and 126(2). In some examples, microphone 126(3) is positioned in close proximity to fan 118 (e.g., within a threshold distance from the fan) (e.g., a few millimeters, a few centimeters, etc. from fan 118) to optimize microphone 126(3)'s capture of fan noise 140. In any case, at runtime, model 138 can be used to infer audio parameter values ​​of a noise reduction signal based on audio data captured by microphone 126(3) to reduce unwanted fan noise 140 at the user's ear location.

[0045] In some examples, the ANR techniques described herein can utilize any suitable feedback and / or feedforward noise reduction techniques. Feedback noise reduction techniques may refer to, for example, techniques for providing feedback for determining the effectiveness of the ANR algorithm using one or more feedback sensors (e.g., microphone 126 of HMD 102, one or more additional microphones positioned at or near the position of the external ear speaker 114 of HMD 102, etc.). Feedforward noise reduction techniques may refer to, for example, adaptive or dynamic equalization techniques. In some examples, the audio data representing the fan noise 140 captured by the microphone 126 of HMD 102 may be decomposed into frequency space, such as by calculating Fourier transform, and the resulting power spectrum may be analyzed to determine the main vibration frequency band. Taylor series expansion may be used to simplify the main vibration frequency band into sine terms and cosine terms, each of which has phase, frequency and amplitude. The least squares equation or any other suitable optimization equation may then be used to determine the variables of the phase, frequency and / or amplitude (i.e., audio characteristics) of the optimized noise reduction sound 142. This is an example of a feed-forward ANR technique, and it should be understood that any other suitable ANR techniques and / or mechanisms may be used for ANR, such as feed-forward filters (e.g., equalizers, such as adaptive equalizers, dynamic equalizers, or feedback equalizers, including finite impulse response (FIR) filters, infinite impulse response (IIR) filters, etc.), feedback filters, fixed filters, programmable filters, programmable filter controllers, amplifiers, speed-to-noise converters, and / or any other suitable techniques and / or mechanisms. The feed-forward ANR technique may allow the audio parameter values ​​of the noise reduction signal to be dynamically adjusted in real time based on input data (e.g., data received at box 202 of process 200), so that the noise reduction sound 142 output via the external ear speaker 114 converges to audio characteristics that optimize the noise reduction performance of the HMD system 100.

[0046] Figure 4 1 is a flow chart of an exemplary process 400 for proactively reducing noise 140 generated by a fan 118 of an HMD 102 in an energy-efficient manner according to embodiments disclosed herein. For discussion purposes, the process 400 is described with reference to the previous figures. In addition, the process 400 can be implemented by the HMD system 100 described herein.

[0047] At 402, the processor 116 of the HMD system 100 may determine whether the fan 118 of the HMD 102 of the HMD system 100 is on or off. As used herein, the fan 118 being “on” means that the fan 118 is operating and, therefore, generating noise 140. In some examples, the determination at block 402 may be made by determining whether the fan 118 is drawing power from a power source 128 (e.g., a battery) of the HMD 102. If the fan 118 is not on at block 402 (e.g., if the fan 118 is not drawing power from the power source 128 of the HMD 102), the process 400 may follow a “no” route from block 402 to iterate the determination at block 402 (e.g., by continuously monitoring the on / off state of the fan 118). In other words, if the fan 118 is not operating, the processor 116 can determine to avoid executing the ANR algorithm in order to save energy of the HMD system 100 (e.g., to save battery power) because performing the ANR techniques described herein requires resources, including power resources. Therefore, if the fan 118 is not operating, energy can be saved by avoiding performing ANR and by continuing to monitor the on / off state of the fan 118. In response to determining that the fan 118 is on (e.g., in response to determining that the fan 118 is drawing power from the power source 128 of the HMD 102), the process 400 can follow the "yes" route from box 402 to box 404.

[0048] At 404, the processor 116 may determine whether the fan 118 is driven at a level that satisfies a threshold level. As used herein, "satisfying" a threshold may mean that a value is equal to or greater than a threshold, or it may mean that the value is strictly greater than a threshold. In addition, the drive level and threshold value evaluated at box 404 may be determined according to any suitable metric, such as an index level, a percentage of a maximum drive level, milliamperes of input current, and / or any other suitable metric. For example, the threshold value evaluated at box 404 may be set to 10% of the maximum drive level of the fan 118. In this example, if the threshold value is not met, meaning that the fan 118 is driven at a level less than (or equal to) 10% of the maximum drive level of the fan 118, the process 400 may follow the "no" route from box 404 to iterate boxes 402 and 404, such as by continuously monitoring the on / off state and drive level of the fan 118. For example, when the fan 118 is driven at a relatively low drive level, the noise 140 generated by the fan 118 may be tolerable for a typical HMD user, and in such a scenario, it may not be worth expending additional power to execute the ANR algorithm to reduce the fan noise 140. Therefore, as long as the fan 118 is turned off or driven at a level that does not meet the threshold at block 404, the processor 116 can avoid executing the ANR algorithm. Therefore, the noise 140 generated by the fan 118 may not be actively reduced in order to save energy of the HMD system 100. On the other hand, if the processor 116 determines at block 404 that the fan 118 is driven at a level that meets the threshold, the process 400 can follow the "yes" route from block 404 to block 406 to start executing the ANR algorithm. In other words, when following the "yes" route from block 404, the processor 116 can determine to execute the ANR algorithm to reduce the fan noise 400, at the expense of utilizing some resources, such as power resources. It should be appreciated that this (ie, following the "yes" path from block 404) may also trigger Figure 2 The process 200 is an example of an ANR algorithm. Figure 4 As shown in FIG. 4 , the process 400 after block 404 is Figure 2 The ANR algorithm of process 200 is described in more detail as an example of an ANR algorithm.

[0049] In some examples, before proceeding to block 406, the user 106 may be prompted to approve the execution of the ANR algorithm. That is, the processor 116 may cause an output device (e.g., display, speaker 114, etc.) of the HMD 102 to output a prompt for the user 106 to approve the execution of the ANR algorithm before executing the ANR algorithm. The prompt may be output in any suitable format (e.g., a pop-up notification on the display of the HMD 102, an auditory prompt output via the speaker 114, etc.) and may present any suitable type of message (e.g., a message that reads: "Do you want your headset to actively reduce the noise of the fan? Yes or No?"). If the user 106 approves the execution of the ANR algorithm (e.g., by providing a user input indicating the "yes" option of the prompt), the process 400 may proceed to block 406 to begin executing the ANR algorithm. Otherwise, if the user 106 does not provide his / her approval, the process 400 may not execute the ANR algorithm from block 406 forward after the prompt.

[0050] At 406, before receiving input data used in the ANR algorithm, the processor 116 can determine whether to use or avoid using the microphone 126 of the HMD 102 for ANR. In other words, the processor 116 can determine whether to use the power source 128 of the HMD 102 to power the microphone 126 for the purpose of using the microphone 126 in the ANR algorithm. Because "turning on" the microphone 126 and capturing audio data that can be used in the ANR technology described herein requires additional power, the determination at box 406 can be another power saving determination, which can be based on any suitable data or information. For example, if the remaining / stored / available power of the power source 128 fails to meet (e.g., is less than (or equal to)) a threshold power level, then the determination at box 406 can be a determination to avoid using the microphone 126 for ANR in order to save the remaining / stored / available power so that the power can be used for something other than powering the microphone 126 for ANR. As another example, the user 106 may specify a user preference in the settings of the HMD system 100 regarding whether the user 106 wants to use the microphone 126 for ANR, and the determination at block 406 may be based at least in part on the user preference in the settings of the HMD system 100. As yet another example, the level at which the fan 118 is driven may determine whether the microphone 126 is used for ANR. For example, when the fan 118 is driven at a relatively low level (e.g., if the drive level of the fan fails to meet a second threshold greater than the threshold evaluated at block 404), the determination at block 406 may be to avoid using the microphone 126 for ANR, and if the second threshold is met, the determination at block 406 may be to utilize the microphone 126 for ANR. As yet another example, at block 406, the predicted ANR performance gain may be evaluated for the remaining power level available from the power source 128 of the HMD 102 to determine whether to use the microphone 126 for ANR. For example, if one or more criteria are met, such as the amount of remaining power meets a threshold and the predicted ANR performance gain of using the microphone 126 for ANR also meets a threshold, then the determination at block 406 may use the microphone 126 for ANR, and if one or more criteria are not met, then the determination at block 406 may avoid using the microphone 126 for ANR. In any case, if the processor 116 determines to avoid using the power source 128 of the HMD 102 to power the microphone 126 of the HMD 102 for the purpose of using the microphone 126 in the ANR algorithm, then the process 400 may follow the "no" route from block 406 to block 408.

[0051] At 408, the processor 116 may receive non-audio data indicating noise 140 generated by the fan 118 of the HMD 102. As described herein, the non-audio data may be any suitable type of data, such as, but not limited to, temperature data received from the temperature sensor 124 of the HMD 102, drive data received from the drive circuit 120 of the HMD 102, and / or utilization data regarding utilization of the CPU 130, GPU 132, and / or any other suitable non-audio data. In these examples, the temperature data may indicate a temperature sensed by the temperature sensor 124 of the HMD 102, the drive data may indicate a level at which the fan 118 is driven via the drive circuit 120 of the HMD 102, and / or the utilization data may indicate an amount of utilization of the CPU 130, GPU 132, and / or another electronic component. These are merely exemplary types of non-audio data that may be received at block 408 and may indicate noise 140 generated by the fan 118. It should also be understood that a variety of different types of non-audio data may be received at block 408.

[0052] At 410, the processor 116 may determine one or more audio parameter values ​​for the noise reduction signal based at least in part on the non-audio data received at block 408 and using the model 138. Examples of audio parameter values ​​determined at block 410 are described elsewhere herein and may include one or more audio parameter values ​​described with reference to block 204 of process 200.

[0053] At 412, a sound 142 having one or more audio characteristics based at least in part on the audio parameter values ​​determined at block 410 may be output via an off-ear speaker 114 of the HMD 102 to reduce the noise 140 generated by the fan 118 at the ear location of the user 106 of the HMD 102. Additionally, the noise reduction sound 142 output at block 412 may be one or more first sounds of a plurality of different sounds simultaneously output via the off-ear speaker 114. For example, the HMD system 100 may be executing an application 112 (e.g., a video game being played by the user 106 using the HMD 102). Therefore, at block 412, the off-ear speaker 114 may be outputting one or more second sounds corresponding to the audio content of the executing application 112 (e.g., the audio content of the video game) while outputting the first noise reduction sound 142. However, because the first noise reduction sound 142 destructively interferes with the fan noise 140 at the ear location of the user 106, the user 106 cannot hear the first noise reduction sound 142. Instead, user 106 hears the sounds corresponding to the audio content of executing application 112 (eg, the audio content of a video game), as well as any other ambient noise surrounding user 106 , excluding fan noise 140 .

[0054] In a specific implementation of the HMD 102 having multiple external ear speakers 114, such as a first (e.g., left) external ear speaker 114(1) and a second (e.g., right) external ear speaker 114(2), the output at block 412 may involve, at block 414, outputting a first sound 142 having one or more first audio characteristics via the first (e.g., left) external ear speaker 114(1) to reduce noise 140 generated by the fan 118 at a first (e.g., left) ear location of the user 106, and at block 416, outputting a second sound 142 having one or more second audio characteristics via the second (e.g., right) external ear speaker 114(2) to reduce noise 140 generated by the fan 118 at a second (e.g., right) ear location of the user 106. In some examples, the audio characteristics of the sound 142 output at block 412 may include, but are not limited to, a frequency of the sound 142 (or sound wave), an amplitude of the sound 142 (or sound wave), and a phase of the sound 142 (or sound wave). In some examples, the first audio characteristics of the first sound 142 output by the first (e.g., left) off-ear speaker 114(1) at block 414 may be different from the second audio characteristics of the second sound 142 output by the second (e.g., right) off-ear speaker 114(2) at block 416. In some examples, the noise reduction sound 142 output at block 412 may be represented by a sound wave having any suitable audio characteristics that destructively interferes with the sound waves of the fan noise 140 at the location of the ear of the user 106 to reduce or attenuate the noise 140 generated by the fan 118 near the user's ear.

[0055] In some examples, the HMD 102 may include multiple external ear speakers 114 that are positioned at different locations relative to the ears of the user 106. For example, the multiple external ear speakers 114 may surround the right ear of the user 106 and / or the left ear of the user 106. Therefore, the HMD 102 may be configured to output sound from any individual external ear speaker 114 at any appropriate time, such as outputting sound via the external ear speaker 114 positioned behind the right ear of the user and / or via the external ear speaker 114 positioned in front of the left ear of the user, etc. For example, a speaker array may be implemented in or on the headband of the HMD 102, with some external ear speakers 114 positioned behind the ears, some external ear speakers 114 positioned in front of the ears, and some external ear speakers 114 aligned laterally with the ears. In some examples, the external ear speakers 114 may be disposed within the HMD 102 (e.g., within the main housing of the HMD 102). In these or other similar implementations of the HMD 102, at block 412, the processor 116 may determine which of the plurality of external ear speakers 114 outputs the noise reduction sound 142. For example, at block 412, the processor 116 may select a first external ear speaker 114 positioned in front of the left ear of the user 106, and may select a second external ear speaker 114 positioned behind the right ear of the user 106, and the processor 116 may cause the noise reduction sound 142 to be output via the selected external ear speakers 114. Thus, the position relative to the ear from which the noise reduction sound 142 is emitted from the selected external ear speakers 114 may be dynamically determined in the ANR techniques described herein.

[0056] Returning to decision block 406 , if the processor 116 determines to power the microphone 126 of the HMD 102 using the power source 128 of the HMD 102 for purposes of using the microphone 126 in the ANR algorithm, the process 400 may follow a “yes” route from block 406 to block 418 .

[0057] At 418, power may be supplied to the microphone 126 of the HMD 102 for use in the ANR algorithm. This is sometimes referred to as "turning on" the microphone 126 to capture the sounds in its environment. In some examples, at box 418, a subset of the microphones 126 may be selected for use in the ANR algorithm, and power may be supplied specifically to the selected subset of the microphones 126. For example, at box 418, the processor 116 may select the first microphone 126(1) and may supply power to the first microphone 126(1) without supplying power to the second microphone 126(2), or vice versa. This is merely one example of how a subset of the microphones 126 may be selected and supplied with power. In other examples, at box 418, power may be supplied to the entire set of microphones 126 of the HMD 102.

[0058] At 420, the processor 116 may receive audio data representing the fan noise 140 captured by the powered microphone 126 of the HMD 102. This is because the fan 118 is operating at or above the threshold drive level (e.g., following the "yes" route from block 404), and is therefore generating the noise 140. The fan noise 140 is also audible near the powered microphone 126, which allows the microphone 126 to capture the fan noise 140 and generate audio data based on the captured fan noise 140. As shown by the dashed arrow from block 420 to block 408, in addition to receiving audio data at block 420, the process 400 may also include receiving non-audio data at block 408.

[0059] At 424, processor 116 may generate a signal based at least in part on the audio data received at block 420 (and potentially on the non-audio data received at block 408, as described by Figure 4424) and using model 138, determine one or more audio parameter values, such as those described elsewhere herein, including those described with reference to block 204 of process 200. In some examples, processor 116 may use first microphone 126(1) and second microphone 126(2) to determine audio characteristics (e.g., phase, amplitude, etc.) of a waveform corresponding to noise 140 generated by fan 118, and may determine an audio parameter value based at least in part on the audio characteristics of the waveform at block 424. For example, multiple data points of the audio characteristics may be determined over time and may be used to reconstruct a complex amplitude of a waveform corresponding to fan noise 140, which provides processor 116 with spatial information (e.g., directionality) associated with the source of noise 140, and which allows processor 116 to infer how the fan noise sounds at the location of the ear. Following block 424 , the noise canceling sound 142 may be output via the off-ear speaker 114 of the HMD 102 at block 412 , as described above.

[0060] Figure 5 1 shows exemplary components of an HMD system 100 in which the techniques disclosed herein may be implemented according to embodiments disclosed herein. As mentioned above, the HMD system 100 may be a distributed system including an HMD 102 and one or more additional computers 500 communicatively coupled to the HMD 102. Figure 5 In the example, the additional computer 500 may represent Figure 1 The host computer 104 and / or remote system 108. Alternatively, the HMD system 100 can be a stand-alone HMD 102.

[0061] The HMD 102 may be implemented as a device to be worn by the user 106 (e.g., worn on the head of the user 106). In some embodiments, the HMD 102 may be head-mounted, such as by allowing the user 106 to secure the HMD 102 on his / her head using a securing mechanism (e.g., an adjustable band) sized to fit around the head of the user 106. In some embodiments, the HMD 102 includes a VR or AR headset that includes a near-eye or near-to-eye display. Therefore, the terms "wearable device", "wearable electronic device", "VR headset", "AR headset", and "head-mounted display (HMD)" may be used interchangeably herein to refer to the device 102. However, it should be understood that these types of devices are merely examples of the HMD 102, and it should be understood that the HMD 102 may be implemented in various other form factors.

[0062] In the illustrated implementation, the HMD system 100 includes Figure 1 134 (e.g., computer readable medium 134). In some specific implementations, processor 116 may include CPU 130, GPU 132, both CPU 130 and GPU 132, microprocessor, digital signal processor, or other processing unit or component known in the art. Alternatively or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc. Additionally, each processor 116 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0063] The memory 134 may include volatile and non-volatile memory, removable and non-removable media, implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Such memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), flash memory or other storage technology, compact disk ROM (CD-ROM), digital versatile disk (DVD) or other optical storage device, cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, redundant array of independent disks (RAID) storage system, or any other medium that can be used to store the required information and can be accessed by the computing device. The memory 134 can be implemented as a computer-readable storage medium ("CRSM"), which can be any available physical medium that can be accessed by the processor 116 to execute the instructions stored on the memory 134. In a basic specific implementation, the CRSM can include RAM and flash memory. In other specific implementations, the CRSM can include, but is not limited to, ROM, EEPROM or any other non-temporary and / or tangible medium that can be used to store the desired information and can be accessed by the processor 116.

[0064] In general, the HMD system 100 may include logic (e.g., software, hardware, and / or firmware, etc.) configured to implement the techniques, functions, and / or operations described herein. The computer-readable medium 134 is shown to include various modules (such as instructions, data storage, etc.) that may be configured to execute on the processor 116 for implementing the techniques, functions, and / or operations described herein. Some exemplary functional modules are shown to be stored in the computer-readable medium 134 and executable on the processor 116, although the same functions may alternatively be implemented in hardware, firmware, or as a system on a chip (SOC) and / or other logic.

[0065] For the benefit of other modules, the operating system module 502 can be configured to manage hardware within the HMD 102 and / or the HMD system 100 and coupled to the HMD and / or the HMD system. In addition, in some cases, the HMD system 100 may include one or more applications 112 stored in the memory 134 or otherwise accessible by the HMD system 100. In this specific implementation, the application 112 includes a gaming application. However, the HMD system 100 may include any number or type of applications and is not limited to the specific examples shown herein. The ANR component 136 can be configured to perform the ANR techniques described herein to reduce the fan noise 140 at the ear location of the user 106. The memory 134 can further store the model 138 described herein, which can be used to implement the ANR techniques described herein.

[0066] Typically, the HMD system 100 has an input device 504 and an output device 506. The input device 504 may include a control button. In some implementations, one or more microphones 126 may be used as an input device 504 to receive audio input (such as user voice input) and to capture audio data representing fan noise 140 for use in the ANR technology described herein. In some examples, the microphone 126 has a frequency response in the range of about 20 Hz to 24 kHz, and a sensitivity of about -25 decibels relative to full scale (dBFS) / Pascal (Pa) at 1kHz. In some implementations, one or more cameras 508 or other types of sensors 122, such as the above-mentioned temperature sensor 124, an inertial measurement unit (IMU) 510, etc., may be used as an input device 504. For example, the IMU 510 may be configured to detect head movement of the user 106, including for gesture input purposes. The sensors 122 may further include sensors for generating motion, position, and orientation data, such as gyroscopes, accelerometers, magnetometers, color sensors, or other motion, position, and orientation sensors. The sensors 122 may also include sub-portions of sensors, such as a series of active or passive markers that can be viewed from the outside by a camera or color sensor to generate motion, position, and orientation data. For example, a VR headset may include multiple markers on its exterior, such as reflectors or lights (e.g., infrared or visible light), which, when viewed by an external camera or illuminated by light (e.g., infrared or visible light), can provide one or more reference points for software interpretation to generate motion, position, and orientation data. The sensors 122 may include light sensors that are sensitive to light (e.g., infrared or visible light) projected or broadcast by a base station in the environment of the HMD 102. The IMU 510 may be an electronic device that generates calibration data based on measurement signals received from accelerometers, gyroscopes, magnetometers, and / or other sensors suitable for detecting motion, correcting errors associated with the IMU 510, or some combination thereof. Based on the measurement signals, such motion-based sensors such as the IMU 510 can generate calibration data indicating an estimated position of the HMD 102 relative to an initial position of the HMD 102. For example, a plurality of accelerometers can measure translational motion (forward / backward, up / down, left / right), and a plurality of gyroscopes can measure rotational motion (e.g., pitch, yaw, and roll). The IMU 510 can, for example, rapidly sample the measurement signals and calculate an estimated position of the HMD 102 based on the sampled data. For example, the IMU 510 can integrate the measurement signals received from the accelerometers over time to estimate a velocity vector, and integrate the velocity vector over time to determine an estimated position of a reference point on the HMD 102. A reference point is a point that can be used to describe the position of the HMD 102.Although the reference point can be generally defined as a point in space, in various embodiments, the reference point is defined as a point within the HMD 102 (e.g., the center of the IMU 510). Alternatively, the IMU 510 provides sampled measurement signals to an external console (or other computing device), which determines the calibration data.

[0067] The sensor 122 can operate at a relatively high frequency to provide sensor data at a high rate. For example, sensor data can be generated at a rate of 1000 Hz (or 1 sensor reading every 1 millisecond). In this way, one thousand readings are obtained every second. When the sensor generates so much data at this rate (or at a greater rate), the data set used to predict motion is quite large, even in a relatively short time period of the order of tens of milliseconds. As mentioned, in some embodiments, the sensor 122 may include a light sensor that is sensitive to light emitted by a base station in the environment of the HMD 102, for the purpose of tracking the position and / or orientation, posture, etc. of the HMD 102 in three-dimensional (3D) space. The calculation of position and / or orientation can be based on the timing characteristics of the light pulses and the presence or absence of light detected by the sensor 122.

[0068] In some embodiments, additional input devices 504 may be provided in the form of a keyboard, keypad, mouse, touch screen, joystick, etc. In other embodiments, a keyboard, keypad, or other similar form of mechanical input may be omitted from the HMD system 100. In some examples, the input devices 504 may include control mechanisms such as basic volume control buttons for increasing / decreasing the volume, as well as power and reset buttons.

[0069] The output device 504 may include a display or display panel 512 (e.g., a stereo display panel pair). The display panel 512 of the HMD system 100 may utilize any suitable type of display technology, such as an emissive display that utilizes a light emitting element (e.g., a light emitting diode (LED)) to emit light during the presentation of a frame on the display panel 512. As an example, the display panel 512 of the HMD system 100 may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an inorganic light emitting diode (ILED) display, or any other suitable type of display technology for HMD applications. The output device 506 may further include, but is not limited to, an optical element (e.g., LED), a vibrator for generating a tactile sensation, and the above-mentioned fan 118 and associated drive circuit 120, and an external ear speaker 114. In some examples, the external ear speaker 114 is a 37.5 millimeter (mm) external ear balanced mode radiator (BMR), wherein the frequency response is in the range of about 40 Hz and 24 kHz, the impedance is 6 ohms, and the sound pressure level (SPL) at 1 centimeter (cm) is 98.96 dB SPL. In some examples, the off-ear speakers 114 are implemented in the form of sound bars, speakers embedded in the HMD body, and / or speakers embedded in the headband of the HMD 102.

[0070] The HMD system 100 may include a power source 128, such as one or more batteries. Additionally or alternatively, the HMD 102 may include a power cable port to connect to an external power source via a wired manner, such as a cable.

[0071] The HMD system 100 may further include a communication interface 514, such as a wireless unit coupled to an antenna to facilitate wireless connection to a network. Such a wireless unit may implement one or more of a variety of wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), etc. It should be understood that the HMD 102 may further include physical ports to facilitate wired connections to a network, connected peripheral devices (including a computer 500, such as a host computer 104, which may be a PC, a game console, etc.), or plug-in network devices that communicate with other wireless networks.

[0072] HMD 102 may further include an optical subsystem 516 that uses one or more optical elements to direct light from the electronic display panel 512 to the user's eyes. The optical subsystem 516 may include various types and combinations of different optical elements, including but not limited to apertures, lenses (e.g., Fresnel lenses, convex lenses, concave lenses, etc.), filters, and the like. In some embodiments, one or more optical elements in the optical subsystem 516 may have one or more coatings, such as anti-reflective coatings. The amplification of the image light by the optical subsystem 516 allows the electronic display panel 512 to be physically smaller, lighter, and consume less power than a larger display. Additionally, the amplification of the image light can increase the field of view (FOV) of the displayed content (e.g., image). For example, the FOV of the displayed content is such that the displayed content is presented using almost all of the user's FOV (e.g., 120 degrees-150 degrees diagonal), and in some cases using all of the user's FOV. AR applications may have a narrower FOV (e.g., about 40 degrees FOV). The optical subsystem 516 may be designed to correct one or more optical errors, such as, but not limited to, barrel distortion, pincushion distortion, longitudinal chromatic aberration, lateral chromatic aberration, spherical aberration, coma, field curvature, astigmatism, etc. In some embodiments, the content provided to the electronic display panel 512 for display is pre-distorted, and when the optical subsystem 516 receives image light generated based on the content from the electronic display panel 512, the optical subsystem corrects the distortion.

[0073] The HMD system 100 may further include an eye tracking system 518. The camera 508 or other optical sensor within the HMD 102 may capture image information of the user's eyes, and the eye tracking system 518 may use the captured information to determine the interpupillary distance, the interocular distance, the 3D position of each eye relative to the HMD 102 (e.g., for distortion adjustment purposes), including the magnitude of twist and rotation (i.e., roll, pitch, and yaw), and the gaze direction of each eye. In one example, infrared light is emitted within the HMD 102 and reflected from each eye. The reflected light is received or detected by a camera of the HMD 102 and analyzed to extract eye rotation from changes in the infrared light reflected by each eye. Many methods for tracking the eyes of the user 106 may be used by the eye tracking system 518. Thus, the eye tracking system 518 can track up to six degrees of freedom (i.e., 3D position, roll, pitch, and yaw) for each eye, and can combine at least a subset of the tracked quantities from the two eyes of the user 106 to estimate the gaze point (i.e., the 3D position or orientation in the virtual scene that the user is viewing). For example, the eye tracking system 518 can integrate information from past measurements, measurements identifying the position of the head of the user 106, and 3D information describing the scene presented by the electronic display panel 512. Thus, information about the position and orientation of the eyes of the user 106 is used to determine the gaze point in the virtual scene that the user 106 is viewing and that is presented by the HMD 102.

[0074] The HMD system 100 may further include a head tracking system 520. The head tracking system 520 may utilize one or more sensors 122 to track the head movement of the user 106, including head rotation, as described above. For example, the head tracking system 520 may track up to six degrees of freedom (i.e., 3D position, roll, pitch, and yaw) of the HMD 102. These calculations may be performed at each frame of a series of frames so that the application 112 (e.g., a video game) may determine how to render the scene in the next frame based on the head position and orientation. In some embodiments, the head tracking system 520 is configured to predict the future position and / or orientation of the HMD 102 based on current and / or past data. This is because the application 112 is required to render a frame before the user 106 actually sees the light (and therefore the image) on the display 512. Therefore, the next frame may be rendered based on this future prediction of the head position and / or orientation made at an earlier point in time (such as approximately 25 milliseconds (ms)-30 milliseconds (ms) before the rendered frame). In a distributed system where the host computer 104 is communicatively (e.g., wirelessly) coupled to the HMD 102, future predictions of the head pose can be made 30 ms or earlier in advance of the illumination time of the frame to account for network delays, compression operations, etc. The rotation data provided by the head tracking system 520 can be used to determine the direction in which the HMD 102 is rotated, as well as the amount of rotation of the HMD 102 in any suitable measurement units. For example, the direction of rotation can be simplified and output in terms of a positive or negative horizontal direction and a positive or negative vertical direction, corresponding to left, right, up, and down. The amount of rotation can be expressed in degrees, radians, etc. The angular velocity can be calculated to determine the rotation rate of the HMD 102.

[0075] The HMD system 100 may further include a controller tracking system 522. The controller tracking system 522 may utilize one or more sensors 122 to track controller motion. For example, the controller tracking system 522 may track up to six degrees of freedom (i.e., 3D position, roll, pitch, and yaw) of a controller held in the user 106's hand. These calculations may be performed at each frame in a series of frames so that an application 112 (e.g., a video game) may determine how to present a virtual controller and / or virtual hand in the scene of the next frame based on the controller position and orientation. In some embodiments, the controller tracking system 522 is configured to predict the future position and / or orientation of the HMD 102 based on current and / or past data, as described above with respect to the head tracking system 522.

[0076] Although the subject matter has been described in language specific to structural features, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features described. Rather, the specific features are disclosed as illustrative forms of implementing the claims.

Claims

1. A head mounted display (HMD) system, comprising: An HMD, wherein the HMD comprises: fan; Left external speaker; and Right ear external speaker; Processor; and a memory storing computer executable instructions that, when executed by the processor, cause the processor to: receiving data indicative of noise generated by the fan; determining one or more audio parameter values ​​based at least in part on the data and using a model; outputting, via the left external ear speaker, a first sound having one or more first audio characteristics based at least in part on the one or more audio parameter values ​​to reduce the noise generated by the fan at a left ear location of a user of the HMD; and A second sound having one or more second audio characteristics based at least in part on the one or more audio parameter values ​​is output via the right external ear speaker to reduce the noise generated by the fan at the location of the user's right ear.

2. The HMD system according to claim 1, wherein: The HMD further comprises: left microphone; and right microphone; and The data include: first audio data representing noise captured by the left microphone; and Second audio data representing noise captured by the right microphone.

3. The HMD system according to claim 1, wherein: The processor includes a central processing unit (CPU); The HMD further comprises at least one of the following: a temperature sensor to sense the temperature of electronic components of the HMD; a driving circuit for driving the fan; or Graphics processing unit (GPU); and The data includes at least one of the following: temperature data indicative of the temperature sensed by the temperature sensor; driving data indicating a level at which the fan is driven via the driving circuit; or Utilization data indicating utilization of the CPU or the GPU.

4. The HMD system of claim 1 , wherein the instructions, when executed by the processor, further cause the processor to: determining that the fan is driven at a level that satisfies a threshold level; and In response to determining that the fan is driven at the level, it is determined to execute an active noise reduction algorithm.

5. The HMD system according to claim 1, wherein: A video game is being played using the HMD; and When the instructions are executed by the processor, the processor further causes: while outputting the first sound, outputting a third sound via the left external ear speaker, the third sound corresponding to audio content of the video game; as well as While the second sound is being output, a fourth sound is output via the right-ear external speaker, the fourth sound corresponding to the audio content.

6. A method comprising: receiving, by a processor, data indicative of noise generated by a fan of a head mounted display (HMD); determining, by the processor, one or more audio parameter values ​​based at least in part on the data and using a model; as well as Sound having one or more audio characteristics based at least in part on the one or more audio parameter values ​​is output via one or more off-ear speakers of the HMD to reduce the noise generated by the fan at the location of ears of a user of the HMD.

7. The method of claim 6, wherein the data comprises audio data representing noise captured by one or more microphones of the HMD.

8. The method according to claim 7, wherein: The one or more microphones include a first microphone and a second microphone; The method further includes determining, by the processor and using the first microphone and the second microphone, audio characteristics corresponding to a waveform of the noise generated by the fan; and The determination of the one or more audio parameter values ​​is based at least in part on the audio characteristics.

9. The method of claim 6, wherein the data comprises at least one of: temperature data indicative of a temperature of an electronic component of the HMD sensed by a temperature sensor of the HMD; driving data indicating a level at which the fan is driven via a driving circuit of the HMD; or Utilization data indicating utilization of a central processing unit (CPU) of the HMD or a graphics processing unit (GPU) of the HMD.

10. The method of claim 9, further comprising, prior to said receiving the data, determining, by the processor, to avoid using a power source of the HMD to power the one or more microphones of the HMD for the purpose of using the one or more microphones in an active noise reduction algorithm.

11. The method according to claim 6, further comprising: determining, by the processor, that the fan is drawing power from a power source of the HMD; as well as Responsive to the determining that the fan is drawing the power, a determination is made by the processor to execute an active noise reduction algorithm.

12. The method according to claim 6, wherein: The one or more extra-ear speakers include a first extra-ear speaker and a second extra-ear speaker; and The outputting the sound having the one or more audio characteristics comprises: outputting a first sound having one or more first audio characteristics via the first external ear speaker to reduce the noise generated by the fan at the location of the first ear of the user; as well as A second sound having one or more second audio characteristics is outputted via the second extra-ear speaker to reduce the noise generated by the fan at the location of the second ear of the user.

13. The method of claim 12, wherein the one or more first audio characteristics of the first sound are different from the one or more second audio characteristics of the second sound.

14. A head mounted display (HMD) system comprising: An HMD, wherein the HMD comprises: Fan; and one or more external ear speakers; Processor; and a memory storing computer executable instructions that, when executed by the processor, cause the processor to: receiving data indicative of noise generated by the fan; determining one or more audio parameter values ​​based at least in part on the data and using a model; and Sound having one or more audio characteristics based at least in part on the one or more audio parameter values ​​is output via the one or more off-ear speakers to reduce the noise generated by the fan at the location of ears of a user of the HMD.

15. The HMD system of claim 14, wherein: The one or more extra-ear speakers include a first extra-ear speaker and a second extra-ear speaker; and Outputting the sound having the one or more audio characteristics comprises: outputting a first sound having one or more first audio characteristics via the first external ear speaker to reduce the noise generated by the fan at the location of the first ear of the user; as well as A second sound having one or more second audio characteristics is outputted via the second extra-ear speaker to reduce the noise generated by the fan at the location of the second ear of the user.

16. The HMD system of claim 15, wherein: The first external ear speaker is located on the left side of the HMD; The first ear is the left ear of the user; The second external ear speaker is located on the right side of the HMD; and The second ear is the user's right ear.

17. The HMD system of claim 14, wherein: The HMD further comprises: First microphone; and Second microphone, The data include: first audio data representing noise captured by the first microphone; and Second audio data representing noise captured by the second microphone.

18. The HMD system of claim 14, wherein: The processor includes a central processing unit (CPU); The HMD further comprises at least one of the following: a temperature sensor to sense the temperature of electronic components of the HMD; a driving circuit for driving the fan; or Graphics processing unit (GPU); and The data includes at least one of the following: temperature data indicative of the temperature sensed by the temperature sensor; driving data indicating a level at which the fan is driven via the driving circuit; or Utilization data indicating utilization of the CPU or the GPU.

19. The HMD system of claim 14, wherein the instructions, when executed by the processor, further cause the processor to: determining that the fan is driven at a level that satisfies a threshold level; and In response to determining that the fan is driven at the level, it is determined to execute an active noise reduction algorithm.

20. The HMD system of claim 14, wherein: The sound is a first sound; and When executed by the processor, the instructions further cause the processor to output a second sound corresponding to the audio content of the application being executed via the one or more external speakers while outputting the first sound.

Images