Systems with device-mounted audio mode

Abstract

The present invention is entitled "System with Device Mount Audio Mode". The present invention describes a system with a head mounted display (HMD) mount and a mobile device. The system's processor can determine whether the mobile device is mounted on the HMD mount and process audio signals transmitted from the mobile device to the wireless headset based on whether the mobile device is mounted on the HMD mount. When the mobile device is not mounted on the HMD mount, the mobile device or the wireless headset can operate in a first audio mode. When the mobile device is mounted on the HMD mount, the mobile device or the wireless headset can operate in a second audio mode. The second audio mode can reduce audio signal delay between the mobile device and the wireless headset and improve motion to sound quality. Other embodiments are also described and protection is claimed.

Description

[0001] This application is a divisional application of the invention patent application with application date of May 5, 2019, application number 201910367569.4, and invention name “System with audio mode of device installation part”.

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 669,322, filed May 9, 2018, which is incorporated herein by reference in its entirety to provide continuity of disclosure. Technical Field

[0003] The present invention discloses embodiments related to head mounted displays. More specifically, embodiments related to head mounted display systems with audio output are disclosed. Background Art

[0004] A head mounted display (HMD) is a display device that can be worn on the head to display images to the user. HMDs are used for gaming, aviation, etc. HMDs include virtual reality and augmented reality headsets, which are head mounted devices that provide users with a realistic simulated experience. Some virtual reality headsets include an HMD mount for holding a mobile device such as a smart phone in front of the user. The mobile device displays an image of the virtual reality scene and can play sounds corresponding to the displayed image. The HMD mount may include electronic devices, such as a processor or an inertial measurement unit, and the mobile device may communicate electrical signals with the HMD mount via cables and connectors. For example, a user may manually control the mobile device by interacting with the HMD mount to input a control signal, which is transmitted to the mobile device via cables and connectors. Summary of the invention

[0005] The present invention discloses a system including a head mounted display (HMD) mount and a mobile device, and a method for activating an audio mode of the system based on a physical relationship between the HMD mount and the mobile device. In one embodiment, the HMD mount and the mobile device are joined by mounting the mobile device on the HMD mount. The system can detect whether the mobile device is mounted on the HMD mount and make changes to the audio mode in response to the detection. For example, the device processor can determine that the mobile device is mounted in the housing of the HMD mount, and responsively change the audio mode of the mobile device from a first (e.g., normal) audio mode to a second (e.g., special) audio mode. Alternatively, the device processor can responsively transmit an audio mode signal to a wireless headset processor of a wireless headset, indicating that the mobile device and the HMD mount are physically paired (adjacent or fixed to each other). The wireless headset (or one or more speakers built into the HMD mount) can determine that the mobile device has been installed in response to receiving the audio mode signal, and change the audio mode from the first audio mode to the second audio mode. The audio mode change of the mobile device or the wireless headset can include reducing the audio signal delay of the delay chain between the mobile device and the wireless headset. For example, the audio signal delay can be reduced by reducing the buffer size used to store the audio signal in the wireless headset memory of the wireless headset. The mobile device can change the audio mode to reduce the audio signal delay by using a different digital audio codec in the second audio mode. Therefore, the second audio mode can have a lower audio signal delay than the first audio mode. The lower audio signal delay can provide a desired motion-to-sound delay between the movement of an object displayed by the mobile device in the virtual reality scene and the corresponding sound played by the wireless headset (or by one or more speakers built into the HMD mount).

[0006] In one embodiment, the change to the audio mode may involve non-delayed adjustment of the mobile device or wireless headset. For example, the wireless headset (or HMD mount) may include a speaker to play sounds corresponding to the image displayed by the mobile device, and in the second audio mode the sounds may be presented in a different direction than in the first audio mode. Alternatively or in addition, the mobile device may include a microphone to pick up sounds from the surrounding environment, and when the system is operating in the second audio mode, the microphone may be adjusted to pick up sounds in a preset direction (e.g., corresponding to the assumed direction of the user's mouth).

[0007] The above summary does not include an exhaustive list of all aspects of the invention. It is contemplated that the invention includes all systems and methods that can be practiced from all appropriate combinations of the various aspects summarized above and disclosed in the following detailed description and specifically pointed out in the claims filed with this patent application. Such combinations have specific advantages not specifically recited in the above summary. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is an illustration of a system with a mobile device mounted on a head mounted display (HMD) mount, according to one embodiment.

[0009] Figure 2 is a block diagram of a system according to one embodiment.

[0010] Figure 3 is a graph of motion to sound quality plotted against the audio signal delay of a system according to one embodiment.

[0011] Figure 4 is a diagram of a delay chain of a system according to one embodiment.

[0012] Figure 5 is a flow chart of a method for adjusting a delay parameter of a system in a second audio mode according to one embodiment.

[0013] Figure 6 is a flow chart of a method for adjusting non-delay parameters of a system in a second audio mode, according to one embodiment.

[0014] Figure 7A-7B is a diagram of a method of adjusting sound presentation of a wireless headset in a second audio mode, according to one embodiment.

[0015] Figure 8A-B is a diagram of a method of adjusting microphone focus by a mobile device in a second audio mode, according to one embodiment. DETAILED DESCRIPTION

[0016] Embodiments of the present invention describe a system having controllable audio modes and including a head mounted display (HMD), and a method of activating an audio mode based on a physical relationship between an HMD mount and a mobile device. The system may be used as a virtual reality or augmented reality headset. However, the system may be used in other applications, such as in aviation, engineering, or medicine, to name a few possible applications.

[0017] In various embodiments, description is made with reference to the accompanying drawings. However, some embodiments may be implemented without one or more of these specific details or without being combined with other known methods and configurations. In the following description, multiple specific details such as specific configurations, sizes and processes are set forth to provide a thorough understanding of each embodiment. In other examples, well-known processes and manufacturing techniques are not particularly described in detail so as not to unnecessarily obscure the description. References to "one embodiment", "embodiment" etc. throughout this specification sheet mean that the specific features, structures, configurations or characteristics described are included in at least one embodiment. Therefore, phrases "one embodiment", "embodiment" etc. appear in various places throughout this specification sheet and do not necessarily refer to the same embodiment. In addition, specific features, structures, configurations or characteristics may be combined in one or more embodiments in any appropriate manner.

[0018] Relative terms are used throughout the description to refer to relative positions or directions. For example, "in front of" may indicate a first direction away from a reference point. Similarly, "behind" may indicate a position in a second direction away from a reference point and opposite to the first direction. However, such terms are provided to establish a relative frame of reference, and such terms are not intended to limit the use or orientation of the system to the specific configurations described in the various embodiments below.

[0019] In one aspect, a system is capable of automatically changing an audio mode based on a physical relationship between an HMD mount and a mobile device. A processor of the system, for example, a device processor of the mobile device and / or a wireless headset processor of a wireless headset of the HMD mount, can recognize that the mobile device is mounted on the HMD mount. The mobile device and / or the wireless headset can responsively change from a first audio mode to a second audio mode. The second audio mode may include one or more changes to the manner in which audio signals generated by the mobile device are processed, and may reduce latency in the audio signals. When the system is operating in the second audio mode, the sound presented to the user by the wireless headset may realistically track the image displayed to the user by the mobile device. Therefore, the mobile device and / or the wireless headset with the device mount audio mode can present a realistic augmented reality or virtual reality experience to the user.

[0020] See also Figure 1, according to one embodiment, a diagram of a system with a mobile device mounted on a head mounted display (HMD) mount is shown. The system 100 may include an HMD mount 102 and a mobile device 104 (shown as hidden within the HMD mount 102). The HMD mount 102 and the mobile device 104 may be independent. For example, the mobile device 104 may be mounted on the HMD mount 102 and may be removed from the HMD mount. In one embodiment, the HMD mount 102 includes a housing 106 that is retained on the user's head by a band 108, temples, arms, or another feature extending from the housing 106 to engage with the user's head. The mobile device 104 may be mounted in the housing 106 of the HMD mount 102. The mobile device 104 may be a smart phone, a tablet computer, etc.

[0021] System 100 may include various electronics and audio hardware to generate sounds 112 associated with images displayed by mobile device 104. More specifically, system 100 may include speaker 110, which may include an electroacoustic transducer to convert audio signals transmitted by mobile device 104 into sounds 112.

[0022] In one embodiment, the audio hardware having the speaker 110 is a wireless headset 101 in signal communication with the mobile device 104. The wireless headset 101 may include several earphones external to the HMD mount 102, such as a left earphone and a right earphone.

[0023] In one embodiment, audio hardware with speakers 110 is built into the HMD mount 102. The HMD mount 102 may include a mount body and an earphone component having one or more speakers 110 to generate sound 112 for the user. The earphone component may be physically connected to the mount body via a cable. Thus, the speakers 110 of the system 100, whether external to the HMD mount 102 or headphones or earbuds built into the mount 102, can present sounds 112 associated with images displayed by the mobile device 104 when the mobile device 104 is mounted on the HMD mount 102.

[0024] The sound 112 creates a variety of audio experiences for the user. The playback of the audio signal by the speaker 110 can present the sound 112 in a standard stereo experience, wherein the center sound image is located inside the user's head. The center sound image may correspond to the ambient noise associated with the virtual reality scene displayed by the mobile device 104. The playback of the audio signal by the speaker 110 can present the sound 112 in a spatialized audio experience, wherein a virtual sound image is presented along the direction of the virtual sound source. The virtual sound image may correspond to a voice associated with a virtual sound source (e.g., a person) displayed by the mobile device 104 within the virtual reality scene. The audio signal may be processed at the mobile device 104 or at a processor contained in the housing of the wireless headset 101 or the HMD mount 102 to produce the desired audio experience.

[0025] The mobile device 104 may include various electronic devices and audio hardware. In one embodiment, the mobile device 104 includes one or more microphones 114. The microphones 114 may be located at several locations on the mobile device 104 to receive sound entering the mobile device 104 from various directions. For example, when the mobile device 104 is used in a handheld configuration, the first microphone 114A may be mounted on the rear surface of the mobile device 104 to receive sound from the rear direction. When the mobile device 104 is used in a handheld configuration, the mobile device 104 may include one or more second microphones 114B mounted on the bottom surface of the mobile device 104 to receive sound from the bottom direction. When the mobile device 104 is mounted in the HMD mount 102, the first microphone 114A may be shielded by the housing 106. In contrast, when the mobile device 104 is used in a head-mounted configuration, the second microphone 114B may not be shielded by the housing 106.

[0026] See also Figure 2, a block diagram of a system according to one embodiment is shown. The mobile device 104 may be any of several types of portable devices or apparatuses having circuits suitable for specific functions. Therefore, the illustrated circuits are provided as examples rather than limitations. The mobile device 104 may include one or more device processors 202 to execute instructions, thereby performing the different functions and capabilities described below. The instructions executed by the device processor 202 of the mobile device 104 may be retrieved from the device memory 204, which may include a non-transitory machine-readable medium. The instructions may be in the form of an operating system program having a device driver and / or an audio rendering engine for presenting a virtual sound source according to the method described below. The device processor 202 may also retrieve audio data 206 from the device memory 204, including audio data associated with a phone and / or a music playback function controlled by a phone or music application running on top of the operating system. Similarly, the audio data 206 may be associated with an augmented reality or virtual reality application running on top of the operating system. In order to perform such functions, the device processor 202 may directly or indirectly implement a control loop and receive input signals from other electronic components and / or provide output signals to other electronic components. For example, the mobile device 104 may receive an input signal from a microphone 114, a menu button, or a reference sensor 214. The mobile device 104 may output an audio signal 210. The audio signal 210 may include audio data 206. The mobile device 104 may generate the audio signal 210 and output the audio signal 210 to the device speaker 208 and / or the wireless headset 101 (or HMD mount 102) via a wired or wireless communication link. For example, the communication link may be established via a wireless connection using the Bluetooth standard, and the device processor 202 may wirelessly transmit the audio signal 210 to the wireless headset processor 217 (or HMD processor 218) via the communication link. As described below, the wireless headset 101 or the HMD mount 102 may receive and process the audio signal 210 to present the sound 112. More specifically, the wireless headset 101 or the HMD mount 102 may receive and play back the audio signal 210 to present the sound 112 from the headset speaker 110.

[0027] The mobile device 104 may transmit various signals. For example, in addition to the audio signal 210, the device processor 202 may generate an audio mode signal 212. When the mobile device 104 is mounted in the HMD mount 102, the audio mode signal 212 may be generated and / or transmitted. The device processor 202 may transmit the audio mode signal 212 to a wireless headset processor 217 of the wireless headset 101 or an HMD processor 218 of the HMD mount 102. The audio mode signal 212 may include a bit field to indicate to the receiving component (e.g., the wireless headset 101 or the HMD mount 102) that the mobile device 104 and the HMD mount 102 are physically paired.

[0028] In one embodiment, the electronic circuitry of the mobile device 104 includes a reference sensor 214 to output reference orientation data corresponding to a reference orientation of the mobile device 104. The reference orientation data may be provided to the device processor 202 or the device memory 204, and the device processor 202 may retrieve the reference orientation data from the device memory 204. The reference sensor 214 may be one or more of any known orientation sensors, such as an accelerometer, a magnetometer, a gyroscope, and the like. For example, the reference sensor 214 may be an inertial measurement unit (IMU) integrated within the housing of the mobile device 104. However, such inertial-based embodiments are not limiting, and the reference sensor 214 may include a non-inertial sensor, such as an optical sensor. The reference orientation data generated by the reference sensor 214 may be used as an input to control an image presented on the display 216 of the mobile device 104. For example, when the mobile device 104 is mounted on the HMD mount 102, movement of the user's head in a rightward direction may generate reference orientation data to cause the device processor 202 to present an image of a person presented on the display 216 in a leftward direction. A portion of the audio signal 210 corresponding to the image (eg, a person's voice) may be adjusted by the device processor 202 to spatialize the sound 112 presented to the user to also be shifted to the left.

[0029] The circuitry in the mounting body of the HMD mount 102 and / or the earphone housing of the wireless earphone 101 may receive the audio signal 210 from the mobile device 104. The wireless earphone 101 may include a wireless earphone processor 217 to receive and process the audio signal 210 transmitted from the device processor 202. Similarly, when the audio hardware of the system 100 is incorporated into the HMD mount 102, the mounting body may contain an HMD processor 218 to receive and process the audio signal 210 transmitted from the device processor 202 or other components of the mobile device 104. The wireless earphone processor 217 may store the audio signal 210 in the wireless earphone memory 219, and the HMD processor 218 may store the audio signal 210 in the HMD memory 220. The audio signal 210 may be retrieved from the memory by the corresponding processor for playback by the speaker 110 of the corresponding earphone component. The wireless earphone memory 219 or the HMD memory 220 may include a non-transitory machine-readable medium having a buffer, such as an audio jitter buffer, to store data contained in the audio signal 210 for playback. More specifically, the wireless headset processor 217 may be configured to present audio output to the user via the speaker 110, and the HMD processor 218 may be configured to present audio output to the user via the left speaker 110 and the right speaker 110. The system 100 may include several wireless headset 101 components, such that the first wireless headset may be configured to present a left channel audio output, and the second wireless headset may be configured to present a right channel audio output. The corresponding processor (wireless headset processor 217 or HMD processor 218) may provide an audio electrical signal corresponding to the buffered audio data to the speaker 110, so that the stereo sound 112 from the speaker 110 presents a virtual sound source in the source direction. For example, the sound 112 may be presented as corresponding to the movement of an object in a virtual reality scene played on the display 216 of the mobile device 104.

[0030] The processors of the wireless headset 101 and / or the HMD mount 102 may be connected to the corresponding RF circuits to receive and process the audio signal 210. In addition, the RF circuits of the HMD mount 102 and the wireless headset 101 may be in signal communication. Therefore, the functions of the HMD mount 102 and the wireless headset 101 may be distributed. For example, a signal may be wirelessly relayed from the mobile device 104 to the wireless headset 101 by the HMD mount 102. Therefore, the mobile device 104 may transmit the audio signal 210 separately to the HMD mount processor 218 for processing or buffering, and the HMD processor 218 may relay the processed signal to the wireless headset processor 217 to be output by the speaker 110 of the wireless headset 101. Therefore, the audio hardware of the system 100 may be incorporated into one or both of the wireless headset 101 and the HMD mount 102.

[0031] See also Figure 3 , according to one embodiment, shows a graph of motion to sound quality plotted against the audio signal delay of the system. The device processor 202 and the receiving processor (wireless headset processor 217 or HMD processor 218) may execute an audio rendering algorithm to determine appropriate audio electrical signals for the left and right speakers 110 to render spatialized sound sources in appropriate directions. More specifically, the processor of the system 100 may operate in an audio mode to provide a virtual reality or augmented reality experience to the user. When the mobile device 104 is mounted on the HMD mount 102, while using an augmented reality or virtual reality application, the rendered sound image should track the video image rendered on the display 216 as the user's head moves.

[0032] In order to achieve the required tracking between the sound image and the video image (which may be referred to as the motion-to-sound delay), the delay of the audio signal 210 must be within a predetermined range. More specifically, when the audio signal delay is less than a predetermined minimum threshold 302, the motion-to-sound quality is good, and the sound image is perceived as tracking the video image. In contrast, when the audio signal delay exceeds a predetermined minimum threshold 304, the motion-to-sound quality is poor, and the sound image is perceived as being out of touch with the video image. For augmented reality and virtual reality applications, the predetermined minimum threshold 302 may be 50 milliseconds, and the predetermined maximum threshold 304 may be 150 milliseconds. Therefore, in one embodiment, in order to achieve the required tracking between the sound image and the video image, the audio signal delay is less than 150 milliseconds, and more preferably less than 50 milliseconds. However, the audio signal delay of the existing wireless media playback mode is greater than 200 milliseconds. Therefore, the generation and playback of the audio signal 210 using the existing wireless media playback mode may not provide a realistic augmented reality or virtual reality experience to the user. To achieve the desired motion-to-sound delay, when the mobile device 104 is mounted on the HMD mount 102 , the system 100 may process the audio signal 210 differently than existing wireless media playback modes.

[0033] refer to Figure 4 , a diagram of a delay chain of a system is shown according to one embodiment. The delay chain 400 represents the delay of an audio signal as a delay period between a motion input 402 entering the system 100 and an audio output 404 leaving the system 100. The motion input 402 may be the movement of a user's head while the user is wearing the system 100 and viewing an augmented reality scene. The audio output 404 may present a sound 112 corresponding to the motion input 402, for example, the audio output 404 may be a moving sound image corresponding to a person displayed in the augmented reality scene. Thus, the entire delay chain 400 corresponds to the motion-to-sound delay of the system 100.

[0034] The audio signal delay may be part of the delay chain 400. The audio signal delay may include a delay period between the audio insertion 408 performed by the device processor 202 and the audio processing 410 performed by the receiving processor (wireless headset processor 217 or HMD processor 218). More specifically, the audio signal delay may represent the delay between the mobile device 104 and the headset component (located external or built-in to the HMD mount 102). The audio insertion 408 may be generated by the motion input 402. For example, head movement may cause the reference sensor 214 to generate reference orientation data as the motion input 402, which is processed by the augmented reality application to cause movement of the displayed scene and adjustment of the audio signal 210. The audio signal adjustment is represented by the audio insertion 408. The audio signal 210 may be transmitted through the air from the transmitter of the mobile device 104 to the receiver of the wireless headset 101 or the HMD mount 102. The receiver may be part of the wireless headset component, and thus the delay optimization described herein may be between the mobile device 104 and the wireless headset connected to the mobile device 104. The wireless headset 101 or HMD mount 102 may receive the audio signal 210 and store the audio signal 210 in a corresponding buffer 412 (wireless headset memory 219 or HMD memory 220). Storing the audio signal 210 in the buffer 412 may provide signal reliability by allowing timing correction of the data stream of the audio signal 210. In the existing wireless media playback mode, the buffer 412 may account for a large portion of the delay chain 400. For example, in the first (e.g., normal) audio mode for music playback, the buffer 412 may have a size 414, e.g., length, of more than 200 milliseconds. The buffered audio signal 210 may then be processed locally by the wireless headset processor 217 or the HMD processor 218, e.g., to perform equalization, and then output to the speaker 110 for playback as an audio output 404. As described below, when the mobile device 104 is mounted on the HMD mount 102, the process of the audio signal delay portion along the delay chain 400 may be reduced to reduce the audio signal delay to below the predetermined minimum threshold 302 and achieve the desired motion-to-sound quality.

[0035] refer to Figure 5, a flow chart of a method for adjusting a delay parameter of a system in a second audio mode is shown according to one embodiment. At operation 502, the wireless headset 101 or the HMD mount 102 receives an audio signal 210 from the mobile device 104. The audio signal 210 may be generated by the device processor 202 at the audio insertion 408 of the delay chain 400 and wirelessly transmitted over the air from the transceiver of the mobile device 104 to the transceiver of the wireless headset 101 or the HMD mount 102. The transmission may be directly to the wireless headset 101 or via the HMD mount 102. In one embodiment, the device processor 202 and / or the receiving processor is configured to process the audio signal 210 in a first audio mode. For example, when the mobile device 104 is not mounted on the HMD mount 102, the first audio mode may be the audio mode used by the system 100. When the mobile device 104 is in a handheld configuration, for example, when the mobile device 104 is running a music application, the first audio mode can be used during the playback of music by a wireless headset component (external to the HMD mount 102 or built into the HMD mount 102). In the first audio mode, the size 414 of the buffer 412 can be greater than 200 milliseconds.

[0036] At operation 504, the system 100 determines whether the mobile device 104 is mounted on the HMD mount 102. This determination may be made by the device processor 202 and / or the receiving processor. For example, the mobile device 104 and / or the HMD mount 102 may include a proximity sensor ( Figure 2 104) to determine whether the mobile device 104 is loaded into the cavity of the housing 106. When the proximity sensor detects that the mobile device 104 and the HMD mount 102 are adjacent, the sensor can transmit a corresponding signal to a corresponding processor.

[0037] Proximity between the mobile device 104 and the HMD mount 102 may be detected in other ways. By way of example, the mobile device 104 and the HMD mount 102 may communicate via near field communication. When the mobile device 104 is mounted on the HMD mount 102, the components may initiate communication. The HMD mount 102 may authenticate the mobile device 104 as a particular type of device, such as a mobile phone. Similarly, the mobile device 104 may authenticate the HMD mount 102 as a particular type of device, such as an augmented reality or virtual reality headset. In either case, the device processor 202 and / or the receiving processor may detect that the mobile device 104 is mounted on the HMD mount 102.

[0038] When the mobile device 104 is mounted on the HMD mount 102, the device processor 202 can generate an audio mode signal 212. More specifically, the device processor 202 can generate the audio mode signal 212 in response to detecting proximity between the mobile device 104 and the HMD mount 102. The wireless headset 217 or the HMD processor 218 can receive the audio mode signal 212. In one embodiment, the receiving processor is configured to determine that the mobile device 104 is mounted on the HMD mount 102 in response to receiving the audio mode signal 212. Therefore, the device processor 202 or the HMD processor 218 can determine that the mobile device 104 is mounted on the HMD mount 102, and the receiving processor can make a determination based on the received signal indicating the physical relationship.

[0039] At operation 506, the system 100 may change the audio mode from the first audio mode to a second (e.g., special) audio mode in response to determining that the mobile device 104 is mounted on the HMD mount 102. The device processor 202 and / or the receive processor is configured to process the audio signal 210 in a different manner in the second audio mode than in the first audio mode. The difference in signal processing may have any of several effects as described in more detail below. For example, the second audio mode may require a change in a delay parameter or a non-delay parameter of the system 100. Therefore, in the second audio mode, the audio parameters may be different compared to the first audio mode.

[0040] The change from the first audio mode to the second audio mode is activated in response to the device installation. Therefore, the second audio mode is the device-mount audio mode. The device-mount audio mode makes the system 100 an active system, compared to a passive system that requires user input to change or control the audio output. That is, the headphone system can automatically change the audio mode by detecting that the mobile device 104 is in the HMD mount 102, without the user manually inputting audio parameters into the augmented reality or virtual reality application.

[0041] At operation 508, the wireless headset processor 217 or the HMD processor 218 processes the audio signal 210 differently, at least in part, by reducing the size 414 of the buffer 412 in the second audio mode. The audio signal 210 may be transmitted from the mobile device 104 to the wireless headset 101 or the HMD mount 102 via a digital wireless interface, such as a Bluetooth adapter, using the Bluetooth standard. Thus, the buffer 412 may be an audio jitter buffer 412. In the second audio mode, the size 414 of the audio jitter buffer 412 may be less than half of the size 414 in the first audio mode. For example, while the size 414 in the first audio mode may be greater than 200 milliseconds, the size 414 in the second audio mode may be less than 50 milliseconds. In one embodiment, the size 414 of the audio jitter buffer 412 in the second audio mode is less than 40 milliseconds.

[0042] At operation 510, the audio signal 210 is stored in the buffer 412. The audio signal 210 may be stored by the wireless headset 219 or the HMD memory 220. For example, the wireless headset processor 217 may write the audio data contained in the audio signal 210 to the buffer 412 in the wireless headset memory 219 upon receiving the audio signal 210 from the mobile device 104. Storing the audio data in the smaller audio jitter buffer 412 is part of the low-latency wireless mode (second audio mode). Reliability in the lower latency wireless mode may be maintained in part without high-latency buffering because the antennas of the mobile device 104 and the wireless headset 101 or HMD mount 102 are close enough to each other, such as within a few inches and within line of sight, to reduce the possibility of data loss or timing errors during data transmission.

[0043] At operation 512, the audio signal 210 is converted to sound 112. One or more speakers may convert the audio signal 210 to sound 112. For example, the speakers 110 may render the sound 112 in a standard stereo experience or in a virtual sound experience as described below. Due to the lower latency wireless mode input when the mobile device 104 is placed in the housing 106, the rendered sound 112 may track the movement of the image displayed on the mobile device 104 well.

[0044] When the mobile device 104 is mounted, for example, placed in the housing 106 of the HMD mount 102, the system 100 can adapt to the new operating mode in several other ways. More specifically, the transition of the audio mode from the first audio mode to the second audio mode may include other operations. In one embodiment, when it is determined that the mobile device 104 is mounted on the HMD mount 102, any current audio playback of the speaker 110 may fade out. For example, the user may be listening to music played by the speaker 110, and inserting the mobile device 104 into the HMD mount 102 may prompt the system 100 that the user is switching from a music listening mode with a standard stereo experience to an augmented reality mode with a spatialized sound experience. System use cases may include virtual reality and augmented reality games, media playback, telephone communications, and interactions with an intelligent personal assistant of an operating system stored on the mobile device 104. Based on whether the mobile device 104 is mounted on the HMD mount 102, the system 100 can switch between different audio experiences for each of these use cases.

[0045] As described above, when the system 100 enters the second audio mode, the low-latency wireless mode may include reducing the buffer size 414. The low-latency wireless mode may include changing one or more other delay parameters of the wireless data exchange stack used to transmit the audio signal 210 from the mobile device 104 to the wireless headset 101 or the HMD mount 102. In one embodiment, the device processor 202 is configured to change the audio encoding format of the audio signal 210. The device processor 202 may change the audio encoding format in response to determining that the mobile device 104 is mounted on the HMD mount 102. More specifically, when the mobile device 104 is mounted on the HMD mount 102, the audio encoding format of the audio signal 210 may be changed from a first audio encoding format to a second audio encoding format having a lower delay than the first audio encoding format. For example, the system 100 may use a digital audio encoder and decoder to transmit the audio signal 210 from the mobile device 104 to the wireless headset 101 or the HMD mount 102. In the first audio mode, the digital audio codec may be an advanced audio coding codec with a low complexity profile (AAC-LC). In the second audio mode, the digital audio codec may be changed to a codec with lower latency between encoding and decoding operations, such as a low complexity sub-band codec (SBC). In one embodiment, the system 100 may not have digital audio compression in the second audio mode. For example, the audio signal 210 may be transmitted from the mobile device 104 to the wireless headset 101 or the HMD mount 102 as uncompressed audio.

[0046] Other changes in the delay parameters in the second audio mode may include changes in the transmit buffer on the transmit side. The device memory 204 may include a transmit buffer that partially defines the frequency at which the device processor 202 provides data to the wireless data exchange stack (e.g., Bluetooth stack). In one embodiment, the transmit buffer may be reduced in the second audio mode compared to the first audio mode. For example, the transmit buffer may be 10 milliseconds in the first audio mode and may be reduced to less than 10 milliseconds in the second audio mode.

[0047] The delay chain 400 may be adjusted to accommodate the second audio mode by changing the audio I / O cycle time. For example, when the mobile device 104 is determined to be mounted on the HMD mount 102, the amplifier of the output speaker of the system 100 (controlled by the wireless headset processor 217 or the HMD processor 218) may request audio data from the audio application stored on the corresponding memory at a different rate. For example, instead of requesting audio data each time a first predetermined number of samples is read from the wireless headset memory 219, the wireless headset processor 217 may request audio data 206 each time a second predetermined number of samples less than the first predetermined number of samples is read from the wireless headset memory 219. By way of example, the wireless headset processor 217 may request audio data each time 1024 samples are read in the first audio mode, and may request audio data each time 64 samples are read in the second audio mode.

[0048] The change to the audio mode of the system 100 may include a change to the non-delay parameter. For example, when the system 100 enters the second audio mode, the virtual sound direction of the sound source associated with the augmented reality or virtual reality scene may be adjusted. Such an embodiment of the non-delay adjustment is described below by way of example and not limitation.

[0049] refer to Figure 6 , according to one embodiment, a flow chart of a method for adjusting non-delay parameters of a system in a second audio mode is shown. Figure 7A-7B yes Figure 6 , and therefore the relevant drawings are alternately referenced in the following description. Note that the description applies to a system 100 having a wireless headset 101 as a sound reproduction unit, but the description is equally applicable to an embodiment including a built-in headset 102 used as a sound reproduction unit.

[0050] At operation 602, the wireless headset 101 receives an audio signal 210 from the mobile device 104. The audio signal 210 may be streamed from the mobile device 104 to the wireless headset 101 in a first audio mode. At operation 604, in the first audio mode, the audio signal 210 is processed to render a sound 112 played by the wireless headset 101. Fig. 7A, the wireless headset 110 can play back various sounds. The HMD mount 102 can play sounds in a spatialized audio experience that presents virtual sounds in different directions. For example, the speaker 110 can play sounds associated with the intelligent personal assistant along a first direction 702. The first direction 702 can be the direction of the mobile device 104. In the first audio mode, the mobile device 104 can be in a handheld configuration, and thus playing the intelligent personal assistant voice 704 to the mobile device 104 in the first direction 702 may cause the user to experience the voice as coming from the mobile device 104. Other sounds, such as sounds from a music source 706, may be presented in other directions. For example, in the first audio mode, the sound from the music source 706 may be spatialized as a sound image centered within the user's head.

[0051] At operation 606, the system 100 may determine whether the mobile device 104 is mounted on the HMD mount 102. At operation 608, in response to determining that the mobile device 104 is mounted on the HMD mount 102, the audio mode of the system 100 may be changed from the first audio mode to the second audio mode. At operation 610, the audio signal 210 may be processed in the second audio mode to present the sound in a direction different from the first audio mode. For example, instead of presenting the sound associated with the intelligent personal assistant voice 704 in the first direction 702, the voice 704 may be presented in the second direction 708 having a predetermined physical relationship with the user. By way of example, the second direction 708 may be behind and / or above the user. The intelligent personal assistant voice 704 may be spatialized in the second direction 708, which may be away from the mobile device 104 mounted on the user's head. Similarly, the sound associated with the music playback may be spatialized to a different position in the second audio mode than the first audio mode. For example, the music source 706 may be presented as coming from the front of the user. The music source 706 may be segmented into different sources 706A, 706B representing, for example, different parts of an orchestra. Thus, a change in audio mode that occurs automatically when the mobile device 104 is mounted on the HMD mount 102 may affect music playback parameters, such as playback volume or sound source direction, without necessarily changing the parameters of the delay chain 400.

[0052] Another non-delay adjustment can be made to the microphone parameters of the system 100. For example, when the system 100 enters the second audio mode, the focus direction of the microphone pickup can be adjusted. Fig. 8A, according to one embodiment, a diagram of a method for adjusting microphone focus by a mobile device 104 in a second audio mode is shown. The audio mode of the system 100 may include an audio pickup focus parameter for focusing the pickup direction of a microphone 114 integrated in the mobile device 104 (or wireless headset 101 or HMD mount 102). By way of example, the microphone 114 of the mobile device 104 may be controlled to pick up speech in a specific direction. When the mobile device 104 is in a handheld configuration, the audio focus parameter may be set so that the microphones 114A and 114B receive sound from the surrounding environment. The directionality of the microphone 114 may generally be outward from the housing of the mobile device 104, for example, it may not be focused toward a specific location.

[0053] See also Figure 8B , when the mobile device 104 is mounted on the HMD mount 102, the device processor 202 can be configured to adjust the audio pickup focus parameters to focus the microphone 114 of the mobile device 104 on the preset direction 802. For example, the audio pickup focus parameters can be based on the desired position of the user's mouth to pick up speech when the mobile device 104 is mounted on the HMD mount 102. The preset direction 802 of microphone pickup can be towards the mouth.

[0054] In one embodiment, microphone adjustment may include disabling certain microphones and / or prioritizing the pickup of some microphones over other microphones. When the mobile device 104 is mounted on the HMD mount 102, the HMD mount 102 may shield the microphone 114A, and the microphone 114B may not be shielded in the mounting configuration. Therefore, when the mobile device 104 is mounted on the HMD mount 102, the microphone pickup of the microphone 114A may be disabled. In contrast, the microphone pickup of the microphone 114B may remain enabled and may be directed to pick up the user's voice. Therefore, the audio mode change that occurs automatically when the mobile device 104 is mounted on the HMD mount 102 may affect the hardware parameters of the mobile device 104 (or the wireless headset 101 or the HMD mount 102), without necessarily changing the parameters of the wireless data exchange stack.

[0055] In the above description, the present invention has been described with reference to specific exemplary embodiments of the present invention. It will be apparent that various modifications may be made to the specific exemplary aspects without departing from the broader spirit and scope of the present invention as set forth in the following claims. Accordingly, the description and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A head mounted display (HMD) system, comprising a mobile device and an HMD mounting member, wherein the HMD system is configured as follows: determining, by the mobile device, proximity between the mobile device and the HMD mount; generating, by the mobile device, an audio mode signal in response to determining proximity between the mobile device and the HMD mount; and The mobile device sends the audio mode signal to the HMD mount or wireless headset, Wherein in response to an audio mode signal received from the mobile device, the HMD mount or the wireless headset is configured to change the audio mode from a first audio mode to a second audio mode, wherein changing the audio mode includes changing non-delay parameters, wherein the non-delay parameters include virtual sound direction parameters and / or audio pickup focus parameters.

2. The HMD system of claim 1 , wherein the HMD system comprises a plurality of speakers for converting an audio signal into sound, and wherein the non-delayed parameter is the virtual sound direction parameter, such that the plurality of speakers present the sound in a first direction in the first audio mode, and present the sound in a second direction different from the first direction in the second audio mode. 3 . The HMD system of claim 2 , wherein the first direction is toward the mobile device, and wherein the second direction is away from the mobile device.

4. The HMD system of claim 2, wherein the plurality of speakers are part of the wireless headset, wherein the wireless headset is configured to: receiving the audio signal; and The plurality of speakers are driven based on the audio signal to present the sound. 5 . The HMD system of claim 4 , wherein the wireless earphones are built-in earphones of the HMD mount.

6. The HMD system according to claim 1, wherein the HMD system comprises a plurality of microphones for picking up sound, and wherein the non-delay parameter is the audio pickup focusing parameter, so that the plurality of microphones are focused in a first direction in the first audio mode and are concentrated in a second direction in the second audio mode.

7. The HMD system of claim 6, wherein the first direction is toward the surrounding environment, and wherein the second direction is toward the user's mouth.

8. The HMD system of claim 6, wherein the plurality of microphones are part of the wireless headset, wherein the wireless headset is configured to disable one or more of the plurality of microphones in response to determining that the mobile device is mounted on the HMD mount.

9. The HMD system of claim 6, wherein the plurality of microphones are part of the wireless headset, wherein the wireless headset is configured to preferentially pick up a first microphone of the plurality of microphones over a second microphone of the plurality of microphones in response to determining that the mobile device is mounted on the HMD mount. 10 . The HMD system of claim 1 , wherein the HMD system is configured to determine that the mobile device is physically paired with the HMD mount based on the audio mode signal.

11. A method performed by a mobile device of a head mounted device (HMD) system, the HMD system comprising an HMD mount, the method comprising: determining proximity between the mobile device and the HMD mount; generating an audio mode signal in response to determining proximity between the mobile device and the HMD mount; as well as sending the audio mode signal to the HMD mount or wireless headset, Wherein in response to the audio mode signal received from the mobile device, the HMD mount or the wireless headset is configured to change the audio mode from a first audio mode to a second audio mode, wherein changing the audio mode includes changing non-delay parameters, wherein the non-delay parameters include virtual sound direction parameters and / or audio pickup focus parameters.

12. The method of claim 11 , wherein the HMD system comprises a plurality of speakers for converting audio signals into sounds, and wherein the non-delayed parameter is the virtual sound direction parameter, such that the plurality of speakers present the sounds in a first direction in the first audio mode, and present the sounds in a second direction different from the first direction in the second audio mode.

13. The method of claim 12, wherein the first direction is toward the mobile device, and wherein the second direction is away from the mobile device.

14. The method of claim 12, wherein the plurality of speakers are part of the wireless headset, wherein the wireless headset is configured to: receiving the audio signal; and The plurality of speakers are driven based on the audio signal to present the sound.

15. The method of claim 14, wherein the wireless earphones are built-in earphones of the HMD mount.

16. The method of claim 11, wherein the HMD system comprises a plurality of microphones for picking up sound, and wherein the non-delay parameter is the audio pickup focusing parameter, such that the plurality of microphones are focused in a first direction in the first audio mode and are focused in a second direction in the second audio mode.

17. The method of claim 16, wherein the first direction is toward the surrounding environment, and wherein the second direction is toward the user's mouth.

18. The method of claim 16, wherein the plurality of microphones are part of the wireless headset, wherein the wireless headset is further configured to disable one or more of the plurality of microphones in response to determining that the mobile device is mounted on the HMD mount.

19. The method of claim 16, wherein the plurality of microphones are part of the wireless headset, wherein the wireless headset is further configured to preferentially pick up a first microphone of the plurality of microphones over a second microphone of the plurality of microphones in response to determining that the mobile device is mounted on the HMD mount.

20. The method according to claim 11, further comprising: A determination is made based on the audio mode signal whether the mobile device is physically paired with the HMD mount.

21. A non-transitory machine-readable medium storing instructions, which, when executed by one or more processors of a mobile device of a head mounted display (HMD) system, causes the mobile device to perform a method, the method comprising: determining proximity between the mobile device and an HMD mount of the HMD system; generating an audio mode signal in response to determining proximity between the mobile device and the HMD mount; as well as sending the audio mode signal to the HMD mount or wireless headset, Wherein in response to receiving the audio mode signal from the mobile device, the HMD mount or the wireless headset is configured to change the audio mode from a first audio mode to a second audio mode, wherein changing the audio mode includes changing non-delay parameters, wherein the non-delay parameters include virtual sound direction parameters and / or audio pickup focus parameters.

22. The non-transitory machine-readable medium of claim 21 , wherein the HMD system comprises a plurality of speakers for converting audio signals into sounds, and wherein the non-delayed parameters are the virtual sound direction parameters, such that the plurality of speakers present the sounds in a first direction in the first audio mode, and present the sounds in a second direction different from the first direction in the second audio mode.

23. The non-transitory machine-readable medium of claim 22, wherein the first direction is toward the mobile device, and wherein the second direction is away from the mobile device.

24. The non-transitory machine-readable medium of claim 22, wherein the plurality of speakers are part of the wireless headset, wherein the wireless headset is configured to: receiving the audio signal; and The plurality of speakers are driven based on the audio signal to present the sound.

25. The non-transitory machine-readable medium of claim 24, wherein the wireless earphones are built-in earphones of the HMD mount.

26. The non-transitory machine-readable medium of claim 21, wherein the HMD system comprises a plurality of microphones for picking up sound, and wherein the non-delay parameter is the audio pickup focus parameter such that the plurality of microphones are focused in a first direction in the first audio mode and are focused in a second direction in the second audio mode.

27. The non-transitory machine-readable medium of claim 26, wherein the first direction is toward a surrounding environment, and wherein the second direction is toward a user's mouth.

28. The non-transitory machine-readable medium of claim 26, wherein the plurality of microphones are part of the wireless headset, wherein the wireless headset is configured to disable one or more of the plurality of microphones in response to determining that the mobile device is mounted on the HMD mount.

29. The non-transitory machine-readable medium of claim 26, wherein the plurality of microphones are part of the wireless headset, wherein the wireless headset is configured to preferentially pick up a first microphone of the plurality of microphones relative to a second microphone of the plurality of microphones in response to determining that the mobile device is mounted on the HMD mount.

30. The non-transitory machine-readable medium of claim 21, further comprising: A determination is made based on the audio mode signal whether the mobile device is physically paired with the HMD mount.

Images