Spatial audio using single audio device

Abstract

Systems, apparatus, processes, and computer-readable media are disclosed. According to some aspects, a method of processing audio data may include obtaining, at a computing device, sensing information from an audio device outputting a spatial audio stream for a user, where the audio device includes a first audio output device and a second audio output device; determining that the second audio output device is not in use based on the sensing information; modifying the spatial audio stream to create a modified spatial audio stream based on determining that the second audio output device is not in use and the user's head pose; and providing the modified spatial audio stream to the first audio output device.

Description

Technical Field

[0001] In some examples, systems and techniques are described for spatial audio using a single audio device. Background Art

[0002] Multimedia systems are widely deployed to provide various types of multimedia communication content, such as voice, video, packet data, messaging, broadcast, etc. These multimedia systems are capable of processing, storing, generating, manipulating, and reproducing multimedia information. Examples of multimedia systems include mobile devices, gaming devices, entertainment systems, information systems, virtual reality systems, model and simulation systems, etc. These systems may employ a combination of hardware and software technologies to support the processing, storage, generation, manipulation, and reproduction of multimedia information, such as client devices, capture devices, storage devices, communication networks, computer systems, and display devices.

[0003] In some cases, portable devices such as headphones can be used with a variety of multimedia systems. True wireless listening devices do not include cables, but wirelessly receive audio data streams from wireless audio sources, which have become popular and can be used in multimedia systems, and can output spatial audio to provide an immersive experience. Summary of the invention

[0004] In some examples, systems and techniques for spatial audio using a single audio device are described. The systems and techniques can improve spatial audio by extending it for use with mono and reduce power consumption by omitting various filtering operations.

[0005] According to at least one example, a method for generating a spatial audio stream for a single audio device is provided. The method includes: obtaining sensing information at a computing device from an audio device that outputs a spatial audio stream for a user, wherein the audio device includes a first audio output device and a second audio output device; determining that the second audio output device is not in use based on the sensing information; modifying the spatial audio stream to create a modified spatial audio stream based on the determination that the second audio output device is not in use and a head posture of the user; and providing the modified spatial audio stream to the first audio output device.

[0006] In another example, an apparatus for device functionality is provided, comprising at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to: obtain sensing information from an audio device that is outputting a spatial audio stream for a user, wherein the audio device comprises a first audio output device and a second audio output device; determine that the second audio output device is not in use based on the sensing information; modify the spatial audio stream to create a modified spatial audio stream based on the determination that the second audio output device is not in use and the user's head posture; and provide the modified spatial audio stream to the first audio output device.

[0007] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which instructions, when executed by one or more processors, cause the one or more processors to: obtain sensing information from an audio device that is outputting a spatial audio stream for a user, wherein the audio device includes a first audio output device and a second audio output device; determine, based on the sensing information, that the second audio output device is not in use; modify the spatial audio stream to create a modified spatial audio stream based on the determination that the second audio output device is not in use and a head posture of the user; and provide the modified spatial audio stream to the first audio output device.

[0008] In another example, an apparatus for device functionality is provided. The apparatus includes: a component for obtaining sensing information from an audio device that outputs a spatial audio stream for a user, wherein the audio device includes a first audio output device and a second audio output device; a component for determining that the second audio output device is not in use based on the sensing information; a component for modifying the spatial audio stream to create a modified spatial audio stream based on the determination that the second audio output device is not in use and a head posture of the user; and a component for providing the modified spatial audio stream to the first audio output device.

[0009] In some aspects, the apparatus is, is part of, and / or includes a wearable device, an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head mounted device (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or a mobile handset and / or a so-called "smartphone" or another mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or component of a vehicle, another device, or a combination thereof. In some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images. In some aspects, the apparatus also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0010] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0011] The foregoing and other features and aspects will become more fully apparent by reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Exemplary aspects of the present application are described in detail below with reference to the following drawings:

[0013] Figure 1 An example wireless audio output device 100 is shown in accordance with some aspects of the present disclosure.

[0014] Figure 2 A conceptual diagram of a true wireless (TWS) audio output system 200 that may be configured to use a single audio output device according to various aspects of the present disclosure is shown.

[0015] Figure 3 is a conceptual diagram illustrating a person consuming spatial audio in accordance with some aspects of the present disclosure.

[0016] Figure 4 A conceptual example of an application executed by a host device according to some aspects of the present disclosure is shown.

[0017] Figure 5A , 5B , 5C, and 5D illustrate examples of spatial audio systems and methods of determining when an audio output device is not in use according to some aspects of the present disclosure.

[0018] Figure 6 is a flow chart illustrating an example of a method for processing audio according to certain aspects of the present disclosure.

[0019] Figure 7 A block diagram of an example host device configured to generate a spatial audio stream for a single audio device is shown in accordance with some aspects.

[0020] Figure 8 is a diagram illustrating an example of a system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0021] Certain aspects of the present disclosure are provided below. Some of these aspects can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of various aspects of the application. However, it is obvious that various aspects can be practiced without these specific details. The drawings and description are not intended to be restrictive.

[0022] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example aspects will provide an enabling description for implementing the example aspects for those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0023] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example aspects will provide an enabling description for implementing the example aspects for those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0024] The terms "exemplary" and / or "example" are used herein to mean "serving as an example, instance, or illustrative illustration." Any aspect described herein as "exemplary" and / or "example" is not necessarily to be construed as preferred or advantageous over other aspects. Likewise, the term "aspects of the disclosure" does not require that all aspects of the disclosure include the discussed feature, advantage, or mode of operation.

[0025] Spatial audio creates a three-dimensional (3D) virtual auditory space that allows a user wearing an auxiliary device with inertial sensors to accurately locate the location of sound sources in the 3D virtual auditory space while watching a movie, playing a video game, or interacting with augmented reality (AR) or virtual reality (VR) content on a source device (e.g., a tablet computer). Spatial audio allows a person listening to the audio (referred to herein as a listener) to accurately locate the audio source within the 3D environment. Spatial audio includes channel-based, binaural, or object-based audio technologies, protocols, standards, formats, or any other audio rendering concepts or technologies that provide a 3D virtual auditory space.

[0026] Audio devices that implement spatial audio must include various sensors, such as an inertial measurement unit (IMU), to detect the motion of the listener that can modify the audio stream, and determine the listener's head pose, which then modifies the audio sources within the audio stream. True wireless (TWS) earbuds and headphones have recently implemented spatial audio features to allow for an immersive experience for the listener when both the earbuds or headphones are attached to the listener.

[0027] Spatial audio naturally requires that the left audio device and the right audio device provide stereo audio streams (e.g., a left audio stream and a right audio stream), and the spatial audio can be interrupted when one of the left audio device and the right audio device is separated from the listener. However, there are situations where the listener may want to hear spatial audio while using a single audio device. For example, many people have limited ability to hear in a single ear, or a single audio device may be charging. In another example, a person may want to monitor external audio by having only a single audio device provide audio to monitor external audio cues (such as a doorbell, an open door, etc.). In some cases, different people can be connected to a single audio device, such as a first person listening to the left audio channel and a second person listening to the right audio channel.

[0028] In some aspects, systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively referred to herein as "systems and techniques") are described for spatial audio using a single audio device. For example, an electronic device may obtain sensing information from an audio device including a first audio output device and a second audio output device. The audio device may output a spatial audio stream for a user. In some aspects, the audio device may be a pair of wireless earbuds capable of providing stereo sound to a listener, wherein the first audio output device includes one earbud and the second audio output device includes a second earbud. In other examples, the audio device may be a headset or an XR device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, etc.) including an earbud or a headset. The electronic device may determine that the second audio output device is not in use based on the sensing information. For example, the sensing information may identify or indicate to the electronic device that the distance from the wireless earbud (e.g., the left earbud or the right earbud) to the person is greater than a threshold distance (e.g., 5 centimeters), and based on the sensing information, the electronic device may determine that the wireless earbud is not in use. The electronic device may modify the spatial audio stream based on determining that the second audio output device is not in use. The electronic device provides the modified spatial audio stream to the first audio output device.

[0029] In one illustrative aspect, an electronic device may modify spatial audio filtering based on a single audio output device in use (e.g., the first audio output device from the above example). In some cases, filtering related to time differences and channel differences may be omitted (or not performed) when modifying a spatial audio stream. An electronic device may provide a spatial audio stream that is mono and can be used by a single audio output device. In another illustrative aspect, the disclosed methods, systems, and techniques may be used to enable multiple listeners to each use a single audio output device with a mono spatial audio stream.

[0030] Additional details and aspects of the disclosure are described in more detail below with reference to the accompanying drawings.

[0031] Figure 1 An example wireless audio output device 100 is shown according to some aspects of the present disclosure. The wireless audio output device 100 provides a single audio channel, either a left channel or a right channel, and can operate with another wireless audio output device (not shown) to provide two audio channels (e.g., a left channel and a right channel). Each

[0032] According to some embodiments, each wireless audio output device 100 may include a housing 105 formed by a body 110 and a stem 115 extending from the body 110. In some aspects, the housing 105 may be formed by an integral external structure such as molded plastic. The body 110 may include an inwardly facing microphone 120 and an outwardly facing microphone 125. The outwardly facing microphone 125 may be positioned in an opening defined by portions of the body 110 and the stem 115. By extending into both the body 110 and the stem 115, the microphone 125 may be large enough to receive sound from a wider area close to the listener. In some embodiments, the housing 105 may define an acoustic port that may guide the sound from the internal audio driver out of the housing 105 and into the listener's ear canal. In other embodiments, the wireless audio output device 100 may include a deformable earplug that may be inserted into the listener's ear canal so that the wireless listening device can be configured as an in-ear hearing device.

[0033] In one example, the stem 115 has a substantially cylindrical configuration and a planar region 130 that does not follow the curvature of the cylindrical configuration. The planar region 130 can indicate an area where the wireless listening device can receive listener input. For example, in some embodiments, listener input can be input by squeezing the stem 115 at the planar region 130. In some embodiments, in addition to or in lieu of pressure sensing capabilities, the planar region 130 can include a touch-sensitive surface that allows the listener to input touch commands, such as contact gestures. The stem 115 can also include a touch-sensitive surface for connecting to a charging case (e.g., Figure 2 The electrical contacts 135 and 140 contact corresponding electrical contacts in the charging box 250).

[0034] The wireless audio output device 100 may include several features that may enable the listener to comfortably wear the device for an extended period of time or even throughout the day. The shape and size of the housing 105 may be designed to fit firmly between the tragus and antitragus of the listener's ear, so that the portable listening device is not easy to fall out of the ear even when the listener is exercising or otherwise actively moving. Its function may also enable the wireless audio output device 100 to provide an audio interface to a host device (e.g., host device 210), so that the listener may not need to utilize the graphical interface of the host device. The audio output device 100 may be complex enough to enable the listener to perform daily operations from the host device only by interacting with the wireless audio output device 100. This may create further independence from the host device by not requiring the listener to physically interact with the host device and / or watch the display screen of the host device, particularly when the function of the wireless audio output device 100 is combined with the voice control capability of the host device. Therefore, the wireless audio output device 100 may achieve a truly wireless and truly hands-free experience for the listener.

[0035] The wireless audio output device 100 may also include various components that are not visually perceptible. For example, the wireless audio output device 100 may include at least one sensor for detecting various aspects of the device. Exemplary illustrative aspects of the device include the state of the device (e.g., whether the wireless audio output device 100 is attached to a person), posture information related to the listener, biometric information (e.g., the listener's temperature), etc. At least one sensor of the wireless audio output device 100 can be configured to output posture information identifying the direction of the listener's head relative to a neutral position (e.g., a neutral head position). The posture information can be used by a host device, and the host device can be configured to change the audio stream presented to the wireless audio output device 100 to provide a spatial audio stream that provides a 3D virtual auditory space

[0036] Figure 2 A conceptual diagram illustrating a TWS audio output system 200 that can be configured to use a single audio output device according to various aspects of the present disclosure. The TWS audio output system 200 includes a host device 210, a pair of audio output devices 230 (eg, a left audio output device 230 and a right audio output device 230), and a charging box 250.

[0037] The host device 210 Figure 2 2 is depicted as a mobile communication device (e.g., a smart phone), but may be any electronic device that can transmit audio data to a wireless audio output device (e.g., wireless audio output device 100). Other non-limiting examples of suitable host devices 210 include laptop computers, desktop computers, tablet computers, smart watches, audio systems, video players, etc.

[0038] In some aspects, each audio output device 230 can receive and generate sounds to provide an enhanced user interface for the host device 210. The audio output device 230 can include a processor 231 that executes computer-readable instructions stored in a memory (not shown) for performing a plurality of functions of the audio output device 230. In some examples, the processor 231 can be one or more suitable computing devices, such as a microprocessor, a computer processing unit (CPU), a digital signal processing unit (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc.

[0039] The processor 231 may be operably coupled to the interface 232, the communication system 233, and the sensor system 234 to enable the audio output device 230 to perform one or more functions. For example, the interface 232 may include a driver (e.g., a speaker) for outputting sound to a user, one or more microphones for inputting sound from the environment or the user, one or more light emitting diodes (LEDs) for providing visual notifications to the user, a pressure sensor or a touch sensor (e.g., a resistive or capacitive touch sensor) for receiving user input, and / or any other suitable input or output device. The communication system 233 may include wireless and wired communication components for enabling the audio output device 230 to send and receive data / commands from the host device 210. For example, the communication system 233 may include a circuit for the audio output device 230 to communicate with the host device 210 via a wireless link 260, and the wireless link 260 is implemented by a standard (e.g., Bluetooth, WiFi Direct, Zigbee, etc.) or a proprietary communication link. The communication system 233 may also enable the audio output device 230 to communicate wirelessly with the charging box 250 via a wireless link.

[0040] In some aspects, sensor system 234 may include proximity sensors (eg, optical sensors, capacitive sensors, radar, etc.), accelerometers, microphones, and any other type of sensor that may measure parameters of an external entity and / or environment.

[0041] The audio output device 230 may also include a battery 235 (e.g., a suitable energy storage device, such as a lithium-ion battery, etc.) that is capable of storing energy and releasing the stored energy to operate the audio output device 230. The released energy can be used to power the electrical components of the audio output device 230. The battery 235 can be a rechargeable battery and allows charging as needed to replenish the stored energy. For example, the battery 238 can be coupled to a battery charging circuit (not shown) that is operably coupled to receive power from a charging box interface (not shown). The housing interface may include electrical contacts to electrically couple the audio output device 230 to the charging box 250. In some aspects, the audio output device 230 can receive power from the charging box 250 via electrical contacts within the charging box. In some aspects, the audio output device 230 can be changed via an inductive communication interface via a wireless power receiving coil within the charging box 250.

[0042] The charging case 250 may include a battery (not shown) that can store energy and release the energy to the power circuit to charge the battery 235 of the audio output device 230. As described above, the audio output device 230 may include electrical contacts (e.g., electrical contacts 135 and electrical contacts 140) that can transmit power to the audio output device 230 via a wired electrical connection between the contacts in the charging case. In some cases, the charging case 250 may be configured to facilitate the establishment of a wireless connection between the host device 210 and the audio output device 230.

[0043] The charging box 250 may also include a processor (not shown) and a communication system (not shown). The processor may be one or more processors, ASICs, FPGAs, microprocessors, etc., for operating the charging box 250. The processor may be coupled to the earbud interface and may control the charging function of the charging box 250 to charge the battery 235 of the audio output device 230, and the processor may also be coupled to the communication system for operating the interactive function of the charging box with other devices (including the audio output device 230). In one example, the communication system of the charging box 250 includes a Bluetooth component or any other suitable wireless communication component that wirelessly sends and receives data with the communication system 233 of the audio output device 230. To this end, the charging box 250 and each audio output device 230 may include an antenna formed by a conductor to send and receive electromagnetic signals.

[0044] The charging case 250 may also include a user interface (e.g., a button, a speaker, a light emitter such as an LED, etc.) that may be operably coupled to the processor to alert the user of various notifications. For example, the user interface may include a speaker that may emit an audible noise that can be heard by the user and / or one or more LEDs or similar lights that may emit light that can be seen by the user. For example, the charging case 250 may output audio or light to indicate whether the at least one audio output device 230 is being charged by the charging case 250, or to indicate that the charging case battery is low or is being charged.

[0045] The host device 210 is configured to connect to the audio output device 230 and provide audio information. The audio output device 230 may also provide information in some contexts, such as whether the audio output device 230 is attached to the listener. In some cases, the host device 210 may include a processor (not shown) coupled to a battery (not shown) and a host storage body (not shown) containing lines of code that can be executed by a host computing system (not shown) for operating the host device 210. The host device 210 may also include a host sensor system (e.g., accelerometer, gyroscope, light sensor, etc.) for allowing the host device 210 to sense the environment, and a host user interface system (e.g., display, speaker, button, touch screen, etc.) for outputting information to the user and receiving input from the user. In addition, the host device 210 may also include a communication system for allowing the host device 210 to send and / or receive data, such as wireless fidelity (WiFi), long term evolution (LTE), code division multiple access (CDMA), global system for mobile (GSM), Bluetooth, etc. The communication system of host device 210 can also communicate with communication system 233 via a wireless communication link, so that host device 210 can send audio data to audio output device 230 to output sound, and receive data from audio output device 230 to receive user input. The communication link can be any suitable wireless communication line, such as a Bluetooth connection. By enabling communication between host device 210 and audio output device 230, audio output device 230 can enhance the user interface of host device 210.

[0046] Figure 3 is a conceptual diagram illustrating a listener 300 consuming spatial audio according to some aspects of the present invention. In some aspects, Figure 3 The example playback system for spatial audio shown is a stereo speaker setup that includes an audio output device 310 and an audio output device 320 that are placed in front of the left and right sides of the listener 300. Figure 3 A speaker is shown, but the audio output device may also be a headset or earphone (eg, wireless audio output device 100). Typically, the speaker 302 is and The angles are placed on the circle, and the width of the auditory spatial image perceived when listening to such a stereo playback system is roughly limited to the area between and behind the two speakers.

[0047] In some aspects, stereo loudspeaker playback relies on the perceptual phenomenon of summation localization, and auditory events can be made to appear anywhere between the loudspeaker pair in front of the listener by controlling the inter-channel time difference (ICTD) and / or inter-channel level difference (ICLD). For example, when only amplitude differences (e.g., ICLD) are introduced between the loudspeaker pairs, phase differences can be generated between the ears, or interaural time differences (ITD) similar to what occurs in natural listening can be generated.

[0048] In some aspects, ICTD is a phase difference, which is a time difference between an audio source relative to a left channel and a right channel, and ICLD is an intensity difference between an audio source relative to a left channel and a right channel. For example, an object to the left of the listener 300 will have a higher intensity (e.g., power spectral density (PSD)) on the left channel (e.g., provided to the left audio output device 230) output by an audio output device 310 located to the left of the listener than on the right channel (e.g., provided to the right audio output device 230). In some aspects, the left channel is output by an audio output device 310 located to the left of the listener's neutral position, and the right channel is output by an audio output device 320 located to the right of the listener's neutral position. For example, the audio output devices 310 and

[0049] In some aspects, ICTD introduces a phase delay and ICLD introduces a strength difference. For example, a source located on the left side results in a stronger signal to the left of the listener than to the right. In other words, the ICLD of two audio output devices is based on the source angle. When the audio is output through the audio output system (for example, speakers, Figure 2 When the audio signals are played back by an audio output device 230 in the audio source, the auditory events will be heard at the same angle as the original source. Related angles Appear.

[0050] In some aspects, spatial audio for a stereo audio output system can be generated by mixing multiple separately available source signals (e.g., multi-track recordings). ICLD is traditionally implemented in audio streams, which may also be referred to as amplitude panning. The concept of amplitude panning is described in Figure 3 Using a signal scaling factor and The audio output device 310 and the audio output device 320 are used to reproduce the sound source s(n). When amplitude panning is applied, the perceived direction of the auditory event roughly follows the stereo sine law, as shown in Equation 1 below.

[0051] (Equation 1)

[0052] in is the angle between the forward axis and the two speakers, is the corresponding angle of the auditory event, and and is the scaling factor that determines the ICLD.

[0053] In some aspects, the stereo tangent law improves the head model compared to the stereo sine law under different listening conditions. In some aspects, the panning law is only an approximation because the perceived direction of auditory events is It also depends on the signal characteristics, such as frequency and signal bandwidth. To this end, spatial audio streams usually implement various filters, such as ICLD filters and ICTD filters, to create spatial audio streams.

[0054] Spatial audio can also be reproduced by a different technique called delay panning, which uses ICTD to create spatial audio. Delay panning, which is often difficult to reproduce in analog systems, is the main reason why ICTD panning is not usually used. In some cases, ICLD may be preferred over ICTD because ICLD is more robust to non-ideal conditions. In some aspects, ICTD can be used when ideal conditions exist, such as when a user is wearing headphones.

[0055] Modern methods of spatial audio can use head-related transfer functions (HTRF), ICLD, ICTD, and inter-channel coherence (ICC) to achieve spatial audio to produce superior results. In some aspects, HTRF transforms audio based on how the human ear perceives audio, and ICC is the relationship of the left channel relative to the right channel. When a listener wears an audio output device (such as headphones or earbuds), the audio output device can be configured to recognize the listener's head posture to identify their direction. HTRF, ICLD, ICC, and ICTD filters can be applied to the audio stream to create a spatial audio stream that changes how the listener perceives the sound auditorily. In some cases, the head posture can be provided to a host device (e.g., host device 210), and the audio stream generated based on the application or function executed in the host device can be modified to create a spatial audio stream. In some cases, the audio stream can include position information associated with objects within the application or function (e.g., a listener playing a 3D game), and the host device can modify the audio produced by the object based on the head posture of the listener 300 relative to the position of these objects.

[0056] Figure 4Conceptual examples of applications executed by a host device according to some aspects of the present disclosure are shown.In some aspects, a 3D application is shown to depict spatial audio that can be rendered by a host device (eg, host device 210).

[0057] exist Figure 4 In an illustrative example of , the application may be a 3D game for simulating a race (e.g., in VR presented by a head mounted device). The audio generated by multiple objects within the 3D game may include positional information. For example, audio from first car 402 will include information identifying the position of first car 402 as being in front of and to the left of a user of the host device, and audio from second car 404 will include information identifying the position of second car 404 as being in front of and to the right of the user of the host device. In this example, airplane 406 may be flying through the scene, and the audio generated by airplane 406 may include information about its position relative to the user of the host device (e.g., the listener).

[0058] In some aspects, a user (e.g., a listener) of a host device may be consuming audio using an audio output device that can determine the user's head posture. In that case, the audio produced by the first car 402, the second car 404, and the airplane 406 can be rendered (e.g., mixed) into a stereo soundtrack based on the user's head posture to provide a spatial audio experience. As described above, HTRF, ICLD, ICTD, ICC, and other effects can be applied to audio sources based on the location of the object within the application.

[0059] For example, as the user changes their head position, the audio generated by each of first car 402, second car 404, and airplane 406 will change relative to the user's head posture. The host device can mix the audio generated by each of first car 402, second car 404, and airplane 406 into a stereo audio stream that provides a spatial effect and provides a left channel audio stream to a left audio output device and a right channel audio stream to a right audio output device based on the user's head posture.

[0060] Figure 5A , 5B , 5C, and 5D illustrate examples of spatial audio systems and methods for determining when an audio output device is not in use according to aspects of the present disclosure. Figure 5A A host device 502 is shown for providing spatial audio to a left audio output device 504 and a right audio output device 506 to a listener 508 over a wireless communication link.

[0061] Figure 5BA listener 508 is shown removing the left audio output device 504 from their ear. The left audio output device 504 includes at least one sensor configured to detect when the listener 508 inserts the left audio output device 504 into their ear or removes the left audio output device 504 from their ear. For example, the left audio output device 504 can include a proximity sensor that detects that a distance 510 from the left audio output device 504 to the listener's head is greater than a threshold value (e.g., 10 centimeters).

[0062] In response to detecting that the left audio output device 504 has been inserted or removed, the left audio output device 504 may determine that the left audio output device 504 is in use (e.g., if the distance is less than a threshold) or is no longer in use (e.g., if the distance is greater than a threshold). The left audio output device 504 may send a message to the host device 502 indicating whether the left audio output device 504 is in use. In one example, the message may indicate that the left audio output device 504 is offline or will be transitioned to an offline state. In some commercially available products, the host device 502 may interrupt the spatial stream based on detecting that one of the audio output devices is not in use.

[0063] In some aspects, host device 502 can be configured to detect that listener 508 is using a single audio output device and can provide a spatial audio stream configured for the single audio output device that provides a single audio channel (eg, a mono audio channel).

[0064] In one exemplary illustrative aspect, the host device 502 is configured to determine whether a source (e.g., an application executing on the host device) includes position information. For example, a music playback application that provides a stereo audio track may not include position information. In another example, a VR game may provide an audio stream from objects within the VR game that identifies the locations of those objects within the VR game. The host device 502 may be configured to process audio differently based on whether the audio includes position information or traditional stereo audio.

[0065] In some aspects, if the audio does not include positional information (e.g., stereo audio), the host device can mix the left and right channels from the source into a mono audio stream and assign a default position to the mono audio stream within the 3D space. The host device can then position the mono audio stream based on the user's head pose and the default position (e.g., distance from a neutral head position). ) applies an ICLD filter to a mono audio stream to produce a spatial audio stream. In this illustrative aspect, no ICTD information and ICC information are used in the creation of the spatial audio stream. For example, ICTD filtering and ICC filtering for creating the spatial stream are omitted. In addition, binaural cue filtering is also omitted from the creation of the spatial audio stream.

[0066] In some other aspects, if the audio includes position information (e.g., audio from a 3D game or other application), the host device can obtain the position information associated with the object generating the audio and apply the ICLD filter to each object generating the audio. The host device can omit any ICTD filter and ICC filter used to create the spatial stream. In addition, binaural cue filtering can also be omitted from the creation of the spatial audio stream. An example of binaural cue filtering can be a game runtime sound, such as a gunshot in a game, and the binaural cue filtering outputs the gunshot from a position that the wearer of the audio device can determine. Another example is a game runtime voice, such as an enemy talking, and the binaural cue filtering outputs the voice so that the wearer can determine the position of the speaker. In another example, the application can be an XR music video, and the singer in the music video is moving position, and the binaural cue filtering can change the singer's voice based on the singer's position and the user's head position. After ICLD filtering, the host is configured to determine the sound scaling factor to be applied to each object based on the listener's head posture and mix the audio stream into the spatial audio stream.

[0067] In this illustrative example, the spatial audio stream may be a single channel of audio that will be provided to an active audio output device and provide audio to the listener. For example, if the listener removes the audio output device from their left ear, the spatial audio stream may include the right channel and the left channel may be omitted.

[0068] Figure 5C Another example of a spatial audio system based on a host device 502 is shown, which provides spatial audio to an audio output device 512 (such as headphones) that is configured to output stereo audio. Figure 5C As shown, the audio output device 512 covers both ears of the listener 508. However, the audio output device 512 can acoustically isolate the listener 508 so that the listener 508 cannot perceive other sounds, such as a doorbell. Figure 5D As shown, the listener 508 can configure the audio output device 512 to cover a single ear to allow the listener 508 to perceive other auditory cues. In this case, the audio output device 512 can include a sensor that can identify that the left audio output channel is not in use.

[0069] In some aspects, host device 502 can be configured to receive information from audio output device 512 indicating that only a single channel of a spatial audio stream is being listened to (e.g., consumed by listener 508), and the host device can provide a spatial audio stream configured for the single channel. As described above, the spatial audio stream of the single channel can continue to provide the immersive experience desired by listener 508.

[0070] Figure 66 is a flowchart illustrating an example of a method 600 for processing audio according to certain aspects of the present invention. The method 600 may be performed by a computing device configured to provide an audio stream, such as a mobile wireless communication device, an extended reality (XR) device (e.g., a VR device, an AR device, an MR device, etc.), a network-connected wearable device (e.g., a network-connected watch), a vehicle or a component or system of a vehicle, a laptop computer, a tablet computer, or another computing device. In one illustrative example, the following description of the method 600 is provided below. Figure 8 The described computing system 800 may be configured to perform all or part of the method 600 .

[0071] At box 605, the computing system may obtain sensing information from an audio device that is outputting a spatial audio stream for a user, wherein the audio device includes a first audio output device and a second audio output device. In one illustrative aspect, the audio device may be a pair of headphones, or may be a TWS headset. The sensing information may indicate that the second audio output device is decoupled from the user, the first audio output device, or the computing device. For example, the second audio output device may be a single wireless headset associated with a pair of wireless headsets. In another example, a single audio output device may be configured to be connected to the computing system in a variety of ways, such as a parent-child relationship associated with the pair of wireless headsets, or each wireless headset may be connected to the computing system.

[0072] The second audio output device may include various sensors, such as proximity sensors and pressure sensors, and provide sensing information to the computing system. For example, the computing system may obtain the sensing information by receiving the sensing information from the proximity sensor of the second audio output device. On the other hand, the computing system may obtain the sensing information by receiving the sensing information from the pressure sensor of the first audio output device or the second audio output device. In another example, the audio device may be an earphone that may detect the rotation of the earphone and determine that the rotation indicates that one earphone is not positioned on the user's ear.

[0073] At box 610, the computing system may determine that the second audio output device is not in use based on the sensed information. In some aspects, the computing system may detect the decoupling of the second audio output device from the user, the first audio output device, or the computing device based on the sensed information. For example, the second audio output device may be disposed in the ear canal of the user, and the sensed information may indicate that the wearer has removed the earphone from the ear canal. In another illustrative aspect, the first audio output device and the second audio output device may have a parent-child relationship, and the first audio output device may provide the computing system with information that the second audio output device is disconnected or in a standby state. In another illustrative aspect, the computing system may determine that the distance between the second audio output device and the user's head is greater than a threshold distance.

[0074] In some other aspects, the computing system may determine the signal strength of the signal from the audio device, and determine that the second audio output device is separated from the user's head based on the signal strength. For example, the audio device may output a signal for measuring the distance, and the measured value of the signal may indicate that the audio device is separated from the user's head. In some other aspects, the computing system may use an ML model to identify multiple parameters to indicate that the second audio output device should be disabled. In another illustrative aspect, determining that the first audio output device is not in use includes receiving a message from the first audio output device or the second audio output device indicating that the first audio output device or the second audio output device is not in use. For example, the TWS earbuds can be removed from the user's ear canal, and the TWS earbuds can detect the removal and report the removal to the computing device.

[0075] At block 615, the computing system may modify the spatial audio stream based on determining that the second audio output device is not in use and the user's head posture to create a modified spatial audio stream. In one illustrative aspect of block 615, the computing system may obtain motion information related to the user's motion from at least the first audio output device. For example, the first audio device may include a motion sensor that tracks the position of the wearer's head.

[0076] The computing system may modify the spatial audio stream at block 615 based on determining that the second audio output device is not in use and the user's head posture. In one aspect, the audio source associated with the spatial audio stream provides position information associated with one or more objects configured to produce audio, such as a game or a VR simulator. For example, the computing system may obtain position information associated with each of the one or more objects. For example, the position information may be associated with an object emitting a sound in a game, such as the position of a car in a racing game, or an alert from a sensor in a flight simulator.

[0077] At block 615, the computing system may also apply at least one spatial filter to each of the one or more objects and mix audio associated with each of the one or more objects into the spatial audio stream. To apply the spatial filter, the computing system may determine that the second audio output device corresponds to a left channel or a right channel, determine an angle associated with the object based on determining that the second audio output device corresponds to the left channel or the right channel, and determine a sound scaling factor based on the angle associated with the object, an inter-channel level difference of the object relative to the left channel and the right channel, and a head posture of the user. In this aspect, inter-channel time difference information and inter-channel coherence information are omitted from the modification of the spatial audio stream.

[0078] In another illustrative aspect of block 615, an audio source associated with the spatial audio stream does not provide position information associated with one or more objects configured to produce audio. For example, the audio source may be an audio stream or a video file that does not include position information. In this aspect, to modify the spatial audio stream, the computing system may mix the left and right channels from the audio source into a mono audio stream, assign a default position to the mono audio stream, and apply an inter-channel level difference filter to the mono audio stream based on the user's head pose and the default position to generate the spatial audio stream. Inter-channel time difference information and inter-channel coherence information may be omitted from the modification of the spatial audio stream.

[0079] In another illustrative aspect of block 615, when the source provides the position information, to modify the spatial audio stream, the computing system may obtain the position information associated with each object generating audio from the one or more objects, exclude at least one binaural cue filter, exclude at least one filter associated with inter-channel time difference or inter-channel coherence, apply an inter-channel level difference filter to each object generating audio from the one or more objects, and mix the audio associated with each object generating audio from the one or more objects into the spatial audio stream. In this aspect, the computing system may apply the inter-channel level difference filter to the object generating audio, identify whether the second audio output device corresponds to the left channel or the right channel, determine an angle associated with the object based on whether the second audio output device corresponds to the left channel or the right channel, and determine a sound scaling factor based on the angle associated with the object, the inter-channel level difference of the object relative to the left channel and the right channel, and the head posture of the user.

[0080] In another illustrative aspect of block 615, when the source does not provide position information, to modify the spatial audio stream, the computing system may mix the left channel and the right channel from the source into the spatial audio stream, assign a default position to the spatial audio stream, exclude at least one binaural cue filter, exclude at least one filter associated with inter-channel time difference or inter-channel coherence,

[0081] At block 620 , the computing system may provide the modified spatial audio stream to the first audio output device.

[0082] Figure 7 A block diagram of an example host device 700 configured to generate a spatial audio stream for a single audio device according to some aspects is shown. In some aspects, the host device 700 is configured to perform one or more of the methods described above.

[0083] The host device 700 may include a head gesture module 702, an audio control module 704, a spatial audio mixing module 706, and an auxiliary communication module 708. Portions of one or more of the modules 702, 704, 706, and 708 may be implemented at least in part in hardware or firmware. For example, the auxiliary communication module 708 may be implemented at least in part by one or more modems (e.g., a Bluetooth modem). In some aspects, at least some of the modules 702, 704, 706, and 708 are implemented at least in part as software stored in a memory. For example, portions of one or more of the modules 702, 704, 706, and 708 may be implemented as non-transitory instructions (or "code") that can be executed by at least one processor to perform the functions or operations of the corresponding module.

[0084] The head posture module 702 may be configured to receive information related to the user's head posture. For example, the wireless audio output device may detect the user's head posture information using an IMU and send the head posture information to the host device 700.

[0085] The audio control module 704 is configured to control the audio output by one or more audio sources (such as applications). The audio control module 704 can be configured to determine whether the audio output includes location information associated with the audio source. The audio control module 704 can also receive information provided from a wireless audio output device, which indicates the status of the wireless audio output device, such as whether the wireless audio output device is in use or about to go offline.

[0086] The spatial audio mixing module 706 is configured to receive the audio streams and any position information, and to mix the audio streams based on the state of the audio output device. For example, when a single audio output device reproduces a single channel of audio, such as when the left audio output device is not attached to the user, the spatial audio mixing module 706 can be configured to control spatial audio generation as described above. For example, the spatial audio mixing module 706 can be configured to omit ICC filtering, ICTD filtering, and binaural cue filtering.

[0087] The auxiliary communication module 708 is configured to send and receive messages from an audio output device, and can be configured to provide a spatial audio stream to at least one audio output device that is providing audio. In some cases, the auxiliary communication module 708 can be configured to be related to wireless communication, but the auxiliary communication module 708 can also communicate with an audio output device that is electrically connected to the host device 700.

[0088] In some examples, the processes described herein (e.g., method 600 and / or other processes described herein) can be performed by a computing device or apparatus. In one example, method 600 can be performed by a computer having Figure 8The computing architecture of the computing system 800 is shown as being implemented by a computing device.

[0089] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a television, and / or any other computing device having resource capabilities to perform the methods described herein (including method 600). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the methods described herein. In some examples, the computing device may include a display, a network interface configured to transmit and / or receive data, any combination thereof, and / or other components. The network interface may be configured to transmit and / or receive IP-based data or other types of data.

[0090] Components of a computing device may be implemented in circuits. For example, a component may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0091] Method 600 is illustrated as a logical flow diagram, the operations of which represent a series of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the operation. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the method.

[0092] Method 600 and / or other methods or processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, through hardware, or a combination thereof. As described above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes multiple instructions that may be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0093] Figure 8 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. In particular, Figure 8 An example of a computing system 800 is shown, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using connection 805. Connection 805 may be a physical connection using a bus, or a direct connection into processor 810, such as in a chipset architecture. Connection 805 may also be a virtual connection, a networked connection, or a logical connection.

[0094] In some aspects, computing system 800 is a distributed system, where the functionality described in the present disclosure may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components, each component performing some or all of the functionality for which the component is described. In some aspects, a component may be a physical or virtual device.

[0095] The example computing system 800 includes at least one processing unit (CPU or processor) 810 and connections 805 coupling various system components including system memory 815, such as ROM 820 and RAM 825, to the processor 810. The computing system 800 may include a cache 812 of high-speed memory directly connected to the processor 810, in close proximity to the processor 810, or integrated as part of the processor 810.

[0096] Processor 810 may include any general purpose processor and hardware services or software services, such as services 832, 834, and 836 stored in storage device 830, configured to control processor 810 as well as special purpose processors where the software instructions are incorporated into the actual processor design. Processor 810 may be essentially a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0097] To enable user interaction, the computing system 800 includes an input device 845, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 800 can also include an output device 835, which can be one or more of a plurality of output mechanisms. In some instances, a multimodal system can enable a user to provide multiple types of input / output to communicate with the computing system 800. The computing system 800 can include a communication interface 840, which can generally govern and manage user input and system output. The communication interface can use a wired and / or wireless transceiver to perform or facilitate the reception and / or transmission of wired or wireless communications, including the use of an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Proprietary wired ports / plugs, Wireless signal transmission, BLE wireless signal transmission, Wireless signal transmission, RFID wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 WiFi wireless signal transmission, WLAN signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), IR communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 840 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 800 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US-based GPS, the Russian-based Global Navigation Satellite System (GLONASS), the China-based Beidou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the basic features herein may be readily substituted for improved hardware or firmware arrangements as such are developed.

[0098] The storage device 830 may be a non-volatile and / or non-transitory and / or computer-readable memory of a device and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a magnetic cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / tape stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk read-only memory (CD-ROM) disc, a rewritable compact disk (CD) disc, a rewritable compact disk (CD) disc, a digital video disk (DVD) disc, a Blu-ray disc (BDD) disc, a holographic Optical disc, another optical media, Secure Digital (SD) card, Micro Secure Digital (microSD) card, MemoryStick® card, Smart card chip, EMV chip, Subscriber Identity Module (SIM) card, Mini / Micro / Nano / Pico SIM card, another integrated circuit (IC) chip / card, RAM, Static RAM (SRAM), Dynamic RAM (DRAM), ROM, Programmable Read Only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash EPROM (FLASHEPROM), Cache Memory ( ), resistive random access memory (RRAM / ReRAM), phase change memory (PCM), spin transfer torque RAM (STT-RAM), another memory chip or tape cartridge, and / or a combination thereof.

[0099] Storage device 830 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 810, enables the system to perform a function. In some aspects, a hardware service that performs a particular function may include a software component stored in a computer-readable medium, which is combined with the necessary hardware components (e.g., processor 810, connection 805, output device 835, etc.) to perform the function. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored and does not include carrier waves and / or transient electronic signals propagated wirelessly or via wired connections. Examples of non-transitory media may include, but are not limited to, disks or tapes, optical storage media (e.g., CDs or DVDs), flash memory, memory, or memory devices. The computer readable medium may have stored thereon code and / or machine executable instructions, which may represent a process, function, subprogram, program, routine, subroutine, module, software package, category, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by transmitting and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be transmitted, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0100] In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, a computing device may include a display, one or more network interfaces configured to transmit and / or receive data, any combination thereof, and / or other components. The one or more network interfaces may be configured to transmit and / or receive wired and / or wireless data, including data in accordance with 3G, 4G, 5G, and / or other cellular standards, data in accordance with Wi-Fi (802.11x) standards, data in accordance with WLAN standards, and / or other wireless communication protocols. Standard data, data according to IP standards and / or other types of data.

[0101] Components of a computing device may be implemented in circuits. For example, a component may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0102] In some aspects, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0103] Specific details are provided in the above description to provide a thorough understanding of the aspects and examples provided herein. However, it will be appreciated by those skilled in the art that these aspects can be practiced without these specific details. For clarity, in some cases, the present technology can be presented as including separate functional blocks, which include functional blocks comprising a combination of a device, a device component, a step or a routine in a method implemented in software, or a combination of hardware and software. In addition to those components shown in the accompanying drawings and / or described in this article, additional components can be used. For example, circuits, systems, networks, processes, and other components can be shown as components in block diagram form, so as not to obscure these aspects with unnecessary details. In other examples, known circuits, processes, algorithms, structures, and techniques can be shown without unnecessary details, so as to avoid blurring these aspects.

[0104] Various aspects may be described above as a process or method, which is depicted as a flow chart, flow diagram, data flow diagram, structure diagram or block diagram. Although a flow chart may describe an operation as a sequential process, many operations may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the return of the function to call the function or the main function.

[0105] The processing procedures and methods according to the above examples can be implemented using computer executable instructions stored in or otherwise available from computer readable media. Such instructions may include, for example, instructions and data that enable or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a specific function or group of functions. Parts of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices, etc.

[0106] Devices implementing the processes and methods disclosed herein may include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may be implemented in any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein may also be embodied in peripheral devices or add-on cards. As another example, such functions may also be implemented in different chips or different processes executed in a single device on a circuit board.

[0107] Instructions, media for delivering such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0108] In the foregoing description, various aspects of the present application are described with reference to their specific aspects, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative aspects of the present application have been described in detail herein, it should be understood that the inventive concept can be implemented and adopted differently in other ways, and the appended claims are intended to be interpreted as including such variations, except for being limited by the prior art. The various features and aspects of the above-mentioned application can be used individually or in combination. In addition, without departing from the broader spirit and scope of this specification, various aspects can be utilized in any number of environments and applications outside the environment and application described herein. Therefore, the description and the accompanying drawings are considered to be illustrative rather than restrictive. For the purpose of illustration, the method is described in a specific order. It should be understood that, in alternative aspects, the method can be performed in an order different from the described order.

[0109] A person of ordinary skill will understand that less than (" ”) and is greater than (“ The symbol or term can be replaced by less than or equal to (“ ”) and greater than or equal to (“ ”) symbol instead.

[0110] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform those operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform those operations, or any combination thereof.

[0111] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is in direct or indirect communication with another component (eg, connected to another component via a wired or wireless connection, and / or other suitable communication interface).

[0112] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0113] The various illustrative logic blocks, modules, circuits and algorithm steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. To clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits and steps have been generally described above with respect to their functionality. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician can implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as causing departure from the scope of the present application.

[0114] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device handset, or an integrated circuit device with multiple uses (including applications in wireless communication device handsets and other devices). Any features described as modules or components may be implemented together in an integrated logic device, or individually as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions for executing one or more of the methods described above when executed. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a RAM, such as a synchronous dynamic random access memory (SDRAM), a ROM, a non-volatile random access memory (NVRAM), an EEPROM, a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0115] The program code may be executed by a processor, which may include one or more processors, such as one or more DSPs, general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. This processor may be configured to perform any of the techniques described in the present invention. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0116] Illustrative aspects of the disclosure include:

[0117] Aspect 1: A method of processing audio data, comprising: at a computing device, obtaining sensing information from an audio device that outputs a spatial audio stream for a user, wherein the audio device includes a first audio output device and a second audio output device; determining that the second audio output device is not in use based on the sensing information; modifying the spatial audio stream to create a modified spatial audio stream based on the determination that the second audio output device is not in use and a head posture of the user; and providing the modified spatial audio stream to the first audio output device.

[0118] Aspect 2: The method according to aspect 1 further includes: obtaining motion information related to the user's motion from at least the first audio output device; and determining the user's head posture based on the motion information.

[0119] Aspect 3: The method according to any one of Aspects 1 to 2, wherein the sensing information indicates that the second audio output device is decoupled from the user, the first audio output device, or the computing device, and further includes: detecting the decoupling of the second audio output device from the user, the first audio output device, or the computing device based on the sensing information.

[0120] Aspect 4: The method according to any one of aspects 1 to 3, wherein obtaining the sensing information comprises receiving the sensing information from a proximity sensor of the second audio output device.

[0121] Aspect 5: The method according to any one of aspects 1 to 4, wherein obtaining the sensing information comprises receiving the sensing information from a pressure sensor of the first audio output device or the second audio output device.

[0122] Aspect 6: The method according to any one of aspects 1 to 5, wherein obtaining the sensing information comprises receiving the sensing information from the first audio output device or the second audio output device.

[0123] Aspect 7: The method according to any one of aspects 1 to 6, wherein determining that the second audio output device is not in use comprises: determining that a distance between the second audio output device and the user's head is greater than a threshold distance.

[0124] Aspect 8: The method according to any one of Aspects 1 to 7, wherein determining that the second audio output device is not in use comprises: determining a signal strength of a signal from the audio device; and determining that the second audio output device is separated from the user's head based on the signal strength.

[0125] Aspect 9: A method according to any one of Aspects 1 to 8, wherein determining that the first audio output device or the second audio output device is not in use includes: receiving a message from the first audio output device or the second audio output device at the computing device indicating that the first audio output device or the second audio output device is not in use.

[0126] Aspect 10: A method according to any one of Aspects 1 to 9, wherein an audio source associated with the spatial audio stream provides position information associated with one or more objects configured to produce audio, and wherein modifying the spatial audio stream includes: obtaining position information associated with each of the one or more objects; applying at least one spatial filter to each of the one or more objects; and mixing audio associated with each of the one or more objects into the spatial audio stream.

[0127] Aspect 11: A method according to any one of Aspects 1 to 10, wherein applying at least one spatial filter to an object among one or more objects includes: determining that a second audio output device corresponds to a left channel or a right channel; determining an angle associated with the object based on determining that the second audio output device corresponds to the left channel or the right channel; and determining a sound scaling factor based on the angle associated with the object, an inter-channel level difference of the object relative to the left channel and the right channel, and a head posture of the user.

[0128] Aspect 12: The method according to any one of aspects 1 to 11, wherein inter-channel time difference information and inter-channel coherence information are omitted from the modification of the spatial audio stream.

[0129] Aspect 13: A method according to any one of Aspects 1 to 12, wherein an audio source associated with the spatial audio stream does not provide position information associated with one or more objects configured to produce audio, and wherein modifying the spatial audio stream includes: mixing a left channel and a right channel from the audio source into a mono audio stream; assigning a default position to the mono audio stream; and applying an inter-channel level difference filter to the mono audio stream based on a user's head posture and the default position to generate the spatial audio stream.

[0130] Aspect 14: The method according to any one of aspects 1 to 13, wherein inter-channel time difference information and inter-channel coherence information are omitted from the modification of the spatial audio stream.

[0131] Aspect 15: A method according to any one of Aspects 1 to 14, wherein, when the source provides position information, the modification of the spatial audio stream includes: obtaining position information associated with each object generating audio from one or more objects; excluding at least one binaural cue filter; excluding at least one filter associated with inter-channel time difference or inter-channel coherence; applying an inter-channel level difference filter to each object generating audio from one or more objects; and mixing the audio associated with each object generating audio from one or more objects into the spatial audio stream.

[0132] Aspect 16: A method according to any one of Aspects 1 to 15, wherein applying an inter-channel level difference filter to an object that generates audio includes: identifying that a second audio output device corresponds to a left channel or a right channel; determining an angle associated with the object based on that the second audio output device corresponds to the left channel or the right channel; and determining a sound scaling factor based on the angle associated with the object, the inter-channel level difference of the object relative to the left channel and the right channel, and the user's head posture.

[0133] Aspect 17: A method according to any one of Aspects 1 to 16, wherein, when the source does not provide position information, the modification of the spatial audio stream includes: mixing the left channel and the right channel from the source into the spatial audio stream; assigning a default position to the spatial audio stream; excluding at least one binaural cue filter; excluding at least one filter associated with inter-channel time difference or inter-channel coherence; and applying inter-channel level difference filtering to the spatial audio stream based on the user's head posture and the default position.

[0134] Aspect 18: An apparatus, comprising at least one memory (e.g., implemented in a circuit) and at least one processor (or processors) coupled to the memory. The at least one processor (or processors) is configured to: obtain sensing information from an audio device that outputs a spatial audio stream for a user, wherein the audio device includes a first audio output device and a second audio output device; determine that the second audio output device is not in use based on the sensing information; modify the spatial audio stream to create a modified spatial audio stream based on the determination that the second audio output device is not in use and the user's head posture; and provide the modified spatial audio stream to the first audio output device.

[0135] Aspect 19: The apparatus according to Aspect 18, wherein the at least one processor is configured to: obtain motion information related to the user's motion from at least the first audio output device; and determine the user's head posture based on the motion information.

[0136] Aspect 20: The apparatus according to any one of aspects 18 to 19, wherein the at least one processor is configured to: detect decoupling of the second audio output device from the user, the first audio output device or the apparatus based on the sensing information.

[0137] Aspect 21: The apparatus according to any one of aspects 18 to 20, wherein, to obtain the sensing information, the at least one processor is configured to receive the sensing information from a proximity sensor of the second audio output device.

[0138] Aspect 22: The apparatus according to any one of aspects 18 to 21, wherein, in order to obtain the sensing information, the at least one processor is configured to receive the sensing information from a pressure sensor of the first audio output device or the second audio output device.

[0139] Aspect 23: The apparatus according to any one of aspects 18 to 22, wherein, in order to obtain the sensing information, the at least one processor is configured to receive the sensing information from the first audio output device or the second audio output device.

[0140] Aspect 24: The apparatus according to any one of Aspects 18 to 23, wherein at least one processor is configured to: determine that the distance between the second audio output device and the user's head is greater than a threshold distance.

[0141] Aspect 25: The apparatus according to any one of aspects 18 to 24, wherein the at least one processor is configured to: determine a signal strength of a signal from the audio device; and determine that the second audio output device is separated from the user's head based on the signal strength.

[0142] Aspect 26: The apparatus according to any one of aspects 18 to 25, wherein a processor is configured to: receive a message from the first audio output device or the second audio output device indicating that the first audio output device or the second audio output device is not in use.

[0143] Aspect 27: An apparatus according to any one of Aspects 18 to 26, wherein an audio source associated with the spatial audio stream provides position information associated with one or more objects configured to produce audio, and wherein, in order to modify the spatial audio stream, at least one processor is configured to: obtain position information associated with each of the one or more objects; apply at least one spatial filter to each of the one or more objects; and mix audio associated with each of the one or more objects into the spatial audio stream.

[0144] Aspect 28: An apparatus according to any one of Aspects 18 to 27, wherein at least one processor is configured to: determine that the second audio output device corresponds to a left channel or a right channel; determine an angle associated with the object based on determining that the second audio output device corresponds to the left channel or the right channel; and determine a sound scaling factor based on the angle associated with the object, the inter-channel level difference of the object relative to the left channel and the right channel, and the user's head posture.

[0145] Aspect 29: The apparatus according to any one of aspects 18 to 28, wherein inter-channel time difference information and inter-channel coherence information are omitted from the modification of the spatial audio stream.

[0146] Aspect 30: An apparatus according to any one of Aspects 18 to 29, wherein an audio source associated with the spatial audio stream does not provide position information associated with one or more objects configured to produce audio, and wherein, to modify the spatial audio stream, at least one processor is configured to: mix left and right channels from the audio source into a mono audio stream; assign a default position to the mono audio stream; and apply an inter-channel level difference filter to the mono audio stream based on a user's head posture and the default position to generate the spatial audio stream.

[0147] Aspect 31: The apparatus according to any one of aspects 18 to 30, wherein inter-channel time difference information and inter-channel coherence information are omitted from the modification of the spatial audio stream.

[0148] Aspect 32: An apparatus according to any one of Aspects 18 to 31, wherein, in order to modify the spatial audio stream, at least one processor is configured to: obtain position information associated with each object generating audio from one or more objects; exclude at least one binaural cue filter; exclude at least one filter associated with inter-channel time difference or inter-channel coherence; apply an inter-channel level difference filter to each object generating audio from one or more objects; and mix the audio associated with each object generating audio from one or more objects into the spatial audio stream.

[0149] Aspect 33: An apparatus according to any one of Aspects 18 to 32, wherein at least one processor is configured to: identify that the second audio output device corresponds to a left channel or a right channel; determine an angle associated with the object based on that the second audio output device corresponds to the left channel or the right channel; and determine a sound scaling factor based on the angle associated with the object, the inter-channel level difference of the object relative to the left channel and the right channel, and the user's head posture.

[0150] Aspect 34: An apparatus according to any one of Aspects 18 to 33, wherein, in order to modify the spatial audio stream, at least one processor is configured to: mix a left channel and a right channel from a source into the spatial audio stream; assign a default position to the spatial audio stream; exclude at least one binaural cue filter; exclude at least one filter associated with inter-channel time difference or inter-channel coherence; and apply inter-channel level difference filtering to the spatial audio stream based on the user's head posture and the default position.

[0151] Aspect 35: A non-transitory computer-readable medium comprising instructions, which when executed by one or more processors cause the one or more processors to perform the operations according to any one of aspects 1 to 34.

[0152] Aspect 36: An apparatus comprising components for performing the operations according to any one of aspects 1 to 34.

Claims

1. A method for processing audio data, comprising: At a computing device, obtaining sensing information from an audio device that outputs a spatial audio stream for a user, wherein the audio device includes a first audio output device and a second audio output device; determining, based on the sensing information, that the second audio output device is not in use; modifying the spatial audio stream to create a modified spatial audio stream based on a determination that the second audio output device is not in use and a head gesture of the user; and The modified spatial audio stream is provided to the first audio output device.

2. The method according to claim 1, further comprising: obtaining motion information related to the motion of the user from at least the first audio output device; as well as The head posture of the user is determined based on the motion information.

3. The method according to claim 1, wherein: The sensing information indicates that the second audio output device is decoupled from the user, the first audio output device, or the computing device, and further comprising: Decoupling of the second audio output device from the user, the first audio output device, or the computing device is detected based on the sensed information.

4. The method according to claim 1, wherein: Obtaining the sensing information includes receiving the sensing information from a proximity sensor of the second audio output device.

5. The method according to claim 1, wherein: Obtaining the sensing information includes: receiving the sensing information from a pressure sensor of the first audio output device or the second audio output device.

6. The method according to claim 1, wherein: Obtaining the sensed information includes receiving the sensed information from the first audio output device or the second audio output device.

7. The method of claim 1 , wherein determining that the second audio output device is not in use comprises: It is determined that a distance between the second audio output device and the user's head is greater than a threshold distance.

8. The method of claim 1, wherein determining that the second audio output device is not in use comprises: determining a signal strength of a signal from the audio device; as well as It is determined based on the signal strength that the second audio output device is separated from the user's head.

9. The method of claim 1, wherein determining that the first audio output device or the second audio output device is not in use comprises: A message is received at the computing device from the first audio output device or the second audio output device indicating that the first audio output device or the second audio output device is not in use.

10. The method according to claim 1, wherein: An audio source associated with the spatial audio stream provides position information associated with one or more objects configured to produce audio, and wherein modifying the spatial audio stream comprises: obtaining the location information associated with each of the one or more objects; applying at least one spatial filter to each of the one or more objects; and Audio associated with each of the one or more objects is mixed into the spatial audio stream.

11. The method according to claim 10, wherein: Applying the at least one spatial filter to an object of the one or more objects comprises: Determining whether the second audio output device corresponds to a left channel or a right channel; determining an angle associated with an object based on determining that the second audio output device corresponds to the left channel or the right channel; and A sound scaling factor is determined based on the angle associated with the object, an inter-channel level difference of the object relative to the left channel and the right channel, and the head pose of the user.

12. The method according to claim 10, wherein: Inter-channel time difference information and inter-channel coherence information are omitted from the modification of the spatial audio stream.

13. The method according to claim 10, wherein: An audio source associated with the spatial audio stream does not provide position information associated with one or more objects configured to produce audio, and wherein modifying the spatial audio stream comprises: Mix the left and right channels from an audio source into a mono audio stream; assigning a default position to the mono audio stream; and An inter-channel level difference filter is applied to the mono audio stream based on the head pose of the user and the default position to generate the spatial audio stream.

14. The method according to claim 13, wherein: Inter-channel time difference information and inter-channel coherence information are omitted from the modification of the spatial audio stream.

15. An apparatus comprising: at least one memory; as well as At least one processor coupled to at least one memory and configured to: obtaining sensing information from an audio device that outputs a spatial audio stream for a user, wherein the audio device includes a first audio output device and a second audio output device; determining, based on the sensing information, that the second audio output device is not in use; modifying the spatial audio stream to create a modified spatial audio stream based on a determination that the second audio output device is not in use and a head pose of the user; as well as The modified spatial audio stream is provided to the first audio output device.

16. The device according to claim 15, wherein: The at least one processor is configured to: obtaining motion information related to the motion of the user from at least the first audio output device; and The head posture of the user is determined based on the motion information.

17. The device according to claim 15, wherein: The at least one processor is configured to: Decoupling of the second audio output device from the user, the first audio output device, or the apparatus is detected based on the sensed information.

18. The device according to claim 15, wherein: In order to obtain the sensing information, the at least one processor is configured to: The sensing information is received from a proximity sensor of the second audio output device.

19. The device according to claim 15, wherein: In order to obtain the sensing information, the at least one processor is configured to: The sensing information is received from a pressure sensor of the first audio output device or the second audio output device.

20. The device according to claim 15, wherein: In order to obtain the sensing information, the at least one processor is configured to: The sensing information is received from the first audio output device or the second audio output device.

21. The device according to claim 15, wherein: The at least one processor is configured to: It is determined that a distance between the second audio output device and the user's head is greater than a threshold distance.

22. The device according to claim 15, wherein: The at least one processor is configured to: determining a signal strength of a signal from the audio device; and It is determined based on the signal strength that the second audio output device is separated from the user's head.

23. The device according to claim 15, wherein: The at least one processor is configured to: A message is received from the first audio output device or the second audio output device indicating that the first audio output device or the second audio output device is not in use.

24. The device according to claim 15, wherein: An audio source associated with the spatial audio stream provides position information associated with one or more objects configured to produce audio, and wherein, to modify the spatial audio stream, the at least one processor is configured to: obtaining the location information associated with each of the one or more objects; applying at least one spatial filter to each of the one or more objects; and Audio associated with each of the one or more objects is mixed into the spatial audio stream.

25. The device according to claim 24, wherein: The at least one processor is configured to: Determining whether the second audio output device corresponds to a left channel or a right channel; determining an angle associated with an object based on determining that the second audio output device corresponds to the left channel or the right channel; as well as A sound scaling factor is determined based on the angle associated with the object, an inter-channel level difference of the object relative to the left channel and the right channel, and the head pose of the user.

26. The device according to claim 24, wherein: Inter-channel time difference information and inter-channel coherence information are omitted from the modification of the spatial audio stream.

27. The device according to claim 24, wherein: An audio source associated with the spatial audio stream does not provide position information associated with one or more objects configured to produce audio, and wherein, to modify the spatial audio stream, the at least one processor is configured to: Mixing the left channel and the right channel from the audio source into a mono audio stream; assigning a default position to the mono audio stream; and An inter-channel level difference filter is applied to the mono audio stream based on the head pose of the user and the default position to generate the spatial audio stream.

28. The device according to claim 27, wherein Inter-channel time difference information and inter-channel coherence information are omitted from the modification of the spatial audio stream.

29. The device according to claim 24, wherein: To modify the spatial audio stream, the at least one processor is configured to: obtaining the position information associated with each object generating audio from the one or more objects; Excluding at least one binaural cue filter; excluding at least one filter associated with inter-channel time difference or inter-channel coherence; applying an inter-channel level difference filter to each object producing audio from the one or more objects; as well as Audio of each object with generated audio from the one or more objects is mixed into the spatial audio stream.

30. The device according to claim 24, wherein: To modify the spatial audio stream, the at least one processor is configured to: identifying that the second audio output device corresponds to a left channel or a right channel; determining an angle associated with the object based on the second audio output device corresponding to the left channel or the right channel; as well as A sound scaling factor is determined based on the angle associated with the object, an inter-channel level difference of the object relative to the left channel and the right channel, and the head pose of the user.

Images