Spatialized Audio for Mobile Peripheral Devices

Abstract

An audio system, method, and computer program product including a wearable audio device and a mobile peripheral device. Each device is capable of determining its respective absolute or relative position and orientation. Once the relative positions and orientations between the devices are known, a virtual sound source is generated at a fixed position and orientation relative to the peripheral device, such that any change in the peripheral device's position and / or orientation generates a proportional change in the virtual sound source's position and / or orientation. Additionally, primary and secondary reflected audio paths may be simulated for each virtual sound source to increase the realism of the simulated sound source. Each sound path may be generated by modifying the original audio signal using a head-related transfer function (HRTF) to simulate audio as if it were being perceived by the user's left and right ears as coming from each virtual sound source.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. patent application Ser. No. 16 / 904,087, filed Jun. 17, 2020, entitled “Spatialized Audio Relative to a Mobile Peripheral Device,” which is hereby incorporated by reference in its entirety. [Background technology]

[0002] Aspects and implementations of the present disclosure are directed generally to audio systems, including, for example, mobile peripheral devices and wearable audio devices.

[0003] Audio systems, for example augmented reality audio systems, can utilize a technique called sound externalization to render audio signals to a listener that tricks the listener into thinking that they are perceiving sounds from a physical location in the environment. Specifically, when listening to audio, especially through stereo headphones, many listeners perceive sounds as coming from "inside the listener's head." Sound externalization refers to the process of simulating and rendering sounds so that they are perceived by the user as coming from the surrounding environment, i.e., as if the sounds are "outside" the listener.

[0004] Because these augmented reality audio systems may be implemented using mobile devices, simulating or externalizing sound sources at a given location may not be desirable for some users. Summary of the Invention

[0005] The present disclosure relates to an audio system, method, and computer program product including a wearable audio device and a mobile peripheral device. The wearable audio device and the peripheral device are capable of determining their respective positions and / or orientations in an environment and relative to each other. Once the relative positions and orientations, e.g., between the wearable audio device and the peripheral device, are known, a virtual sound source can be generated at a fixed position and orientation relative to the peripheral device such that any change in the position and / or orientation of the peripheral device generates a proportional change in the position and / or orientation of the virtual sound source. Additionally, one or more orders of reflected audio paths can be simulated for each virtual sound source to increase the sense of realism of the simulated sound source. For example, by modifying the original audio signal using multiple left and multiple right head-related transfer functions (HRTFs), each sound path, e.g., a direct sound path, as well as a first reflected sound path and a second reflected sound path, can be generated to simulate the audio as if it were coming from each virtual sound source by the user's left and right ears, respectively.

[0006] Thus, the present disclosure includes audio systems, methods, and computer program products for generating spatialized and externalized audio "pinned" to a peripheral device. The systems, methods, and computer program products may utilize 1) a means for tracking a user's head location and / or orientation, 2) a means for tracking a peripheral device location and / or orientation, and 3) a means for rendering a spatialized audio signal in which the location of the virtual sound source is anchored or pinned in some manner to the peripheral device. This may include placing virtual sound sources for the left and right channel audio signals at the virtual left and virtual right of the peripheral device. The virtual sound sources may also include a discrete central virtual sound source, an extracted central virtual sound source, or a phantom central virtual sound source for the center channel audio. The concepts disclosed herein may also be scaled to additional channels, for example, to include additional channels for the implementation of a virtual surround sound system (e.g., virtual 5.1 or 7.1). These concepts may also include object-oriented rendering, such as that provided by the Dolby Atmos system, which can add a virtual height channel to a virtual surround sound system (e.g., virtual 5.1.2 or 5.1.4).

[0007] In one embodiment, a computer program product for simulating an audio signal is provided, the computer program product including a set of non-transitory computer readable instructions stored in a memory, the set of non-transitory computer readable instructions being executable by a processor and including: acquiring or receiving an orientation of a wearable audio device relative to a peripheral device in an environment; generating a first modified audio signal, the first modified audio signal being modified using a first head-related transfer function (HRTF) based at least in part on an orientation of the wearable audio device relative to the peripheral device; and generating a second modified audio signal. a first modified audio signal configured to be rendered using a first speaker of the wearable audio device and a second modified audio signal configured to be rendered using a second speaker of the wearable audio device, the second modified audio signal being modified using a second head-related transfer function (HRTF) based at least in part on an orientation of the wearable audio device relative to a peripheral device; and transmitting the first modified audio signal and the second modified audio signal to the wearable audio device, the first modified audio signal being configured to be rendered using a first speaker of the wearable audio device and the second modified audio signal being configured to be rendered using a second speaker of the wearable audio device.

[0008] In one aspect, the set of non-transitory computer readable instructions is further configured to obtain or receive a position of the wearable audio device relative to a position of a peripheral device in the environment, and modifying the first modified audio signal and modifying the second modified audio signal includes an attenuation based at least in part on a calculated distance between the position of the wearable audio device and the position of the peripheral device.

[0009] In one aspect, the set of non-transitory computer readable instructions is further configured to obtain or receive an orientation of the peripheral device relative to the wearable audio device, wherein the first HRTF and the second HRTF are based in part on the orientation of the peripheral device relative to the wearable device.

[0010] In one aspect, the first modified audio signal and the second modified audio signal are configured to simulate a first direct sound originating from a first virtual sound source proximate to a center of the peripheral device.

[0011] In one aspect, generating the first modified audio signal and generating the second modified audio signal include simulating a first direct sound originating from a first virtual sound source proximate the position of the peripheral device in the environment and simulating a second direct sound originating from a second virtual sound source proximate the position of the peripheral device.

[0012] In one aspect, generating the first modified audio signal and generating the second modified audio signal includes simulating surround sound.

[0013] In one aspect, generating the first modified audio signal and generating the second modified audio signal include using the first HRTF and the second HRTF, respectively, for only a subset of all available audio frequencies and / or channels.

[0014] In one aspect, the first HRTF and the second HRTF are further configured to utilize localization data from an environmental localization module corresponding to locations of a plurality of acoustically reflective surfaces within the environment.

[0015] In one aspect, generating the first modified audio signal includes simulating a first direct sound originating from a first virtual sound source proximate to the peripheral device and simulating a first reflected sound corresponding to a simulated reflection of the first direct sound from a first acoustically reflective surface of the plurality of acoustically reflective surfaces.

[0016] In one aspect, generating the first modified audio signal includes simulating a secondary reflected sound corresponding to a simulated reflection of the primary reflected sound from a second acoustically reflective surface of the plurality of acoustically reflective surfaces.

[0017] In one aspect, the first modified audio signal and the second modified audio signal correspond to video content displayed on the peripheral device.

[0018] In one aspect, the orientation of the wearable audio device relative to the peripheral device is determined using at least one sensor located on, within, or in proximity to the wearable audio device or the peripheral device, the at least one sensor selected from a gyroscope, an accelerometer, a magnetometer, a global positioning sensor (GPS), a proximity sensor, a microphone, a lidar sensor, or a camera.

[0019] In another example, a method of simulating an audio signal is provided, the method including receiving a first modified audio signal from a peripheral device via a wearable audio device, the first modified audio signal being modified using a first head-related transfer function (HRTF) based at least in part on an orientation of the wearable audio device relative to the peripheral device; receiving a second modified audio signal from the peripheral device via the wearable audio device, the second modified audio signal being modified using a second head-related transfer function (HRTF) based at least in part on an orientation of the wearable audio device relative to the peripheral device; rendering the first modified audio signal using a first speaker of the wearable audio device; and rendering the second modified audio signal using a second speaker of the wearable audio device.

[0020] In one aspect, the method further includes obtaining a position of the wearable audio device relative to peripheral devices in the environment, and modifying the first modified audio signal and modifying the second modified audio signal are based at least in part on a calculated distance between the position of the wearable audio device and the position of the peripheral devices.

[0021] In one aspect, the method further includes obtaining an orientation of a peripheral device relative to the wearable audio device, the first HRTF and the second HRTF being based in part on the orientation of the peripheral device.

[0022] In one aspect, the first modified audio signal and the second modified audio signal are configured to simulate a first direct sound originating from a first virtual sound source proximate to a center of the peripheral device.

[0023] In one aspect, rendering the first modified audio signal and rendering the second modified audio signal include simulating a first direct sound originating from a first virtual sound source proximate the position of the peripheral device in the environment and simulating a second direct sound originating from a second virtual sound source proximate the position of the peripheral device.

[0024] In one aspect, generating the first modified audio signal and generating the second modified audio signal includes simulating surround sound.

[0025] In one aspect, generating the first modified audio signal and generating the second modified audio signal include using the first HRTF and the second HRTF, respectively, for only a subset of all available audio frequencies and / or channels.

[0026] In one aspect, the method further includes receiving localization data from a localization module within the environment and determining locations of a plurality of acoustically reflective surfaces within the environment based on the localization data.

[0027] In one aspect, rendering the first modified audio signal includes simulating a first direct sound originating from a first virtual sound source proximate to the peripheral device and simulating a first reflected sound corresponding to a simulated reflection of the first direct sound from a first acoustically reflective surface of the plurality of acoustically reflective surfaces.

[0028] In one aspect, rendering the first modified audio signal includes simulating a secondary reflection corresponding to a simulated reflection of the primary reflection from a second acoustically reflective surface of the plurality of acoustically reflective surfaces.

[0029] In one aspect, the peripheral device includes a display configured to display video content associated with the first modified audio signal and the second modified audio signal.

[0030] In one aspect, the orientation of the wearable audio device relative to the peripheral device is determined using at least one sensor located on, within, or in proximity to the wearable audio device or the peripheral device, the at least one sensor selected from a gyroscope, an accelerometer, a magnetometer, a global positioning sensor (GPS), a proximity sensor, a microphone, a lidar sensor, or a camera.

[0031] In a further embodiment, an audio system for simulating audio is provided, the system including a peripheral device configured to acquire or receive an orientation of a wearable audio device relative to the peripheral device in an environment, the peripheral device further configured to generate a first modified audio signal using a first head-related transfer function (HRTF) based on an orientation of the wearable audio device relative to the peripheral device and generate a second modified audio signal using a second head-related transfer function (HRTF) based on an orientation of the wearable audio device relative to the peripheral device, and a wearable audio device, the wearable audio device including a processor configured to receive the first modified audio signal and to receive the second modified audio signal, a first speaker configured to render the first modified audio signal using the first speaker, and a second speaker configured to render the second modified audio signal using the second speaker.

[0032] These and other aspects of the various embodiments will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief description of the drawings]

[0033] In the drawings, like reference numbers generally refer to the same parts throughout the different views. Also, the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of various embodiments. [Figure 1] 1 is a schematic perspective view of an audio system according to the present disclosure; [Figure 2A] 1 is a schematic representation of components of a wearable audio device according to the present disclosure. [Figure 2B] 1 is a schematic representation of components of a peripheral device according to the present disclosure. [Diagram 3] FIG. 1 is a schematic top view of components of an audio system according to the present disclosure. [Figure 4] 1 is a schematic top view of components of an audio system in an environment according to the present disclosure. [Diagram 5] 1 is a schematic top view of components of an audio system in an environment according to the present disclosure. [Figure 6] FIG. 1 is a schematic top view of components of an audio system according to the present disclosure. [Figure 7] 1 is a schematic top view of components of an audio system in an environment according to the present disclosure. [Figure 8] 1 is a schematic top view of components of an audio system in an environment according to the present disclosure. [Figure 9] 1 is a schematic top view of components of an audio system in an environment according to the present disclosure. [Figure 10] 1 is a flow diagram illustrating steps of a method according to an aspect of the present disclosure. [Figure 11] 1 is a flow diagram illustrating steps of a method according to an aspect of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] The present disclosure relates to audio systems, methods, and computer program products that include a wearable audio device (e.g., headphones or earbuds) and a peripheral device, such as a mobile peripheral device (e.g., a smartphone or tablet computer). The wearable audio device and the peripheral device are capable of determining their respective positions and / or orientations in the environment and relative to each other. Once the relative positions and orientations, e.g., between the wearable audio device and the peripheral device, are known, a virtual sound source may be generated at a fixed position and orientation relative to the peripheral device, such that any change in the position and / or orientation of the peripheral device generates a proportional change in the position and / or orientation of the virtual sound source. Additionally, one or more orders of reflected audio paths (e.g., first order, and optionally second order) may be simulated for each virtual sound source to increase the sense of realism of the simulated sound source. By modifying the original audio signal using multiple left and multiple right head-related transfer functions (HRTFs), orders of each sound path, e.g., direct sound paths and reflected sound paths (e.g., first order and, optionally, second order), can be generated to simulate the audio as if it were coming from each virtual sound source by the user's left and right ears, respectively.

[0035] As used in this application, the term "wearable audio device" is intended to mean a device that fits around, on, in, or near the ear (including open-ear audio devices worn on the user's head or shoulders) and radiates acoustic energy into or toward the ear. Wearable audio devices may also be referred to as headphones, earphones, earpieces, headsets, earbuds, or sports headphones, and may be wired or wireless. Wearable audio devices include an acoustic driver for converting audio signals into acoustic energy, which may utilize air conduction and / or bone conduction techniques. The acoustic driver may be housed in an ear cup. Although some of the following figures and descriptions may show a single wearable audio device with a pair of ear cups (each including an acoustic driver), it should be understood that the wearable audio device may be a single standalone unit with only one ear cup. Each earcup of a wearable audio device may be mechanically connected to another earcup or headphone, for example, by a headband and / or by leads that carry audio signals to acoustic transducers in the earcups or headphones. The wearable audio device may include components that wirelessly receive audio signals. The wearable audio device may include components of an active noise reduction (ANR) system. The wearable audio device may also include other features such as a microphone so that the device can function as a headset. While FIG. 1 shows one example of an audio glasses form factor, in other examples the headset may be an in-ear, supra-ear, around-the-ear, or near-ear headset. In some examples, the wearable audio device may be an open-ear device that includes an acoustic driver that radiates acoustic energy toward the ear while opening the ear to the ear's environment and surroundings.

[0036] The term "head-related transfer function" or the acronym "HRTF" as used herein, in addition to its ordinary meaning to those skilled in the art, is intended to broadly reflect any method of calculating, determining, or approximating the binaural sound perceived by the human ear so that a listener can approximate the original location of the sound in space. For example, an HRTF may be a mathematical formula or a set of mathematical formulas that can be applied to or convolved with an audio signal so that a user listening to the modified audio signal can perceive the sound as originating at a particular point in space. These HRTFs referred to herein may be generated specifically for each user, for example, taking into account the user's unique physiology (e.g., size and shape of the head, ears, nasal cavity, oral cavity, etc.). Alternatively, it should be understood that a generalized HRTF may be generated that is applied to all users, or multiple generalized HRTFs may be generated that are applied to a subset of users (e.g., based on certain physiological characteristics that at least roughly indicate the unique head-related transfer function of that user, such as age, sex, head size, ear size, or other parameters). In one embodiment, certain aspects of the HRTFs may be determined accurately while other aspects are roughly approximated (eg, determining the interaural delay accurately but determining the amplitude response roughly).

[0037] The following description should be read in light of Figures 1-9. Figure 1 is a schematic diagram of an audio system 100 according to the present disclosure. The audio system 100 includes a wearable audio device 102 and a peripheral device 104. The wearable audio device 102 is intended to be a device capable of receiving an audio signal, such as an audio signal discussed below, such as modified audio signals 146A-146B (shown in Figures 2A and 2B), and generating or rendering the signal into acoustic energy in an environment E and proximate to a user's or wearer's ear. In one embodiment, as illustrated in Figure 1, the wearable audio device 102 comprises a glasses form factor audio device capable of rendering acoustic energy outside and proximate to a user's ear. It should be understood that in other embodiments, the wearable audio device 102 may be selected from over-the-ear or in-the-ear headphones, earphones, earpieces, headsets, earbuds, or sports headphones. The peripheral device 104 may be selected from any electronic device capable of generating and / or transmitting audio signals, such as modified audio signals 146A-146B discussed below, to a separate device, such as the wearable audio device 102. In one embodiment, as illustrated in Figures 1 and 3-9, the peripheral device 104 is intended to be a tablet. However, it should be understood that the peripheral device 104 may be selected from a smartphone, a laptop or personal computer, a case configured to matingly engage and / or charge the wearable audio device 102, or any other portable and / or movable computing device.

[0038] As illustrated in FIG. 2A, the wearable audio device 102 further includes a first circuit 106. The first circuit 106 includes a first processor 108 and a first memory 110 configured to execute and store a first set of non-transitory computer-readable instructions 112 to perform various functions of the first circuit 106 and the wearable audio device 102, respectively, as described herein. The first circuit 106 further includes a first communication module 114 configured to transmit and / or receive data, e.g., audio data, via a wired or wireless connection, e.g., a data connection 142 (discussed below), with the peripheral device 104. In some embodiments, the transmitted and / or received audio data includes modified audio signals 146A-146B, discussed below. It should be understood that the first communication module 114 may further include a first antenna 116 for the purpose of transmitting and / or receiving data, as discussed above. Further, although not illustrated, it should be understood that the wearable audio device 102 may include a battery, capacitor, supercapacitor, or other power source located on, within, or in electronic communication with the first circuit 106.

[0039] The first circuit 106 also includes at least one sensor, namely, the first sensor 118. The first sensor 118 may be located on, within, or in communication with the wearable audio device 102. The first sensor 118 is selected from at least one of a gyroscope, an accelerometer, a magnetometer, a global positioning sensor (GPS), a proximity sensor, one or more microphones, one or more cameras (e.g., a front-mounted camera and a rear-mounted camera), or any other sensor device capable of obtaining at least one of a first position P1 of the wearable audio device 102 in the environment E, a first position P1 relative to the peripheral device 104, a first orientation O1 of the wearable audio device 102 relative to the environment E, a first orientation O1 of the wearable audio device 102 relative to the peripheral device 104, or a distance between the wearable audio device 102 and the peripheral device 104. The first position P1 and the first orientation O1 are discussed in more detail below. Additionally, the first circuit 106 may also include at least one speaker 120. In one embodiment, the first sensor 118 is one or more cameras, e.g., a front-mounted camera and a rear-mounted camera, capable of acquiring image data of the environment E and / or a relative position and a relative orientation of the peripheral device 104, as discussed below. In one embodiment, the first circuit 106 includes a plurality of speakers 120A-120B configured to receive audio signals, e.g., modified audio signals 146A-146B (discussed below), and generate audio reproduction APB to generate audible acoustic energy associated with the audio signals proximate to the user's ears.

[0040] As illustrated in FIG. 2B, the peripheral device 104 further includes a second circuit 122. The second circuit 122 includes a second processor 124 and a second memory 126 configured to execute and store a second set of non-transitory computer-readable instructions 128 to perform various functions of the second circuit 122 and the peripheral device 104, respectively, as described herein. The second circuit 122 further includes a second communication module 130 configured to transmit and / or receive data, e.g., audio data, via a wired or wireless connection to the wearable audio device 102 (discussed below) and / or a device capable of connecting to the Internet, e.g., a local router or a cellular tower. In some embodiments, the transmitted and / or received audio data includes modified audio signals 146A-146B, discussed below. It should be appreciated that the second communication module 130 may further include a second antenna 132 for the purpose of transmitting and / or receiving data, as discussed above. Further, although not illustrated, it should be understood that the peripheral device 104 may include a battery, capacitor, supercapacitor, or other power source located on, in, or in electronic communication with the second circuit 122.

[0041] The second circuit 122 may also include at least one sensor, namely the second sensor 134. The second sensor 134 may be located on, in, or in communication with the peripheral device 104. The second sensor 134 is selected from at least one of a gyroscope, an accelerometer, a magnetometer, a global positioning sensor (GPS), a proximity sensor, a microphone, a camera or multiple cameras (e.g., front and rear cameras), or any other sensor device capable of obtaining at least one of a second position P2 of the peripheral device 104 in the environment E, a second position P2 relative to the wearable audio device 102, a second orientation O2 of the peripheral device 104 relative to the environment E, a second orientation O2 of the peripheral device 104 relative to the wearable audio device 102, or a distance between the wearable audio device 102 and the peripheral device 104. The second position P2 and the second orientation O2 are discussed in more detail below. In one embodiment, the second sensor 134 is one or more cameras, such as a front-mounted camera and a rear-mounted camera, capable of acquiring image data of the environment E and / or the relative position and relative orientation of the wearable audio device 102, as discussed below.

[0042] Further, the second circuit 122 may also include at least one device speaker 136 and a display 138. In one embodiment, the at least one device speaker 136 is configured to receive the audio signal or a portion of the audio signal, e.g., modified audio signals 146A-146B (discussed below), and generate an audio playback APB to generate audible acoustic energy associated with the audio signal at a second position P2 of the peripheral device 104 that is a fixed distance from the wearable audio device 102. The display 138 is intended to be a screen capable of displaying video content 140. In one embodiment, the display 138 is a Liquid-Crystal Display (LCD) and may also include touch screen functionality, e.g., utilizing resistive or capacitive sensing to determine the contact and position of a user's finger relative to the screen surface. It should also be understood that the display 138 may be selected from at least one of a Light-Emitting Diode (LED) screen, an Organic Light-Emitting Diode (OLED) screen, a plasma screen, or any other display technology capable of presenting pictures or video, e.g., video content 140, to a viewer or user.

[0043] As mentioned above, the wearable audio device 102 and / or the peripheral device 104 are configured to obtain their respective positions and orientations within the environment E and / or relative to each other using the first sensor 118 and the second sensor 134, respectively. In one embodiment, the environment E is a room, e.g., a space defined by a floor surrounded by at least one wall and covered by a ceiling or roof, in which a single position may be modeled and defined by a three-dimensional Cartesian coordinate system as having an X, Y, and Z position within the defined space associated with length, width, and height dimensions, respectively. Thus, obtaining the first position P1 of the wearable audio device 102 may be absolute within the environment E, e.g., defined purely by the Cartesian coordinates of the wearable audio device within the room, or may be relative to the position of another device, i.e., the peripheral device 104.

[0044] Similarly, each device may obtain its own orientation defined by a respective yaw, pitch, and roll in a spherical coordinate system having an origin at the center of each device, where yaw includes rotation about a vertical axis passing through the device and perpendicular to the floor beneath the device, pitch includes rotation about a first horizontal axis perpendicular to the vertical axis and extending from at least one wall of the room, and roll includes rotation about a second horizontal axis perpendicular to the vertical axis and the first horizontal axis. In one embodiment, when a first orientation O1 of the wearable audio device 102 and a second orientation O2 of the peripheral device 104 are defined relative to each other, each device may determine a vector representing a relative elevation angle and a relative azimuth angle between each device based in part on the yaw, pitch, and roll of each device. It should also be understood that the first orientation O1 and the second orientation O2 may also be obtained absolutely within the environment E, for example, relative to a predefined and / or fixed position within the environment E.

[0045] As mentioned above, each circuit of the devices of the audio system 100, e.g., the first circuit 106 of the wearable audio device 102 and the second circuit 122 of the peripheral device 104, can establish a data connection 142 and transmit and / or receive wired or wireless data via the data connection 142. For example, the first antenna 116 of the first communication module 114 is configured to establish the data connection 142 with the second antenna 132 of the second communication module 130. The data connection 142 may utilize one or more wired or wireless data protocols selected from at least one of Bluetooth, Bluetooth Low-Energy (BLE) or LE Audio, Radio Frequency Identification (RFID) communication, Low-Power Radio frequency transmission (LP-RF), Near-Field Communication (NFC), or any other protocol or communication standard capable of establishing a permanent or semi-permanent connection, also referred to as a pairing connection, between the first circuit 106 and the second circuit 122. It should be appreciated that the data connection 142 may be utilized by the first circuit 106 of the wearable audio device 102 and the second circuit 122 of the peripheral device 104 to transmit and / or receive data regarding each device's respective position and orientation, e.g., first position P1, second position P2, first orientation O1, second orientation O2, and the distance between the devices, as discussed above, thereby enabling each device to be aware of the position and orientation of itself and / or other devices in the audio system 100. Additionally, as noted above, the data connection 142 may also be used to transmit and / or receive audio data, e.g., modified audio signals 146A-146B (discussed below), between the devices of the audio system 100.

[0046] In addition to the ability to obtain the respective positions and orientations of each device of audio system 100, audio system 100 is also configured to render externalized sound to a user within environment E using modified audio signals 146A-146B (also discussed below), e.g., filtered or modified using at least one head-related transfer function (HRTF) (discussed below). In one embodiment of audio system 100, sound externalization for use with augmented reality audio systems and programs is achieved by modeling environment E, creating virtual sound sources, e.g., virtual sound sources 144A-144G (collectively referred to as "multiple virtual sound sources 144" or "virtual sound sources 144") at various locations within environment E, and modeling or simulating sound waves and their respective paths from virtual sound sources 144 (shown in FIGS. 3-9) to the user's ear positions to simulate the perception of sound to the user as if the virtual sound sources 144 were real or tangible sound sources, e.g., physical speakers located at each virtual sound source position. For each modeled or simulated sound path, computational processing is used to apply or convolve at least a pair of HRTFs (one associated with the left ear and one associated with the right ear) to the audio signal to generate modified audio signals 146A-146B. Once the HRTFs have been applied and the modified audio signals 146A-146B have been generated, the modified audio signals 146A-146B can be played through multiple speakers 120A-120B (left and right speakers) of the wearable device 102 to trick the user into thinking that they are perceiving sounds from actual externalized sound sources located at the positions of the respective virtual sound sources 144. As described below, the quality of the simulated realism of these modified audio signals 146A-146B can be increased by simulating primary and secondary acoustic reflections from each virtual sound source in the environment E and by attenuating or delaying the simulated signals to approximate the time of flight of the propagation of sound signals through air.It should be understood that either the wearable audio device 102 and / or the peripheral device 104 may process, apply, or convolve HRTFs as discussed herein to simulate virtual sound sources. However, because the form factor, and therefore the space for additional processing components, is typically limited in a wearable audio device, such as the wearable audio device 102, it should also be understood that the application or convolution of HRTFs with the considered audio signal will likely be accomplished by circuitry in the peripheral device 104, and the modified audio signal 146A-146B may then be transmitted or streamed to the wearable audio device for rendering as an audio playback APB.

[0047] In some examples, the position of each virtual sound source of the plurality of virtual sound sources 144 relative to the position of the wearable audio device 102 can be used to calculate and simulate a respective plurality of direct sound paths 148A-148G (collectively referred to as "plurality of direct sound paths 148" or "direct sound path 148"), i.e., at least one direct sound path 148 from each virtual sound source 144 directly to the user's ear. Each sound path can be associated with a calculated distance of the respective direct sound path 148 from the virtual sound source 144 to the wearable audio device 102 (e.g., calculated distance D1 shown in FIG. 3 and calculated distances D2-D3 shown in FIGS. 5 and 7). Because real sound wave propagation dissipates as a function of distance or radius from an origin, the calculated distances can be used by the HRTFs to attenuate and / or delay sound signals as a function of the calculated distance, e.g., 1 / distance for each sound path considered herein. For each direct sound path 148, the audio system 100 may utilize at least one of a plurality of left HRTFs 150 and a plurality of right HRTFs 152 to filter or modify the original audio signal to account for the directionality and / or the calculated distance. In one embodiment, the HRTFs may utilize the azimuth, elevation, and distance between each virtual sound source 144 and the wearable audio device 102 to filter and / or attenuate the audio signal. It should be appreciated that in one embodiment, the left and right HRTFs may be obtained from a predefined database, where the particular pair or individual HRTFs selected are selected based on a particular relative azimuth and / or a particular relative elevation between the devices. Thus, in some example implementations, each HRTF is stored as a database of filter coefficients for different azimuth and / or relative elevation angles, rather than being directly calculated.

[0048] In one embodiment illustrated in Figures 3 and 4, the audio system 100 is configured to simulate direct sound from a single virtual sound source 144A. As shown in Figure 3, the audio system 100 includes a wearable audio device 102 at a first position P1 and a first orientation O1, and a peripheral device 104 at a second position P2 and a second orientation O2. As shown, the single virtual sound source 144A is generated or simulated at a center C of the peripheral device 104. The virtual sound source 144A is intended to simulate a central audio channel of a given audio signal along a direct sound path 148A. Additionally, since the positions of the wearable audio device 102 and the peripheral device 104 are absolutely known relative to each other or in the environment E, the position of the virtual sound source 144A is also known, and therefore the distance between the first sound source 144A and the wearable audio device 102 can be calculated, for example, as the calculated distance D1 shown in Figure 3. As discussed above and illustrated in Fig. 4, the audio system 100 can modify the audio signal to simulate the center channel audio as if it were generated at a position and distance corresponding to the center C of the peripheral device 104 by applying and convolving the original center channel audio signal with the left HRTF 150 and right HRTF 152 to result in modified audio signals 146A-146B that can be played through left and right speakers (e.g., speakers 120A and 120B shown in Fig. 2) to simulate a direct sound path 148A from the virtual sound source 144A to the user's left and right ears, respectively. It should be appreciated that in Fig. 4, the direct sound path 148A is divided diagrammatically to illustrate how the direct sound path 148A can represent both the modified audio signal 146A modified by the left HRTF 150 and the modified audio signal 146B modified by the right HRTF 152. For simplicity, the illustrations and descriptions below refer only to individual sound paths.However, it should be understood that each sound path may generally represent two separate modified audio signals modified using left and right HTRFs as discussed above.

[0049] Similar to the virtual sound source 144A associated with the center channel audio signal, the left channel audio signal and the right channel audio signal may be simulated through additional virtual sound sources, e.g., 144B and 144C, as illustrated in Fig. 5. As illustrated, the virtual sound source 144B may be generated adjacent to the left side L of the peripheral device 104 to simulate the left channel audio, and the virtual sound source 144C may be generated adjacent to the right side R of the peripheral device 104 to simulate the right channel audio. It should also be appreciated that these audio signals may be generated such that a phantom center channel is created equidistant between the virtual sound sources 144B and 144C, thereby eliminating the need to simulate the center channel audio through the virtual sound source 144A. 5, the virtual sound sources 144B and 144C may be positioned such that, when using the first position P1 of the wearable audio device 102 as the origin, the angle α created between the virtual sound sources 144B and 144C is about 30 degrees, e.g., −15 to +15 degrees, about the center line CL. It should be appreciated that this angle may be selected from any angle within the range of 0 to 180 degrees, e.g., −75 to +75 degrees, −50 to +50 degrees, −30 to +30 degrees, or −5 to +5 degrees, about the center line CL.

[0050] Additionally, other virtual sound source configurations are possible. For example, FIG. 6 illustrates a configuration of virtual sound sources 144 that simulate a 5.1 surround sound system. For example, virtual sound sources 144A-144C simulate the space in front of the wearable audio device 102 and adjacent to the peripheral device 104 to simulate front center, front left, and front right channel audio signals, as discussed above. To generate a 5.1 surround sound effect, two additional virtual sound sources, e.g., 144D and 144E, are simulated behind the wearable audio device 102 to simulate rear left and rear right audio signals, respectively. It should be understood that other arrangements and configurations are possible, for example, additional virtual sound sources can be added so that the audio system 100 can simulate a 7.1 surround sound system and a 9.1 surround sound system, and can also include at least one simulated subwoofer to provide simulated bass channel audio, although not illustrated.

[0051] Alternatively, although not illustrated, it should be understood that one or more virtual sound sources 144 in any of the aforementioned exemplary configurations may be replaced by real sound sources, e.g., real, tangible speakers located in the environment E at an approximate location of the virtual sound source they are intended to replace. For example, the center channel audio signal rendered at the location shown for the virtual sound source 144A may be replaced, i.e., not virtually generated at that location, and at least one device speaker 136 may render an audio reproduction APB at a location of the peripheral device 104 where the audio reproduction APB includes only center channel audio. Similarly, since it may be difficult to simulate the directionality of the audio corresponding to the bass audio channel, a real subwoofer may be placed in the environment E to replace the virtual equivalent bass sound source. Additionally or alternatively, it should be understood that one or more virtual sound sources 144 in any of the aforementioned exemplary configurations may be rendered by the wearable audio device 102 without being virtualized or spatialized as discussed herein. For example, in a configuration utilizing left, right, and center audio channels, as discussed above, audio system 100 may choose to virtualize or spatialize any of those channels by generating virtual sound sources 144 in environment E that simulate one or more of those channels. However, in addition to or as an alternative to spatializing one or more of those channels, audio system 100 may render audio at speakers of wearable audio device 102 that is non-spatialized, e.g., one or more of those channels may be rendered into audible sound by wearable audio device 102 and perceived by the user as coming from inside the user's head.

[0052] Additionally, in some implementations, the techniques described herein for spatially pinning audio to a given location (such as the center of a peripheral device's display) can separate audio and spatially pin by frequency and / or channel, such that portions of the audio are spatially pinned and other portions are not. For example, portions of audio related to low frequencies, such as those for subwoofer channels, can be excluded from being spatialized using the techniques variously described herein, since those low frequencies are relatively spatially / directionally agnostic compared to other frequencies. In other words, for low frequencies and / or subwoofer channels, there is little information that the user's brain can use to localize the source of the low frequencies and / or subwoofer channels, and thus including those frequencies and / or channels when converting the audio to be spatially pinned would add computational cost with little psychoacoustic benefit (since the user would not be able to tell where those low frequencies and / or subwoofer channels are coming from anyway). For this reason, subwoofers in audio systems can generally be placed anywhere in the room, since low frequencies are directionally agnostic. In some such implementations, these techniques include isolating frequencies, channels, and / or portions (e.g., low frequencies and / or subwoofer channels) before performing spatial pinning as variously described herein, performing spatial pinning on the remainder of the frequencies, channels, and / or portions, and then combining the non-spatially pinned aspects (e.g., low frequencies and / or subwoofer channels) with the spatial pinning aspects (e.g., all other frequencies and / or all other channels).

[0053] In the following examples corresponding to Figures 7-9, only two virtual sound sources are described and illustrated, namely virtual sound sources 144B and 144C. However, it should be understood that other configurations with more or fewer virtual sound sources are possible, as well as configurations with one or more subwoofers to simulate one or more bass channels, as detailed above. As discussed above, the position and orientation of each virtual sound source 144 is pinned, locked, or otherwise spatially fixed relative to the position and orientation of the peripheral device 104. In other words, if the peripheral device 104 moves, rotates, pivots, tilts, or otherwise changes its position, location, or orientation within the environment E or relative to the wearable audio device 102, the multiple virtual sound sources 144 will move, rotate, pivot, tilt, or otherwise change their position, location, or orientation proportionally, such that the position and orientation of each virtual sound source 144 is fixed relative to the peripheral device 104. Because the devices of the audio system 100 are capable of acquiring their relative positions and orientations with respect to each other or within the environment E, the distance between the devices and / or the virtual sound sources 144 can be utilized by the HRTFs to attenuate and / or delay the sound signals to simulate the actual time of flight that a real sound wave would experience as it propagates through the air from the location of the respective virtual sound source 144. Thus, by modifying the modification of the original audio signal using the left HRTF 150 and the right HRTF 152 into modified audio signals 146A and 146B, the real-world directionality and real-world time delay that would be experienced by multiple real external sound sources can be simulated for the wearer, user, or listener via the wearable audio device 102. Additionally, in some embodiments, the positions of the virtual sound sources in the environment E are pinned or fixed relative to the position and orientation of the peripheral device 104, e.g., may move, rotate, pivot, tilt, or otherwise change position, location, or orientation relative to movement of the peripheral device 104, while in some embodiments the height of each virtual sound source is fixed or limited to a particular height relative to the floor directly below the user.For example, if a user pivots the peripheral device 45 degrees in a rotation that positions the screen of the peripheral device to substantially face the ceiling above the user, any front virtual sound sources (e.g., in a 5.1 surround sound configuration) spatialized or virtualized opposite or behind the position of the peripheral device will pivot proportionately and may be closer to the floor directly below the user or may be inside this floor, while rear virtual sound sources spatialized or virtualized behind the user will pivot proportionately and may be closer to the ceiling above the user or may be inside this ceiling. Thus, in some embodiments, the height of the virtual sound sources, e.g., at least the front and rear simulated virtual sound sources, may be fixed or locked to a particular height above the floor, e.g., approximately the height of the wearable audio device 102 above the floor. In other embodiments, the height of the virtual sound sources may be fixed or locked relative to the height of a pedestal or other object in the environment E.

[0054] 7, the audio system 100 can simulate two virtual sound sources, e.g., virtual sound sources 144B and 144C corresponding to a left channel audio signal and a right channel audio signal, which are spatially pinned, locked, or otherwise fixed relative to the second orientation O2 and the second position P2 of the peripheral device 104. As illustrated, when a user rotates or otherwise changes the orientation of the peripheral device 104, e.g., when the peripheral device 104 is rotated approximately 45 degrees clockwise around the second position P2, the positions of the virtual sound sources 144B and 144C will rotate approximately 45 degrees around the position P2 at a fixed distance from the peripheral device 104, such that after the rotation of the peripheral device 104, the positions of the virtual sound sources 144B and 144C relative to the peripheral device 104 are the same as they were before the rotation. In particular, by rotating the peripheral device 104 by 45 degrees while the user maintains the original head position, i.e., the first position P1 and the first orientation O1 of the wearable audio device 102, the position of each virtual sound source 144B and 144C relative to the wearable audio device 102 is changed. For example, as shown in Figure 7, rotating the peripheral device 104 clockwise by approximately 45 degrees will cause the virtual sound source 144B to move away from the wearable audio device 102 while the virtual sound source 144C will move closer to the wearable audio device 102. In other words, as shown, the calculated distance D2 will increase while the calculated distance D3 will decrease. Thus, to account for the rotation of the peripheral device 104 relative to the wearable audio device 102, the left HRTF 150 may include a change in the calculated distance D2 of the virtual sound source 144B to simulate an increase in the distance to the wearable audio device 102, while the right HRTF 152 may include a change in the calculated distance D3 of the virtual sound source 144C to simulate a decrease in the distance to the wearable audio device 102.As discussed above, it should be understood that any number of virtual sound sources 144 may be simulated in any of the above example configurations, and each virtual sound source 144 may be spatially pinned, locked, or fixed relative to the peripheral device 104 as disclosed herein. Additionally, while the above examples only disclose a simple rotation of the peripheral device 104 by 45 degrees in a clockwise rotation, more complex changes in orientation or position, such as tilting, translation, pivoting, or any combination of these movements, may be considered in a manner similar to that described above.

[0055] In another embodiment, the audio system 100 may utilize the localization data to further increase the simulated realism of the externalized and / or virtualized sound sources 144. In addition to simulating the direct sound path from each virtual sound source 144 as described above, one way to increase the realism of the simulated sound is to add additional virtual sound sources 144 that simulate the primary and secondary reflections that a real audio source generates as the propagating sound signal reflects off acoustically reflective surfaces and returns to the user. In other words, the real sound sources generate not only directional waves, but also spherical waves that reflect, for example, from acoustically reflective surfaces 154A-154D (collectively referred to as "acoustic reflective surfaces 154" or "surfaces 154"), which may include, but are not limited to, walls, floors, ceilings, and other acoustically reflective surfaces such as furniture. Thus, localization refers to the process of acquiring data of the immediate or nearby area or environment E surrounding the user, e.g., surrounding the wearable audio device 102 and / or peripheral device 104, indicative of the location, orientation, and / or acoustic reflection properties of objects in the user's environment E. Once localized, a reflection path may be calculated between each virtual sound source 144 and each surface 154. The points where the path touches each surface 154, referred to herein as contact points CP, may be utilized to generate new virtual sound sources that, when simulated, generate a sound that simulates the acoustic reflection of the original virtual sound source 144. One method for generating these new virtual sound sources is to create a mirrored virtual sound source of each virtual sound source, where the mirrored virtual sound source is mirrored around the sound 154, as described with respect to FIG. 8 below. It should be appreciated that to assist in obtaining localization data regarding the environment E surrounding the user, the wearable audio device 102, and / or the peripheral device 104, the audio system 100 may further include a localization module 156 (shown in FIGS. 2A and 2B ), which may be provided as a separate device or may be integrated within the wearable audio device 102 or the peripheral device 104.For example, a separate localization module 156 may be provided, where the separate localization module 156 is selected from at least one of a range finder (e.g., a LIDAR sensor), a proximity sensor, a camera or cameras, a global positioning sensor (GPS), or any sensor, device, component, or technology capable of acquiring, collecting, or generating localization data regarding the location of the user, the wearable audio device 102, the peripheral device 104, and the acoustically reflective surfaces 154. In one embodiment, the localization module 156 includes at least one camera integrated within either the wearable audio device 102 or the peripheral device 104, e.g., as the first sensor 118 or the second sensor 134. The localization module 156 may also include or employ an artificial neural network, deep learning engine or algorithm, or other machine learning algorithm trained to visually detect acoustic properties, locations, and orientations of the acoustically reflective surfaces 154 within the environment E from image data captured by the camera. In another example, the localization module 156 is arranged to collect data relating to the reverberation time and / or acoustic attenuation characteristics of the environment in which the user, the wearable audio device 102, or the peripheral device 104 is located. For example, the localization module 156 may include a dedicated speaker and may be configured to generate a designated sound signal (e.g., a "ping" or other signal outside the range of human hearing) and measure the reflected response (e.g., with a dedicated microphone). In one example, an absorption coefficient is calculated from the reverberation time or other characteristics of the overall environment and applied to the acoustically reflective surface 154 as an approximation. If the sound signal is specifically directed or directed to the acoustically reflective surface 154, the difference between the original signal and the originally received reflection may be used to calculate the absorption coefficient of the acoustically reflective surface 154. In one embodiment, the localization module may include, for example, a Global Positioning System (GPS) sensor embedded in the wearable audio device 102 or peripheral device 104, and the localization module 156 may selectively utilize data from acoustically reflective surfaces 154 that are within a certain threshold distance of each virtual sound source 144.

[0056] For example, once the localization data is obtained using the localization module 156, a path between each virtual sound source 144 and each acoustically reflective surface 154 can be determined in addition to the direct sound paths 148A and 148B discussed above. At the junction between each determined path and each acoustically reflective surface 154, there is a contact point CP. In one embodiment, as illustrated in FIG. 8, which is a top view of the audio system 100 in an environment E, the audio system 100 includes primary mirrored virtual sound sources 158A and 158B (collectively referred to as "primary mirrored virtual sound sources 158" or primary mirrored sound sources 158"). Each primary mirrored virtual sound source 158 is a new virtual sound source generated at a position equivalent to the position of the original virtual sound source 144 and mirrored with respect to the acoustically reflective surface 154. For example, as illustrated, a path (indicated by a dashed line in FIG. 8) between the virtual sound source 144B and the acoustically reflective surface 154A (illustrated as a wall) is determined. The point where the determined path intersects with the acoustically reflective surface 154A is labeled as the contact point CP. A copy of the virtual sound source 144B is generated as a primary mirrored virtual sound source 158A at a location equivalent to the location of the virtual sound source 144B after being mirrored with respect to the acoustically reflective surface 154A. When generated at the illustrated location, the simulated sound generated from this location of the primary mirrored sound source 158A simulates a primary or first reflected sound path 160A (shown in FIG. 8 as a dotted line) that simulates the sound from the virtual sound source 144B as if it were generated in the environment E and reflected from the acoustically reflective surface 154A to the location of the user's ear, i.e., approximately the location of the wearable audio device 102. A similar path can be determined and simulated to generate a primary mirrored virtual sound source 158B corresponding to the primary or first reflected sound path 160B of the virtual sound source 144C.

[0057] Similarly, the audio system 100 can generate secondary mirrored virtual sound sources 162A-162B (collectively referred to as "secondary mirrored virtual sound sources 162" or "secondary mirrored sound sources 162"). Each secondary mirrored virtual sound source 162 is a new virtual sound source generated at a position equivalent to the position of the original virtual sound source 144 and mirrored with respect to a different acoustically reflecting surface 154. For example, as illustrated, a two-part path (indicated by two dashed lines in FIG. 8 ) is determined, i.e. a first part extends from the virtual sound source 144B to an acoustically reflecting surface 154A (illustrated as a wall) and a second part extends from the end of the first part of the path to a second acoustically reflecting surface 154B (illustrated as a wall). The point where the second part of the determined path intersects with the acoustically reflecting surface 154B is labeled as contact point CP. A copy of the virtual sound source 144B is generated as a secondary mirrored virtual sound source 162A at a location equivalent to the location of the virtual sound source 144B after it has been mirrored with respect to the acoustically reflective surface 154B. When generated at the illustrated location, the simulated sound generated from this secondary mirrored sound source 162A location simulates a secondary or second reflected sound path 164A (shown in FIG. 8 as a dotted line) that simulates the sound from the virtual sound source 144B as if it were generated in the environment E and reflected from the acoustically reflective surfaces 154A and 154B to the location of the user's ears, i.e., approximately the location of the wearable audio device 102. A similar path can be determined and simulated to generate a secondary mirrored virtual sound source 162B corresponding to the secondary or second reflected sound path 164B reflected from the acoustically reflective surfaces 154A and 154C to simulate the secondary reflected audio of the virtual sound source 144C.

[0058] 7, the primary mirrored virtual sound source 158 and the secondary mirrored virtual sound source 162 are pinned or otherwise spatially locked to the orientation and position of the peripheral device 104. In other words, if the peripheral device 104 moves, rotates, pivots, tilts, or otherwise changes its position, location, or orientation within the environment E or relative to the wearable audio device 102, the virtual sound sources 144 in the environment E will move, rotate, pivot, tilt, or otherwise change their position, location, or orientation proportionally such that the position and orientation of each virtual sound source 144 is fixed relative to the peripheral device 104. Because the location, position, and / or orientation of the virtual sound source 144 will change with the peripheral device 104, each primary mirrored virtual sound source 158 and each secondary mirrored virtual sound source will also move such that they continue to simulate the reflection of the virtual sound source 144 around each acoustically reflective surface.

[0059] It should be understood that the primary and secondary mirror virtual sound sources 160 and 164 can be simulated using the primary and secondary mirror virtual sound sources 158 and 162 for any virtual source configuration discussed above, e.g., 5.1, 7.1, and 9.1 surround sound configurations, as well as configurations including at least one virtual subwoofer associated with a bass channel audio signal. Additionally, the present disclosure is not limited to primary and secondary reflections. For example, higher order reflections are possible, e.g., third order reflections, fourth order reflections, fifth order reflections, etc., but the computational power and processing time scale exponentially as the number of additional orders of reflections, and thus virtual sound sources, increases. In one embodiment, the audio system 100 is configured to simulate six virtual sound sources 144, e.g., corresponding to a 5.1 surround sound configuration. For each virtual sound source 144, a direct sound path 148 is calculated. For each virtual sound source 144, there are six primary or first-order reflected sound paths 160, corresponding to the first-order reflections from the four walls, the ceiling, and the floor (e.g., acoustically reflective surfaces 154). Each primary reflected path may reflect again from the other five remaining surfaces 154 to generate an exponential number of virtual sound sources and reflected sound paths. It should be understood that in some exemplary implementations of the audio system 100, the number of secondary reflections 164 depends on the geometry of the environment E, e.g., the shape of the room relative to the positions of the wearable audio device 102 and the virtual sound source 144. For example, in a rectangular room shape, once the primary or first-order reflected sound paths 160 are selected, certain secondary reflections 164 may not be physically possible, e.g., if the contact point CP needs to be located outside the room to obtain a valid secondary reflection path. Therefore, it should be appreciated that in an example having a rectangular room shape, rather than simulating five secondary reflection paths 164 for each primary reflection path 160, only three secondary reflection paths 164 may be simulated to account for invalid secondary reflections 164 caused by the particular room shape.For example, rather than simulating six primary reflections 160 and thirty secondary reflections 164 (e.g., where each of the six primary sound paths 160 is reflected from each of the five remaining walls), audio system 100 may simulate only six primary reflections 160 and eighteen secondary reflected sound paths 164 (e.g., each of the six primary reflections 160 from three of the five remaining walls). It should also be appreciated that audio system 100 may be configured to perform validity tests across all simulated paths to ensure that the path from each simulated sound source, for example to wearable audio device 102, is a valid path, i.e., physically feasible depending on the geometry of environment E.

[0060] Additionally, due to the potential processing power required to generate these first and second reflections in real-time, in one embodiment, the audio system 100 utilizes the processing power of the second circuit 122 of the peripheral device 104, e.g., using the second processor 124, the second memory 126, and / or the second set of non-transitory computer readable instructions 128. However, it should be appreciated that in some exemplary implementations of the audio system 100, the audio system 100 can utilize the processing power of the first circuit 106 of the wearable audio device 102 to simulate the first and second reflected sound sources discussed herein, e.g., using the first processor 108, the first memory 110, and / or the first set of non-transitory computer readable instructions 112. Furthermore, it should be appreciated that the audio system 100 can split the processing load between the first circuit 106 and the second circuit 122 in any conceivable combination.

[0061] 9, the audio system 100 can simulate two virtual sound sources, e.g., virtual sound sources 144B and 144C corresponding to a left channel audio signal and a right channel audio signal, which are spatially pinned, locked, or otherwise fixed relative to the second orientation O2 and the second position P2 of the peripheral device 104. As illustrated, when a user rotates or otherwise changes the orientation of the peripheral device 104, e.g., when the peripheral device 104 is rotated approximately 45 degrees clockwise around the second position P2, the positions of the virtual sound sources 144B and 144C will rotate approximately 45 degrees around the position P2 at a fixed distance from the peripheral device 104, such that after the peripheral device 104 is rotated, the positions of the virtual sound sources 144B and 144C relative to the peripheral device 104 will be the same as before the rotation. In particular, by a user rotating the peripheral device 104 45 degrees while maintaining the user's original head position, i.e., the first position P1 and first orientation O1 of the wearable audio device 102, the positions of each virtual sound source 144B and 144C, each primary mirrored sound source 158, and each secondary mirrored sound source 162 relative to the wearable audio device 102 will be changed. For example, as shown in Figure 9, rotating the peripheral device 104 clockwise approximately 45 degrees will cause the virtual sound source 144B to move away from the wearable audio device 102 while the virtual sound source 144C will move closer to the wearable audio device 102. Additionally, these changes result in proportional mirroring changes of each primary mirrored virtual sound source 144 and each secondary mirrored virtual sound source 162 to account for the movement of the virtual sound source 158 relative to the position P1 of the wearable audio device 102.Thus, at least one left HRTF 150 may include a change in the calculated distance of the virtual sound source 144B to simulate an increased distance to the wearable audio device 102, at least one left HRTF 150 may include a change in the calculated distance of the primary mirrored virtual sound source 158A to simulate an increased distance to the wearable audio device 102, and at least one left HRTF 150 may include a change in the calculated distance of the secondary mirrored virtual sound source 162A to simulate an increased distance to the wearable audio device 102. Similarly, at least one right HRTF 150 may include a change in the calculated distance of the virtual sound source 144B to simulate an increase in the distance to the wearable audio device 102, at least one left HRTF 150 may include a change in the calculated distance of the primary mirrored virtual sound source 158A to simulate an increase in the distance to the wearable audio device 102, and at least one left HRTF 150 may include a change in the calculated distance of the secondary mirrored virtual sound source 162A to simulate an increase in the distance to the wearable audio device 102. Similar modifications may be made using the left HRTF 150 and right HRTF 152 based on changes in the position and / or orientation of the virtual sound source 144C. Furthermore, while the above-described embodiment only discloses a simple rotation of the peripheral device 104 by 45 degrees in a clockwise rotation, more complex changes in orientation or position, such as tilting, translation, pivoting, or any combination of these movements, may be considered in a similar manner to that described above.

[0062] 10 and 11 illustrate example steps of a method 200 according to the present disclosure. The method 200 may, for example, include receiving (step 202) a first modified audio signal 146A from the peripheral device 104 via the wearable audio device 102, the first modified audio signal 146A being modified using a first head-related transfer function (HRTF) 150 based at least in part on an orientation O1 of the wearable audio device 102 relative to the peripheral device 104, receiving (step 204) a second modified audio signal 146B from the peripheral device 104 via the wearable audio device 102, the second modified audio signal 146B being modified using a second head-related transfer function (HRTF) 152 based at least in part on an orientation O1 of the wearable audio device 102 relative to the peripheral device 104, and obtaining a position P1 of the wearable audio device 102 relative to the peripheral device 104 within an environment E. The method includes obtaining a first modified audio signal 146A and a second modified audio signal 146B based at least in part on a calculated distance D1-D3 between a position P1 of the wearable audio device 102 and a position P2 of the peripheral device 104 (step 206); obtaining an orientation O2 of the peripheral device 104 relative to the wearable audio device 102, where the first HRTF 150 and the second HRTF 152 are based in part on the orientation O2 of the peripheral device 104 (step 208); rendering the first modified audio signal 146A using a first speaker 120A of the wearable audio device 102 (step 210); and rendering the second modified audio signal 146B using a second speaker 120B of the wearable audio device 102 (step 212). Optionally, method 200 may further include receiving localization data from a localization module 156 in environment E (step 214) and determining locations of multiple acoustically reflective surfaces 154 in environment E based on the localization data (step 216).

[0063] All definitions and those used herein should be understood to control for any dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0064] The indefinite articles "a" and "an," as used in the specification and claims, unless expressly indicated otherwise, should be understood to mean "at least one."

[0065] The phrase "and / or" as used in the specification and claims should be understood to mean "either or both" of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with "and / or" should be construed in the same manner, i.e., "one or more" of the elements so conjoined. Other elements, whether related or unrelated to the elements specifically identified, may optionally be present other than the elements specifically identified by the "and / or" clause.

[0066] As used herein and in the claims, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" should be interpreted as inclusive, i.e., including at least one, but also including more than one, of a number or list of elements, and optionally including additional non-listed items. Only when terms such as "only one of" or "exactly one of," or, when used in the claims, "consisting of," refer to the inclusion of exactly one element of a number or list of elements. In general, the term "or" as used herein is only interpreted as indicating exclusive alternatives (i.e., "one or the other but not both") when preceded by terms of exclusivity such as "either," "one of," "only one of," or "exactly one of."

[0067] As used in this specification and claims, the phrase "at least one" in reference to a list of one or more elements should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed in the list of elements, and not excluding any combination of elements in the list of elements. This definition also allows for elements, whether related or unrelated to the specifically identified elements, to be optionally present other than the elements specifically identified in the list of elements to which the phrase "at least one" refers.

[0068] It is also to be understood that, unless expressly indicated otherwise, in any method claimed herein that includes more than one step or action, the order of the method steps or actions is not necessarily limited to the order in which the method steps or actions are recited.

[0069] In the claims, as well as in the above specification, all transitional phrases such as "comprising," "including," "carrying," "having," "containing," "involving," "holding," "composed of," and the like, are to be understood to be open-ended, i.e., to mean including but not limited to. The transitional phrases "consisting of" and "consisting essentially of" only are closed or semi-closed transitional phrases, respectively.

[0070] The above-described examples of the described subject matter can be implemented in any of numerous ways. For example, some aspects may be implemented in hardware, software, or a combination thereof. If at least a portion of any aspect is implemented as software, the software code can be executed on any suitable processor, or collection of processors, whether provided on a single device or computer, or distributed across multiple devices / computers.

[0071] The present disclosure may be implemented as a system, method, and / or computer program product at any level of technical detail contemplated. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions that cause a processor to perform aspects of the present disclosure.

[0072] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, punch cards or raised structures in grooves or mechanically encoded devices having instructions recorded thereon, and suitable combinations of the above. As used herein, a computer-readable storage medium is not to be construed as a transient signal itself, such as a freely propagating electromagnetic wave such as an electric wave, an electromagnetic wave propagating through a transmission medium such as a waveguide (e.g., a light pulse passing through a fiber optic cable), or an electrical signal traveling down an electrical wire.

[0073] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card, or network interface, of each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0074] The computer readable program instructions for carrying out the operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state configuration data, integrated circuit configuration data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, procedural programming languages ​​such as the "C" programming language, or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet Service Provider). In some examples, electronic circuitry, including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by individualizing the electronic circuitry using state information of the computer-readable program instructions to perform aspects of the present disclosure.

[0075] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to examples of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0076] Computer-readable program instructions may be provided to a processor of a special purpose computer, or other programmable data processing device, to manufacture a machine, whereby the instructions executing via the processor of the computer or other programmable data processing device create means for performing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. Furthermore, these computer-readable program instructions may be stored on a computer-readable storage medium that can direct a computer, programmable data processing device, and / or other device to function in a particular manner, whereby the computer-readable storage medium on which the instructions are stored includes an article of manufacture having instructions that implement aspects of the functions / operations specified in the flowcharts and / or block diagrams or blocks.

[0077] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to generate a computer-implemented process for a series of operational steps to be performed on the computer, other programmable apparatus, or other device, such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0078] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various examples of the present disclosure. In this regard, each block of the flowcharts or block diagrams may correspond to a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions described in the blocks may occur out of the order described in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or in some cases, the blocks may be executed in the reverse order depending on the functionality involved. Furthermore, it should be noted that each block of the block diagrams and / or flowchart illustrations, as well as combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented in a dedicated hardware-based system that performs a specific function or operates or executes a combination of dedicated hardware and computer instructions.

[0079] Other implementations are within the scope of the following claims as well as other claims to which the applicant may be entitled.

[0080] While various examples have been described and illustrated herein, those of ordinary skill in the art will readily conceive of various other means and / or structures for performing the functions and / or results and / or obtaining one or more of the advantages described herein, and each of such modifications and / or variations is deemed to be within the scope of the examples described herein. More generally, those of ordinary skill in the art will readily appreciate that all of the parameters, dimensions, materials, and configurations described herein are exemplary, and furthermore, that the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications for which the teachings of the present invention are used. Those of ordinary skill in the art will recognize or be able to ascertain, using no more than routine experimentation, many equivalents to the specific examples described herein. Thus, it is to be understood that the foregoing examples are presented by way of example only, and that, within the scope of the appended claims and equivalents thereof, the examples may be practiced otherwise than as specifically described and claimed. Examples of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is within the scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent. [Explanation of symbols]

[0081] 100 Audio System 102 Wearable Devices 104 Peripheral Devices 106 First Circuit 108 First Processor 110 First Memory 112 First set 114 First communication module 116 First Antenna 118 First Sensor 120 Speaker 120A 1st speaker 120B Second speaker 122 Second Circuit 124 Second Processor 126 Second Memory 128 Second Set 130 Second communication module 132 Second Antenna 134 Second Sensor 136 Device Speaker 138 Display 140 Video Content 142 Data Connection

Claims

1. 1. A computer program for simulating an audio signal, the computer program comprising a set of non-transitory computer readable instructions stored in a memory, the set of non-transitory computer readable instructions being executable by a processor; and Obtaining or receiving an orientation of a wearable audio device relative to peripheral devices in an environment; generating a first modified audio signal, the first modified audio signal being modified using a first head-related transfer function (HRTF) based at least in part on the orientation of the wearable audio device relative to the peripheral device; generating a second modified audio signal, the second modified audio signal being modified using a second head-related transfer function (HRTF) based at least in part on the orientation of the wearable audio device relative to the peripheral device; transmitting the first modified audio signal and the second modified audio signal to the wearable audio device, the first modified audio signal being configured to be rendered using a first speaker of the wearable audio device and the second modified audio signal being configured to be rendered using a second speaker of the wearable audio device; the first HRTF and the second HRTF are further configured to utilize localization data from a localization module in the environment corresponding to locations of a plurality of acoustically reflective surfaces in the environment, the localization data being obtained by measuring reflected responses in the environment of a predetermined sound signal generated by the localization module and collecting data relating to the reverberation time and / or acoustic attenuation characteristics of the environment.

2. The computer program described in claim 1, wherein the specified sound signal is a sound signal outside the human audible range.

3. The set of non-transitory computer readable instructions comprises:

2. The computer program product of claim 1, further configured to obtain or receive a position of the wearable audio device relative to a position of the peripheral device in the environment, and wherein modifying the first modified audio signal and modifying the second modified audio signal includes attenuation based at least in part on a calculated distance between the position of the wearable audio device and the position of the peripheral device.

4. The set of non-transitory computer readable instructions comprises:

2. The computer program product of claim 1 , further configured to obtain or receive an orientation of the peripheral device relative to the wearable audio device, and wherein the first HRTF and the second HRTF are based in part on the orientation of the peripheral device relative to the wearable audio device.

5. 2. The computer program product of claim 1 , wherein the first modified audio signal and the second modified audio signal are configured to simulate a first direct sound originating from a first virtual sound source proximate to a center of the peripheral device.

6. 2. The computer program product of claim 1 , wherein generating the first modified audio signal and generating the second modified audio signal comprises simulating a first direct sound originating from a first virtual sound source proximate a position of the peripheral device in the environment and simulating a second direct sound originating from a second virtual sound source proximate the position of the peripheral device.

7. The computer program product of claim 1 , wherein generating the first modified audio signal and generating the second modified audio signal comprises simulating surround sound.

8. 2. The computer program product of claim 1 , wherein generating the first modified audio signal and generating the second modified audio signal comprises using the first HRTF and the second HRTF, respectively, for only a subset of all available audio frequencies and / or channels.

9. 2. The computer program product of claim 1, wherein generating the first modified audio signal comprises simulating a first direct sound originating from a first virtual sound source proximate to the peripheral device, and simulating a first reflected sound corresponding to a simulated reflection of the first direct sound from a first acoustically reflective surface of the plurality of acoustically reflective surfaces.

10. 10. The computer program product of claim 9, wherein generating the first modified audio signal includes simulating a secondary reflection corresponding to a simulated reflection of the primary reflection from a second acoustically reflective surface of the plurality of acoustically reflective surfaces.

11. The computer program product of claim 1 , wherein the first modified audio signal and the second modified audio signal correspond to video content displayed on the peripheral device.

12. 2. The computer program product of claim 1, wherein the orientation of the wearable audio device relative to the peripheral device is determined using at least one sensor located on, within, or in proximity to the wearable audio device or the peripheral device, and the at least one sensor is selected from a gyroscope, an accelerometer, a magnetometer, a global positioning sensor (GPS), a proximity sensor, a microphone, a lidar sensor, or a camera.

13. 1. A method for simulating an audio signal, comprising the steps of: receiving localization data from a localization module in the environment; determining locations of a plurality of acoustically reflective surfaces within the environment based on the localization data; receiving, via a wearable audio device from a peripheral device, a first modified audio signal, the first modified audio signal being modified using a first head-related transfer function (HRTF) based at least in part on an orientation of the wearable audio device relative to the peripheral device; receiving a second modified audio signal from the peripheral device via the wearable audio device, the second modified audio signal being modified using a second head-related transfer function (HRTF) based at least in part on the orientation of the wearable audio device relative to the peripheral device; Rendering the first modified audio signal using a first speaker of the wearable audio device; and and rendering the second modified audio signal using a second speaker of the wearable audio device; 20. A method according to claim 19, wherein the localization data is obtained by measuring a response of a predetermined sound signal generated by the localization module reflected within the environment and collecting data relating to the reverberation time and / or sound attenuation characteristics of the environment.

14. The method of claim 13, wherein the predetermined sound signal is a sound signal outside the range of human audibility.

15. The method further comprising:

14. The method of claim 13, further comprising obtaining a position of a wearable audio device relative to the peripheral device in an environment, and wherein modifying the first modified audio signal and modifying the second modified audio signal are based at least in part on a calculated distance between the position of the wearable audio device and a position of the peripheral device.

16. 14. The method of claim 13, further comprising obtaining an orientation of the peripheral device relative to the wearable audio device, wherein the first HRTF and the second HRTF are based in part on the orientation of the peripheral device.

17. 14. The method of claim 13, wherein the first modified audio signal and the second modified audio signal are configured to simulate a first direct sound originating from a first virtual sound source proximate to a center of the peripheral device.

18. 14. The method of claim 13, wherein rendering the first modified audio signal and rendering the second modified audio signal comprises simulating a first direct sound originating from a first virtual sound source proximate a position of the peripheral device in an environment, and simulating a second direct sound originating from a second virtual sound source proximate the position of the peripheral device.

19. The method of claim 13 , wherein generating the first modified audio signal and generating the second modified audio signal comprises simulating surround sound.

20. 14. The method of claim 13, wherein generating the first modified audio signal and generating the second modified audio signal comprises using the first HRTF and the second HRTF, respectively, for only a subset of all available audio frequencies and / or channels.

21. 14. The method of claim 13, wherein rendering the first modified audio signal comprises: simulating a first direct sound originating from a first virtual sound source proximate to the peripheral device; and simulating a first reflected sound corresponding to a simulated reflection of the first direct sound from a first acoustically reflective surface of the plurality of acoustically reflective surfaces.

22. 22. The method of claim 21 , wherein rendering the first modified audio signal includes simulating a secondary reflection corresponding to a simulated reflection of the primary reflection from a second acoustically reflective surface of the plurality of acoustically reflective surfaces.

23. The method of claim 13 , wherein the peripheral device includes a display configured to display video content associated with the first modified audio signal and the second modified audio signal.

24. 14. The method of claim 13, wherein the orientation of the wearable audio device relative to the peripheral device is determined using at least one sensor located on, within, or in proximity to the wearable audio device or the peripheral device, and the at least one sensor is selected from a gyroscope, an accelerometer, a magnetometer, a global positioning sensor (GPS), a proximity sensor, a microphone, a lidar sensor, or a camera.

25. 1. A system for simulating audio, comprising: A peripheral device configured to acquire or receive an orientation of a wearable audio device relative to the peripheral device in an environment, the peripheral device further configured to generate a first modified audio signal using a first head-related transfer function (HRTF) based on the orientation of the wearable audio device relative to the peripheral device, and to generate a second modified audio signal using a second head-related transfer function (HRTF) based on the orientation of the wearable audio device relative to the peripheral device; The wearable audio device comprises: receiving the first modified audio signal; receiving the second modified audio signal; and a first speaker configured to render the first modified audio signal using the first speaker; a second speaker configured to render the second modified audio signal using the second speaker; the peripheral device further comprising a localization module; The system, wherein the first HRTF and the second HRTF are further configured to utilize localization data from a localization module in the environment corresponding to locations of a plurality of acoustically reflective surfaces in the environment, the localization data being obtained by measuring a response of a predetermined sound signal generated by the localization module reflected within the environment and collecting data related to the reverberation time and / or acoustic attenuation characteristics of the environment.

26. The system of claim 25, wherein the predetermined sound signal is a sound signal outside the range of human hearing.

Images