Head-mounted device for generating binaural audio

Abstract

A head mounted device for generating binaural audio content includes a first rod coupled to a first microphone housing, the first microphone housing respectively surrounding a first front microphone and a first rear microphone generating an acoustic signal. The first microphone housing includes a first front port facing downward and a first rear port facing backward. The device includes a second rod coupled to a second microphone housing, the second microphone housing respectively surrounding a second front microphone and a second rear microphone generating an acoustic signal. The second microphone housing includes a second front port facing downward and a second rear port facing backward. The device includes a binaural audio processor, the binaural audio processor including a beamformer and a storage device. The beamformer generates a first beamformer signal based on an acoustic signal from the first front microphone and the first rear microphone, and generates a second beamformer signal based on an acoustic signal from the second front microphone and the second rear microphone. The storage device stores the first and second beamformer signals as a dual channel file.

Description

[0001] Priority claim

[0002] This application claims priority to U.S. patent application Ser. No. 16 / 370,190, filed on Mar. 29, 2019, which is incorporated herein by reference in its entirety. Background Art

[0003] Currently, many consumer electronic devices are suitable for capturing audio and / or visual content. For example, a user can use a built-in camera on a mobile device to quickly capture events or moments that occur in the user's life.

[0004] However, the time required to retrieve the mobile device may still be too long to capture some fleeting events or moments. Users may also feel as if they must remove themselves from being present and experiencing the event or moment to retrieve their mobile device. Furthermore, when played back, the audio portion of the content recorded using the mobile device may not adequately reproduce the sound perceived by the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In the drawings, which are not necessarily drawn to scale, the same reference numerals may describe similar components in different views. The same reference numerals with different letter suffixes may represent different instances of similar components. Some embodiments are illustrated by way of example and not limitation in the accompanying drawings, in which:

[0006] Figure 1 A perspective view of a head mounted device for generating binaural audio is shown according to an example embodiment.

[0007] Figure 2 According to an example embodiment, Figure 1 A bottom view of a head mounted device.

[0008] Figure 3A-3B According to an example embodiment, Figure 1 Detail of a portion of a microphone housing of a head-mounted device.

[0009] Figure 4 is used according to various aspects of the present disclosure from Figure 1 An exemplary flow chart of a process for generating binaural audio using a head mounted device.

[0010] Figure 5 A method for generating a ... Figure 1 Block diagram of a system for binaural audio in a head mounted device.

[0011] Figure 6 An example embodiment includes Figure 5 Block diagram of the binaural audio processor in the system in FIG.

[0012] Figure 7 is a block diagram illustrating a representative software architecture that may be used in conjunction with the various hardware architectures described herein.

[0013] Figure 8 is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium (eg, a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to some example embodiments.

[0014] Fig. 9 is a high-level functional block diagram of an example head mounted device communicatively coupled to a mobile device and a server system via various networks. DETAILED DESCRIPTION

[0015] The following description includes systems, methods, techniques, instruction sequences and computing machine program products that embody the illustrative embodiments of the present disclosure. In the following description, for the purpose of explanation, many specific details are set forth to provide an understanding of the various embodiments of the subject matter of the present invention. However, it will be apparent to those skilled in the art that embodiments of the subject matter of the present invention can be practiced without these specific details. Typically, well-known instruction instances, protocols, structures and techniques need not be shown in detail.

[0016] In order to improve audio recordings captured by current electronic mobile devices, some embodiments of the present disclosure are directed to a head mounted device 100 that can capture audio content that, when played back, is an imitation of the sounds perceived by the user of the head mounted device 100. Specifically, the head mounted device 100 can record audio using microphones that are arranged to create a 3-dimensional (3D) sound sensation for the listener as it existed when the audio was recorded. This is called binaural audio. The playback of the captured audio content will have the effect of binaural audio with stereo separation and spectral content that mimics the response of the human ear and head. The design of the head mounted device 100 uses, among other things, the diffraction pattern of the human head, the placement of the microphones on both sides of the head, and beamforming techniques.

[0017] Figure 1 A perspective view of a head mounted device 100 for generating binaural audio is shown according to an example embodiment. Figure 2 According to an example embodiment, Figure 1 A bottom view of the head mounted device 100 is shown. Figure 1 and Figure 2, the head-mounted device 100 is a pair of glasses. In some embodiments, the head-mounted device 100 may be sunglasses or goggles. Some embodiments may include one or more wearable devices, such as a pendant with an integrated camera that is integrated with, communicates with, or is coupled to the head-mounted device 100 or a client device. Any desired wearable device may be used in conjunction with embodiments of the present disclosure, such as a watch, headphones, a wristband, earbuds, clothing (such as a hat or jacket with an integrated electronic device), a clip-on electronic device, or any other wearable device. It should be understood that, although not shown, one or more portions of the system included in the head-mounted device may be included in a client device (e.g., Figure 8 In the machine 800). For example, Figure 5 and Figure 6 One or more of the elements shown in may be included in the head mounted device 100 and / or the client device.

[0018] As used herein, the term "client device" may refer to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics, a game console, a set-top box, or any other communication device that a user may use to access a network.

[0019] exist Figure 1 and Figure 2 In the embodiment, the head mounted device 100 is a pair of glasses, which includes a frame 103, which includes an eyeline (or a frame) coupled to two rods (or temples) via hinges and end pieces, respectively. The eyeline of the frame 103 carries or holds a pair of lenses 104_1, 104_2. The frame 103 includes a first (e.g., right) side coupled to the first rod and a second (e.g., left) side coupled to the second rod. The first side is opposite to the second side of the frame 103.

[0020] The device 100 further includes a camera module, which includes camera lenses 102_1, 102_2 and at least one image sensor. The camera lens can be a perspective camera lens or a non-perspective camera lens. The non-perspective camera lens can be, for example, a fisheye lens, a wide-angle lens, an omnidirectional lens, etc. The image sensor captures digital video through the camera lens. The image can also be a still image frame or a video including multiple still image frames. The camera module can be coupled to the frame 103. As shown in FIG. Figure 1 and Figure 2As shown in FIG, frame 103 is coupled to camera lenses 102_1, 102_2 so that the camera lenses face forward. Camera lenses 102_1, 102_2 may be perpendicular to lenses 104_1, 104_2. The camera module may include dual front cameras separated by the width of frame 103 or the width of the head of a user of device 100.

[0021] exist Figure 1 and Figure 2 , two rods (or temples) are coupled to microphone housings 101_1, 101_2, respectively. The first rod and the second rod are coupled to opposite sides of the frame 103 of the head mounted device 100. The first rod is coupled to the first microphone housing 101_1 and the second rod is coupled to the second microphone housing 101_2. The microphone housings 101_1, 101_2 can be coupled to the rods between the location of the frame 103 and the temple foot cover. When the user wears the device 100, the microphone housings 101_1, 101_2 can be located on either side of the user's temple.

[0022] like Figure 2 As shown in , microphone housings 101_1, 101_2 surround a plurality of microphones 110_1 to 110_N (N>1). Microphones 110_1 to 110_N are air interface pickup devices that convert sound into electrical signals. More specifically, microphones 110_1 to 110_N are transducers that convert sound pressure into electrical signals (e.g., acoustic signals). Microphones 110_1 to 110_N may be digital or analog micro-electromechanical systems (MEMS) microphones. The acoustic signals generated by microphones 110_1 to 110_N may be pulse density modulation (PDM) signals.

[0023] exist Figure 2 In the embodiment, the first microphone housing 101_1 surrounds the microphones 110_3 and 110_4, and the second microphone housing 101_2 surrounds the microphones 110_1 and 110_2. In the first microphone housing 101_1, the first front microphone 110_3 and the first rear microphone 110_4 are separated by a predetermined distance d. 1 And form a first-order differential microphone array. In the second microphone housing 101_2, the second front microphone 110_1 and the second rear microphone 110_2 are also separated by a predetermined distance d. 2 And form a first-order differential microphone array. The predetermined distance d 1 and d 2 It can be the same distance or different distances. The predetermined distance d can be set based on the Nyquist frequency 1 and d 2 Content above the beamformer's Nyquist frequency is not recoverable, especially speech. The Nyquist frequency is determined by the following equation:

[0024]

[0025] In this equation, c is the speed of sound and d is the spacing between the microphones. Using this equation, in one embodiment, the predetermined distance d 1 and d 2 It can be set to any value of d that results in frequencies above 6kHz (which is the cutoff frequency for wideband speech).

[0026] Although in Figure 1 In the embodiment, the system 100 includes four microphones 110_1 to 110_4, but the number of microphones may vary. In some embodiments, the microphone housings 101_1, 101_2 may include at least two microphones and may form a microphone array. Each of the microphone housings 101_1, 101_2 may also include a battery.

[0027] The user naturally perceives audio through two ears separated by the head, so that the user can distinguish the direction from which the sound originates. Therefore, by placing the microphone housings 101_1, 101_2 on the poles of the head mounted device 100, the head mounted device 100 can achieve capturing the sound perceived by the user wearing the head mounted device 100.

[0028] refer to Figure 2 , each of the microphone housings 101_1, 101_2 includes a front port and a rear port. The front port of the first microphone housing 101_1 is coupled to the microphone 110_3 (e.g., the first front microphone) and the rear port of the first microphone housing 101_1 is coupled to the microphone 110_4 (e.g., the first rear microphone). The front port of the second microphone housing 101_2 is coupled to the microphone 110_1 (e.g., the second front microphone) and the rear port of the second microphone housing 101_2 is coupled to the microphone 110_2 (e.g., the second rear microphone). In one embodiment, the microphones 101_1 to 101_4 can be further moved toward the temple on the pole of the device 100 (e.g., the back of the device 100) to emphasize the binaural effect captured by the microphones.

[0029] Figure 3A-3B According to an example embodiment, Figure 1 Detail of a portion of a microphone housing of a head mounted device. Specifically, Figure 3A Details of microphone 110_1 (eg, a second front microphone) and its associated front port are shown. Figure 3A Details of the microphone 110_1 coupled to the front port in the second microphone housing 101_2 are shown, but it should be understood that the details of the microphone 110_3 (eg, the first front microphone) coupled to the front port in the first microphone housing 101_1 are similar. Figure 3A Details in.

[0030] Figure 3A 1 is a cross-sectional view of the front microphone 110_1 and the acoustic path 112. As shown, the acoustic path 112 passes through the space between the block 114 (e.g., the second microphone housing 101_2) and the second rod housing 113. The rod housing 113 can be made of metal. The front microphone 110_1 and the front port point (or face) downward. For example, when the user stands and wears the device 100, Figure 3A The front port in the microphone is open in the direction towards the user's feet on the ground. This design allows the length and acoustic mass of the front acoustic port to be minimized while ensuring that the front microphone 110_1 is protected in a small pocket that can reduce noise effects (e.g., wind noise turbulence, etc.).

[0031] Figure 3B According to an example embodiment, Figure 1 Specifically, Figure 3B Details of microphone 110_2 (eg, a second rear microphone) and the rear port associated therewith are shown. Figure 3B Details of the microphone 110_2 coupled to the rear port in the second microphone housing 101_2 are shown, but it should be understood that the details of the microphone 110_4 (eg, the first rear microphone) coupled to the rear port in the first microphone housing 101_1 are similar. Figure 3B Details in.

[0032] Figure 3B 1 is a cross-sectional view of the rear microphone 110_2 and the acoustic path 112. As shown, the acoustic path 112 passes through the space between the block 114 (eg, the second microphone housing 101_2) and the housing 113 of the second pole. Figure 3B The housing 113 of the rod in the device may also be made of metal. The rear microphone 110_2 and the rear port point (or face) backward. For example, when the user wears the device 100, Figure 3B The rear port in the device 100 opens in a direction toward the back of the user's head or the rear of the device 100 (e.g., toward the end of the temple). Since the port is rearward in this embodiment, direct wind contact with the rear port is avoided when the user wears the device 100. This design further allows the length and acoustic mass of the rear acoustic port to be minimized while ensuring that the rear microphone 110_2 is protected in a small area that can reduce noise effects (e.g., wind noise turbulence, etc.). In one embodiment, a mechanical filter may be applied to the rear microphone 110_2 to further improve noise immunity.

[0033] like Figure 3A-3BAs shown in , microphones 110_1 to 110_4 can be part of a microphone assembly stack that includes a flexible circuit board and a pressure sensitive adhesive (PSA) stack that includes a waterproof membrane sandwiched between the PSA layers. The waterproof membrane can protect the microphones 110_1 to 110_4 from water ingress and air leakage. The PSA stack is then coupled to a microphone assembly housing. In some embodiments, the microphone assembly housing is a block 114 or housing 113 of the second rod.

[0034] Figure 4 The use of various aspects of the present disclosure comes from Figure 1 100 generates binaural audio. Although the flowchart may describe the operations as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process terminates when its operations are completed. A process may correspond to a method, a procedure, etc. The steps of a method may be performed in whole or in part, may be performed in combination with some or all of the steps in other methods, and may be performed by any number of different systems (such as Figure 1 and / or Figure 8 Process 400 may also be performed by a system including Figure 1 The processor in the head mounted device 100 or the processor included in Figure 8 The processor in the client device 800 executes.

[0035] Process 400 begins at operation 401, where microphones 110_1 to 110_4 generate acoustic signals. Microphones 110_1 to 110_4 may be MEMS microphones that convert sound pressure into electrical signals (e.g., acoustic signals). A first front microphone 110_3 and a first rear microphone 110_4 are enclosed in a first microphone 101_1 housing coupled to a first rod of a head mounted device 100. The first front microphone 110_3 and the first rear microphone 110_4 form a first order differential microphone array. A second front microphone 110_1 and a second rear microphone 110_2 are enclosed in a second microphone housing 101_2 coupled to a second rod of the head mounted device 100. The second front microphone 110_1 and the second rear microphone 110_2 form a first order differential microphone array. The first rod and the second rod are coupled to opposite sides of a frame 103 of the head mounted device 100. The acoustic signal may be a pulse density modulation (PDM) signal.

[0036] At operation 402, the audio codecs 501_1, 501_2 decode acoustic signals from the microphones 110_1 to 110_4. The audio codec 501_2 decodes the acoustic signals from the first front microphone 110_ and the first rear microphone 110_4 to generate a first decoded acoustic signal, and the audio codec 501_1 decodes the second front microphone 110_1 and the second rear microphone 110_2 to generate a second decoded acoustic signal. The first and second decoded acoustic signals are pulse code modulation (PCM) signals. The first decoded acoustic signal is a PCM signal based on the acoustic signal from the first front microphone 110_3 and the first rear microphone 110_4. The second decoded acoustic signal is a PCM signal based on the acoustic signal from the second front microphone 110_1 and the second rear microphone 110_2.

[0037] At operation 403, the TDM 502 processes the decoded acoustic signals from the audio codecs 501_1, 501_2. The TDM 502 processes the first and second decoded acoustic signals by time division multiplexing the first and second decoded acoustic signals. The TDM processed signal includes the first decoded acoustic signal and the second decoded acoustic signal.

[0038] At operation 404, the beamformer 601 beamforms the TDM processed signal. As further described below, the beamformer 601 may be a fixed beamformer including a fixed beam pattern that is sub-cardioid or cardioid. The beamformer 601 beamforms the first decoded signal to generate a first beamformer signal, and beamforms the second decoded signal to generate a second beamformer signal.

[0039] At operation 405, the storage device 604 stores the beamformer signal as a two-channel file. The two-channel file may be a two-channel PCM file or a two-channel Advanced Audio Coding (AAC) / PCM file. The storage device 604 may be a flash memory device.

[0040] In one embodiment, the noise suppressor 602 suppresses noise from the first beamformer signal and the second beamformer signal and generates a first noise-suppressed signal and a second noise-suppressed signal. The speech enhancer 603 can enhance the speech from the first noise-suppressed signal and the second noise-suppressed signal to generate a first clean signal and a second clean signal, respectively. In this embodiment, the storage device 604 stores the first and second clean signals as dual-channel PCM files.

[0041] Figure 5 A method for generating a ... Figure 1100. In some embodiments, one or more portions of the system 500 may be included in the head mounted device 100 or may be included in a client device (e.g., Figure 8 In the machine 800).

[0042] The system 500 includes microphones 110_1 to 110_N, audio codecs 501_1, 501_2, a time division multiplexer (TDM) 502, and a binary audio processor 503. The first front microphone 110_3 and the first rear microphone 110_4 enclosed in the first microphone housing 101_1 form a first-order differential microphone array. Similarly, the second front microphone 110_1 and the second rear microphone 110_2 enclosed in the second microphone housing 101_2 form another first-order differential microphone array. The microphones 110_1 to 110_4 may be analog or digital MEMS microphones. The acoustic signals generated by the microphones 110_1 to 110_4 may be pulse density modulation (PDM) signals.

[0043] The audio codec 501_1 decodes the acoustic signal from the first front microphone 110_3 and the first rear microphone 110_4 to generate a first decoded acoustic signal. The audio codec 501_2 decodes the acoustic signal from the second front microphone 110_1 and the second rear microphone 110_2 to generate a second decoded acoustic signal. The first and second decoded acoustic signals may be pulse code modulation (PCM) signals. In one embodiment, the audio codecs 501_1, 501_2 decode the acoustic signal as a PDM signal from a single-bit PDM format to a multi-bit pulse code modulation (PCM) format. The audio codecs 501_1, 501_2 may include a PDM input with a filter that converts the PDM signal to a PCM format. In one embodiment, the audio codecs 501_1, 501_2 use a microcontroller with a synchronous serial interface to capture the PDM data stream from the microphones 110_1 to 110_4, and use a filter implemented in software to convert the PDM data stream to a PCM format.

[0044] The PCM signal may be interpreted by an interface of the binaural audio processor 503. In some embodiments, the binaural audio processor 503 is a silicon on chip (SoC). The SoC may include an interface, such as an I2S interface, to receive and interpret the PCM signal.

[0045] In one embodiment, the interface of the binaural audio processor 503 can only process one packet (e.g., 2 channels of audio), and the time division multiplexer (TDM) 502 in the system 500 receives and processes the first and second decoded acoustic signals (e.g., PCM signals) to generate a TDM-processed signal. The TDM 502 time-division multiplexes the first and second decoded acoustic signals to generate a TDM-processed signal. In one embodiment, the system 500 further oversamples the TDM-processed signal to allow input of four microphone signals. The system 500 may also include a switch for creating additional slots to allow microphone signals.

[0046] In one embodiment, the microphones 110_1 to 110_4 are digital MEMS microphones. The acoustic signals generated by the digital MEMS microphones are relatively unaffected by noise, but signal integrity is still an issue due to distortion caused by parasitic capacitance, resistance, and inductance between the microphones 110_1 to 110_4 outputs and the SoC (such as the binaural audio processor 503). Impedance mismatches can also produce reflections that can distort the signal in applications where the distance between the digital microphones 110_1 to 110_4 and the SoC is long. In one embodiment, the microphones 110_1 to 110_4 are attached to a flexible circuit that is designed to maximize signal integrity and also minimize trace lengths between components. In this embodiment, the flexible circuit is enclosed in the microphone housings 101_1, 101_2.

[0047] Figure 6 An example embodiment includes Figure 5 FIG. 5 is a block diagram of a binaural audio processor 503 in a system 500 in FIG. 5 . The binaural audio processor 503 includes a beamformer 601 , a noise suppressor 602 , a speech enhancer 603 and a storage device 604 .

[0048] In one embodiment, the binaural audio processor 503 includes an interface for receiving TDM processed signals. As described above, the TDM processed signals are generated based on the acoustic signals from the first front microphone 110_3, the first rear microphone 110_4, the second front microphone 110_1, and the second rear microphone 110_2. In one embodiment, the microphones 110_1 to 110_4 are inherently omnidirectional digital MEMS microphones.

[0049] The beamformer 601 with directional steering characteristics is a differential beamformer that allows a flat frequency response except for the Nyquist frequency. The beamformer 601 uses the transfer function of a first-order differential microphone array. For two microphones, the transfer function of a first-order differential microphone array is as follows:

[0050]

[0051]

[0052] In the equations above, thetaθ is an angle and betaβ is 180 degrees, which simplifies to E=A+B cos θ for a fixed frequency (or frequency independent) beam (e.g., a beamformer signal). E in the simplified equation is the fixed frequency output of the beamformer. In one embodiment, the beamformer 601 is a fixed beamformer including a fixed beam pattern that is sub-cardioid with A and B coefficients of 0.25 and 0.75, respectively. In one embodiment, the beamformer 601 is a fixed beamformer including a cardioid fixed beam pattern with A and B coefficients of 0.5 and 0.5, respectively.

[0053] In one embodiment, the beamformer 601 receives acoustic signals from the first front microphone 110_3, the first rear microphone 110_4, the second front microphone 110_1, and the second rear microphone 110_2. In one embodiment, the beamformer 601 receives a TDM-processed signal. The beamformer 601 generates a first beamformer signal based on the acoustic signals from the first front microphone 110_3 and the first rear microphone 110_4, and generates a second beamformer signal based on the acoustic signals from the second front microphone 110_1 and the second rear microphone 110_2. The storage device 604 can store the first and second beamformer signals as a dual-channel file.

[0054] The noise suppressor 602 suppresses noise from the first beamformer signal and the second beamformer signal. The noise suppressor 602 is a dual channel noise suppressor and generates a first noise suppressed signal and a second noise suppressed signal. In one embodiment, the noise suppressor 602 may implement a noise suppression algorithm.

[0055] The speech enhancer 603 enhances the speech from the first noise-suppressed signal and the second noise-suppressed signal to generate the first clean signal and the second clean signal. In one embodiment, the speech enhancer 603 can implement model-based speech enhancement. The speech enhancer 603 can search for multiple speech features in the first and second noise-suppressed signals. When the speech enhancer 603 identifies the first and second noise-suppressed signals in the part that matches at least one speech feature in the speech feature, the speech enhancer 603 enhances or emphasizes the identified part. In one embodiment, the speech enhancer 603 can implement a speech enhancement algorithm.

[0056] The storage device 604 stores the first and second clean signals from the speech enhancer 603 as a dual channel file. The dual channel file may be a dual channel PCM file (or dual channel AAC / PCM file) representing left and right channels. The storage device 604 may be a flash memory device.

[0057] Figure 7 is a block diagram illustrating an exemplary software architecture 706 that may be used in conjunction with the various hardware architectures described herein. Figure 7 is a non-limiting example of a software architecture, and it is understood that many other architectures may be implemented to facilitate the functionality described herein. Software architecture 706 may be implemented in a system such as Figure 8 8, including processor 804, memory 814, and I / O components 818. Representative hardware layers 752 are shown and may represent, for example, Figure 8 The machine 800 of the embodiment of the present invention. The representative hardware layer 752 includes a processing unit 754 having associated executable instructions 704. The executable instructions 704 represent executable instructions of the software architecture 706, including implementations of the methods, components, etc. described herein. The hardware layer 752 also includes a memory or storage module memory / storage device 756, which also has executable instructions 704. The hardware layer 752 may also include other hardware 758.

[0058] As used herein, the term "component" may refer to a device, physical entity, or logic having boundaries defined by function or subroutine calls, branch points, application program interfaces (APIs), or other techniques of partitioning or modularization that provide specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processes. A component can be a packaged functional hardware unit designed for use with other components, and is part of a program that typically performs a specific function of related functions.

[0059] A component may constitute a software component (e.g., code embodied on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that is capable of performing certain operations and may be configured or arranged in some physical manner. In various example embodiments, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as a hardware component that operates to perform certain operations as described herein. A hardware component may also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include dedicated circuitry or logic that is permanently configured to perform certain operations.

[0060] A hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). A hardware component may also include a programmable logic or circuit that is temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general purpose processor or other programmable processor. After being configured by the software, the hardware component becomes a specific machine (or a specific component of a machine) that is uniquely customized to perform the configured functions and is no longer a general purpose processor. It should be understood that the decision to mechanically implement a hardware component in a dedicated and permanently configured circuit or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations.

[0061] A processor may be or include any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and generates corresponding output signals for operating a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may further be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0062] Thus, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, i.e., an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed), that operates in a certain manner or performs certain operations described herein. Considering embodiments in which the hardware components are temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each of the hardware components at any one time. For example, in the case where the hardware components include a general-purpose processor that is configured by software to be a special-purpose processor, the general-purpose processor can be configured as a correspondingly different special-purpose processor (e.g., including different hardware components) at different times. The software configures a specific one or more processors accordingly, for example, to constitute a specific hardware component at one time and to constitute a different hardware component at a different time. The hardware components can provide information to other hardware components and receive information from them. Thus, the described hardware components can be considered to be communicatively coupled. In the case where multiple hardware components are present at the same time, communication can be achieved by signal transmission between two hardware components or more hardware components (e.g., through appropriate circuits and buses). In embodiments where multiple hardware components are configured or instantiated at different times, communication between such hardware components may be achieved, for example, through the storage and retrieval of information in memory structures accessible to the multiple hardware components.

[0063] For example, a hardware component can perform an operation and store the output of the operation in a storage device coupled to it in communication. Then, another hardware component can access the storage device later to obtain and process the stored output. The hardware component can also initiate communication with an input or output device and can operate on a resource (e.g., a collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform related operations. Whether temporarily or permanently configured, the processor can constitute a processor-implemented component that is used to perform one or more operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the method described herein can be implemented at least in part by a processor, wherein a specific processor or multiple processors are examples of hardware. For example, at least some of the operations of the method can be performed by one or more processors or processor-implemented components.

[0064] In addition, one or more processors may also operate to support the performance of related operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as an example of a machine including a processor), which may be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application program interfaces (APIs)). The performance of certain operations in the operations may be distributed among processors, not only residing in a single machine, but also deployed on multiple machines. In some example embodiments, the processor or a component implemented by the processor may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, the processor or a component implemented by the processor may be distributed over multiple geographic locations.

[0065] exist Figure 7 In the exemplary architecture of FIG. 706 , the software architecture 706 can be conceptualized as a stack of layers, each of which provides specific functionality. For example, the software architecture 706 may include layers such as an operating system 702, a library 720, an application 716, and a presentation layer 714. In operation, an application 716 or other component within a layer may call an application program interface (API) API call 708 through the software stack and receive a message 712 in response to the API call 708. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile or dedicated operating systems may not provide a framework / middleware 718, while other operating systems may provide such a layer. Other software architectures may include additional layers or different layers.

[0066] The operating system 702 can manage hardware resources and provide public services. The operating system 702 may include, for example, a kernel 722, services 724, and drivers 726. The kernel 722 may act as an abstraction layer between the hardware and other software layers. For example, the kernel 722 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 724 may provide other public services to other software layers. Drivers 726 are responsible for controlling the hardware or interfacing with the underlying hardware. For example, depending on the hardware configuration, drivers 726 include display drivers, camera drivers, drives, flash drives, serial communications drivers (e.g., Universal Serial Bus (USB) drivers), drivers, audio drivers, power management drivers, and more.

[0067] The library 720 provides a common infrastructure used by the application 916 or other components or layers. The library 720 provides functions that allow other software components to perform tasks in a more convenient manner than directly interfacing with the underlying operating system 702 functions (e.g., kernel 722, service 724, or driver 726). The library 720 may include a system library 744 (e.g., C standard library), which may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 720 may include an API library 746, such as a media library (e.g., a library for supporting the presentation and operation of various media formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG), a graphics library (e.g., an OpenGL framework that can be used to present two-dimensional and three-dimensional graphics content on a display), a database library (e.g., SQLite that can provide various relational database functions), a web library (e.g., WebKit that can provide web browsing functions), etc. The library 720 may also include various other libraries 748 that provide many other APIs to the application 716 and other software components / modules.

[0068] Framework / middleware 718 (also sometimes referred to as middleware) provides a higher level common infrastructure that can be used by applications 716 or other software components / modules. For example, framework / middleware 718 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. Framework / middleware 718 can provide a wide range of other APIs that can be used by applications 716 or other software components / modules, some of which can be specific to a particular operating system 702 or platform.

[0069] Applications 716 include built-in applications 738 or third-party applications 940. Examples of representative built-in applications 738 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 740 may include applications developed using a software development kit (SDK) by entities other than the vendor of a particular platform, and may be mobile software running on a mobile operating system. Third-party applications 740 may call API calls 708 provided by a mobile operating system (such as operating system 702) to facilitate the functionality described herein.

[0070] Applications 716 may create user interfaces to interact with users of the system using built-in operating system functionality (e.g., kernel 722, services 724, or drivers 726), libraries 720, and framework / middleware 718. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer, such as presentation layer 714. In these systems, the application / component "logic" may be separated from aspects of the application / component that interact with the user.

[0071] Figure 8 800 is a block diagram illustrating components (also referred to herein as “modules”) of a machine 800 according to some exemplary embodiments, the machine 800 being capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and executing any one or more of the methods discussed herein. Specifically, Figure 8A graphical representation of a machine 800 in the example form of a computer system is shown within which instructions 810 (e.g., software, programs, applications, applet programs, application software, or other executable code) for causing the machine 800 to perform any one or more of the methods discussed herein may be executed. Thus, the instructions 810 may be used to implement the modules or components described herein. The instructions 810 convert a general purpose, unprogrammed machine 800 into a specific machine 800 that is programmed to perform the functions described and illustrated in the manner described. In alternative embodiments, the machine 800 operates as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 800 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 800 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing the instructions 810 specifying actions to be taken by the machine 800. In addition, while only a single machine 800 is shown, the term "machine" should also be construed to include a collection of machines that individually or jointly execute the instructions 1010 to implement any one or more of the methodologies discussed herein.

[0072] The machine 800 may include a processor 804, a memory / storage device 806, and an I / O component 818, which may be configured to communicate with each other, such as via a bus 802. The memory / storage device 806 may include a memory 814, such as a main memory or other memory storage device, and a storage unit 816, both of which are accessible by the processor 804, such as via the bus 802. The storage unit 816 and the memory 814 store instructions 810 that embody any one or more of the methods or functions described herein. The instructions 810 may also reside, in whole or in part, within the memory 814, within the storage unit 816, within at least one of the processors 804 (e.g., within a cache memory of the processor), or within any suitable combination thereof during execution thereof by the machine 800. Thus, the memory 814, the storage unit 816, and the memory of the processor 804 are examples of machine-readable media.

[0073] As used herein, the terms "machine-readable medium", "computer-readable medium", etc. may refer to any component, device, or other tangible medium that is capable of temporarily or permanently storing instructions and data. Examples of such media may include, but are not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache, other types of storage (e.g., erasable programmable read-only memory (EEPROM)), or any suitable combination thereof. The term "machine-readable medium" should be deemed to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) that can store instructions. The term "machine-readable medium" may also be considered to include any medium or combination of multiple media that can store instructions (e.g., code) executed by a machine, such that the instructions, when executed by one or more processors of the machine, cause the machine to perform any one or more of the methods described herein. Therefore, "machine-readable medium" may refer to a single storage device or device, as well as a "cloud-based" storage system or storage network that includes multiple storage devices or devices. The term "machine-readable medium" does not include the signal itself.

[0074] The I / O components 818 may include a variety of components to provide a user interface for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 818 included in the user interface of a particular machine 800 will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It should be understood that the I / O components 818 may include Figure 8Many other components not shown in the figure. The I / O components 818 are grouped according to function only to simplify the following discussion, and the grouping is by no means restrictive. In various exemplary embodiments, the I / O components 818 may include output components 826 and input components 828. The output components 826 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistor mechanisms), other signal generators, etc. The input components 828 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), touch input components (e.g., physical buttons, touch screens that provide the location or force of touch or touch gestures, or other touch input components), audio input components (e.g., microphones), etc. Input components 828 may also include one or more image capture devices, such as a digital camera for generating digital images or videos.

[0075] In further exemplary embodiments, the I / O components 818 may include a biometric component 830, a motion component 834, an environmental component 836, or a positioning component 838, among a number of other components. One or more of these components (or portions thereof) may be collectively referred to herein as a "sensor component" or "sensor" for collecting various data related to the machine 800, the environment of the machine 800, the user of the machine 800, or a combination thereof.

[0076] For example, the biometric component 830 may include a component that detects expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measures biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifies a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component 834 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a velocity sensor component (e.g., a speedometer), a rotation sensor component (e.g., a gyroscope), etc. The environmental component 836 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor for detecting dangerous gas concentrations or measuring pollutants in the atmosphere for safety), or other components that may provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 838 may include a position sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer, which can detect air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), etc. For example, the position sensor component can provide location information associated with system 800, such as the GPS coordinates of system 800 or information about where system 1000 is currently located (e.g., the name of a restaurant or other business).

[0077] Various technologies may be used to implement communications. I / O components 818 may include a communication component 840 operable to couple machine 800 to network 832 or device 820 via coupler 822 and coupler 824, respectively. For example, communication component 840 may include a network interface component or other suitable device to interface with network 832. In further examples, communication component 840 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Components (e.g. Low Power), Components and other communication components that provide communication via other modes. Device 820 can be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via a universal serial bus (USB)).

[0078] In addition, the communication component 840 can detect the identifier or include a component operable to detect the identifier. For example, the communication component 840 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product codes (UPC) bar codes, multi-dimensional bar codes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Supercode, UCC RSS-2D bar codes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). In addition, various information can be derived via the communication component 840, such as location via Internet Protocol (IP) geographic location, location via Internet Protocol (IP), location via Internet Protocol (IP), location via Internet Protocol (IP), location via Internet Protocol (IP), location via Internet Protocol (IP), location via Internet Protocol (IP), location via Internet Protocol (IP), location via Internet Protocol (IP), location via Internet Protocol (IP), location via Internet Protocol (IP), location via Internet Protocol (IP), location via Internet Protocol (IP), location via Internet Protocol (IP), location via Internet Protocol (IP), location via Internet Protocol (IP), location via Internet Protocol (IP), location via Internet Protocol (IP), location via Internet Protocol (IP), location, etc. The location of signal triangulation, the location of an NFC beacon signal via detection that can indicate a specific location, etc.

[0079] Fig. 9 is a high-level functional block diagram of an example head mounted device 100 communicatively coupling a mobile device 800 and a server system 998 via various networks.

[0080] The apparatus 100 includes a camera, such as at least one of a visible light camera 950, an infrared emitter 951, and an infrared camera 952. The camera may include a Figure 1 and 2 The camera module includes lenses 104_1 and 104_2.

[0081] The client device 800 may be able to connect to the apparatus 100 using both a low power wireless connection 925 and a high speed wireless connection 937. The client device 800 connects to a server system 998 and a network 995. The network 995 may include any combination of wired and wireless connections.

[0082] The device 100 further includes two image displays of the optical assembly 980A-B. The two image displays 980A-980B include one display associated with the left lateral side of the device 100 and one display associated with the right lateral side. The device 100 also includes an image display driver 942, an image processor 912, low power circuits 920, and high speed circuits 930. The image displays of the optical assembly 980A-B are used to present images and videos to a user of the device 100, including images that may include a graphical user interface.

[0083] The image display driver 942 commands and controls the image display of the optical components 980A-B. The image display driver 942 may transmit the image data directly to the image display of the optical components 980A-B for presentation, or may have to convert the image data into a signal or data format suitable for transmission to the image display device. For example, the image data may be video data formatted according to a compression format such as H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, ​​etc., and still image data may be formatted according to a compression format such as Portable Network Graphics Group (PNG), Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF), or Exchangeable Image File Format (Exif).

[0084] As described above, the apparatus 100 includes a frame 103 and rods (or temples) extending from lateral sides of the frame 103. The apparatus 100 further includes a user input device 991 (e.g., a touch sensor or button) that comprises an input surface on the apparatus 100. The user input device 991 (e.g., a touch sensor or button) will receive input selections from a user to manipulate a graphical user interface of a presented image.

[0085] Fig. 9 The components for the device 100 shown in the figure are located on one or more circuit boards, such as PCBs or flexible PCBs, in the frame or temples. Alternatively or additionally, the depicted components may be located in the block, frame, hinge, or center beam of the device 100. The left and right visible light cameras 950 may include digital camera elements, such as complementary metal oxide semiconductor (CMOS) image sensors, charge coupled devices, lenses 104_1, 104_2, or any other corresponding visible light or light capturing elements that can be used to capture data (including images of scenes with unknown objects).

[0086] The apparatus 100 includes a memory 934 storing instructions to perform a subset or all of the functions described herein for generating binaural audio content. The memory 934 may also include the storage device 604 . Figure 4 The exemplary process shown in the flowchart of may be implemented in instructions stored in memory 934 .

[0087] like Fig. 9As shown in , the high-speed circuit 930 includes a high-speed processor 932, a memory 934, and a high-speed wireless circuit 936. In this example, the image display driver 942 is coupled to the high-speed circuit 930 and is operated by the high-speed processor 932 to drive the left and right image displays of the optical components 980A-B. The high-speed processor 932 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system required by the device 100. The high-speed processor 932 includes the processing resources required to manage high-speed data transmission on a high-speed wireless connection 937 to a wireless local area network (WLAN) using a high-speed wireless circuit 936. In some examples, the high-speed processor 932 executes an operating system, such as a LINUX operating system or other such operating system of the device 100, and the operating system is stored in the memory 934 for execution. In addition to any other duties, the high-speed processor 932 that executes the software architecture of the device 100 is used to manage data transmission with the high-speed wireless circuit 936. In some examples, the high-speed wireless circuit 936 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, other high speed communication standards may be implemented by high speed wireless circuitry 936 .

[0088] The low power wireless circuit 924 and the high speed wireless circuit 936 of the device 100 may include a short range transceiver (Bluetooth TM ) and a wireless wide area network, local area network, or wide area network transceiver (e.g., cellular or WiFi). Client device 800 (including a transceiver that communicates via low power wireless connection 925 and high speed wireless connection 937) can be implemented using details of the architecture of apparatus 100, as can other elements of network 995.

[0089] The memory 934 includes any storage device capable of storing various data and applications, including camera data generated by the left and right visible light cameras 950, the infrared camera 952, and the image processor 912, and images generated by the image display driver 942 for display on the image display of the optical assembly 980A-B. Although the memory 934 is shown as being integrated with the high-speed circuit 930, in other examples, the memory 934 can be a separate, separate element of the device 100. In some such examples, electrical wiring can provide a connection from the image processor 912 or the low-power processor 922 to the memory 934 through a chip including the high-speed processor 932. In other examples, the high-speed processor 932 can manage the addressing of the memory 934 so that whenever a read or write operation involving the memory 934 is required, the low-power processor 922 will start the high-speed processor 932.

[0090] like Fig. 9As shown in , the processor 932 of the apparatus 100 may be coupled to a camera (a visible light camera 950 ; an infrared transmitter 951 , or an infrared camera 952 ), an image display driver 942 , a user input device 991 (eg, a touch sensor or button) and a memory 934 .

[0091] The apparatus 100 is connected to a host computer. For example, the apparatus 100 is paired with a client device 800 via a high-speed wireless connection 937 or connected to a server system 998 via a network 995. The server system 998 can be one or more computing devices as part of a service or network computing system, for example, including a processor, a memory, and a network communication interface to communicate with the client device 800 and the apparatus 100 through the network 995.

[0092] The client device 800 includes a processor and a network communication interface coupled to the processor. The network communication interface allows communication via the network 925 or 937. The client device 800 may further store at least a portion of the instructions for generating binaural audio content in a memory of the client device 800 to implement the functions described herein.

[0093] The output components of the device 100 include visual components, such as a display, such as a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide. The image display of the optical component is driven by the image display driver 942. The output components of the device 100 further include acoustic components (e.g., speakers), tactile components (e.g., vibration motors), other signal generators, etc. The input components of the device 100, the client device 800, and the server system 998 (such as user input devices 991) may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), touch input components (e.g., physical buttons, touch screens, providing the location and force of touch or touch gestures, or other touch input components), audio input components (e.g., microphones), etc.

[0094] The apparatus 100 may optionally include additional peripheral elements. Such peripheral elements may include biometric sensors, additional sensors, or display elements integrated with the apparatus 100. For example, the peripheral elements may include any I / O component, including an output component, a motion component, a position component, or any other such component described herein.

[0095] For example, the biometric component includes a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biological signals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The positioning component includes a position sensor component for generating position coordinates (e.g., a global positioning system (GPS) receiver component), a WiFi or Bluetooth receiver for generating positioning system coordinates, etc. TM transceiver, altitude sensor assembly (e.g., an altimeter or barometer that detects air pressure, from which altitude can be derived), orientation sensor assembly (e.g., a magnetometer), etc. Such positioning system coordinates may also be received from the client device 800 via low power wireless circuitry 924 or high speed wireless circuitry 936 over wireless connections 925 and 937.

[0096] Where phrases similar to “at least one of A, B, or C,” “at least one of A, B and C,” “one or more A, B, or C,” or “one or more of A, B, and C” are used, it is intended that the phrase be interpreted to mean that A can exist alone in an embodiment, B can exist alone in an embodiment, C can exist alone in an embodiment, or any combination of elements A, B, and C can exist in a single embodiment; for example, A and B, A and C, B and C, or A and B and C.

[0097] Changes and modifications may be made to the disclosed embodiments without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the following claims.

Claims

1. A head mounted device, comprising: frame; a first rod coupled to the first side of the frame and to a first microphone housing, the first microphone housings respectively enclosing a first front microphone and a first rear microphone generating an acoustic signal, The first microphone housing includes a first front port facing downward and a first rear port facing the back of the head of a user of the head mounted device, a second rod coupled to a second side of the frame and coupled to a second microphone housing, the second microphone housing respectively enclosing a second front microphone and a second rear microphone generating an acoustic signal, the second microphone housing comprising a second front port facing downward and a second rear port facing the back of the head of a user of the head mounted device; as well as Binaural audio processor comprising Beamformer, with receiving acoustic signals from the first front microphone, the first rear microphone, the second front microphone, and the second rear microphone, and generate a first beamformer signal based on acoustic signals from the first front microphone and the first rear microphone, and a second beamformer signal based on acoustic signals from the second front microphone and the second rear microphone, and A storage device is used to store the first beamformer signal and the second beamformer signal as a dual-channel file.

2. The head mounted device according to claim 1, wherein: The head mounted device is a pair of glasses, wherein the first side of the frame is opposite to the second side of the frame.

3. The head mounted device according to claim 2, wherein: When a user wears the pair of glasses, the first front port and the second front port face downward toward the user's feet, while the first rear port and the second rear port face the back of the user's head.

4. The head mounted device according to claim 3, wherein: The frame carries a pair of lenses and is coupled to a camera, wherein the camera lens of the camera faces forward.

5. The head mounted device according to claim 1, wherein: The beamformer is a fixed beamformer.

6. The head mounted device according to claim 5, wherein: The fixed beamformer includes a sub-cardioid or cardioid fixed beam pattern.

7. The head mounted device according to claim 1, wherein: The first front microphone and the first rear microphone form a first-order differential microphone array, and the second front microphone and the second rear microphone form a first-order differential microphone array.

8. The head mounted device according to claim 1, wherein: The storage device is a flash memory device.

9. The head mounted device according to claim 1, wherein: The binaural audio processor is a silicon on a chip (SoC).

10. The head mounted device according to claim 1, further comprising: a first audio codec that decodes acoustic signals from the first front microphone and the first rear microphone to generate a first decoded acoustic signal; A second audio codec decodes the acoustic signal from the second front microphone and the second rear microphone to generate a second decoded acoustic signal.

11. The head mounted device according to claim 10, further comprising: A time division multiplexer (TDM) processes the first decoded acoustic signal and the second decoded acoustic signal and generates a TDM processed signal.

12. The head mounted device according to claim 11, wherein: The acoustic signals generated by the first and second front microphones and the first and second rear microphones are pulse density modulated (PDM) signals.

13. The head mounted device according to claim 12, wherein: The first decoded acoustic signal and the second decoded acoustic signal are pulse code modulation (PCM) signals.

14. The head mounted device according to claim 13, wherein: The binaural audio processor further comprises: A noise suppressor suppresses noise from the first beamformer signal and the second beamformer signal and generates a first noise suppressed signal and a second noise suppressed signal.

15. The head mounted device according to claim 14, wherein: The binaural audio processor further comprises: A speech enhancer that enhances speech from the first noise-suppressed signal and the second noise-suppressed signal to generate a first clean signal and a second clean signal, wherein the storage device stores the first clean signal and the second clean signal.

16. A pair of spectacles comprising: a frame that carries a pair of lenses; a camera coupled to the frame, wherein a camera lens of the camera faces in a forward direction; a first rod coupled to the first side of the frame and to a first microphone housing, the first microphone housings respectively enclosing a first front microphone and a first rear microphone generating an acoustic signal, The first microphone housing includes a first front port facing in a downward direction and a first rear port facing in a direction of the back of a head of a user of the glasses, a second rod coupled to a second side of the frame and coupled to a second microphone housing, the second microphone housing respectively enclosing a second front microphone and a second rear microphone generating an acoustic signal, the second microphone housing comprising a second front port facing the downward direction and a second rear port facing in the direction of the back of the head of a user of the eyeglasses; and Binaural audio processor comprising The beamformer generates a first beamformer signal based on acoustic signals from the first front microphone and the first rear microphone, and a second beamformer signal based on acoustic signals from the second front microphone and the second rear microphone, and A storage device is used to store the first beamformer signal and the second beamformer signal as a dual channel file.

17. A pair of glasses according to claim 16, wherein: The beamformer is a fixed beamformer, wherein the fixed beamformer comprises a sub-cardioid or cardioid fixed beam pattern.

18. The pair of glasses according to claim 16, further comprising: a first audio codec that decodes acoustic signals from the first front microphone and the first rear microphone to generate a first decoded acoustic signal; a second audio codec that decodes the acoustic signal from the second front microphone and the second rear microphone to generate a second decoded acoustic signal; as well as A time division multiplexer (TDM) processes the first decoded acoustic signal and the second decoded acoustic signal and generates a TDM processed signal.

19. A pair of glasses according to claim 16, wherein: The binaural audio processor further comprises: a noise suppressor that suppresses noise from the first beamformer signal and the second beamformer signal and generates a first noise-suppressed signal and a second noise-suppressed signal; and A speech enhancer that enhances speech from the first noise-suppressed signal and the second noise-suppressed signal to generate a first clean signal and a second clean signal, wherein the storage device stores the first clean signal and the second clean signal.

20. A method for generating binaural audio content using a head mounted device, comprising: generating acoustic signals by a first front microphone and a first rear microphone, respectively, wherein the first front microphone and the first rear microphone are enclosed in a first microphone housing coupled to a first pole of the head mounted device; generating acoustic signals by a second front microphone and a second rear microphone, respectively, wherein the second front microphone and the second rear microphone are enclosed in a second microphone housing, the second microphone housing is coupled to a second pole of the head mounted device, wherein the first pole and the second pole are coupled to opposite sides of a frame of the head mounted device; decoding, by a first audio codec, the acoustic signals from the first front microphone and the first rear microphone to generate a first decoded acoustic signal, wherein the first decoded acoustic signal is a pulse code modulation (PCM) signal based on the acoustic signals from the first front microphone and the first rear microphone; decoding, by a second audio codec, the acoustic signals from the second front microphone and the second rear microphone to generate a second decoded acoustic signal, wherein the second decoded acoustic signal is a PCM signal based on the acoustic signals from the second front microphone and the second rear microphone; processing the first decoded acoustic signal and the second decoded acoustic signal by a time division multiplexer (TDM) to generate a TDM processed signal, wherein the TDM processed signal includes the first decoded acoustic signal and the second decoded acoustic signal; The TDM-processed signal is beamformed by a beamformer, wherein the TDM-processed signal is beamformed including: beamforming the first decoded signal to generate a first beamformer signal, and beamforming the second decoded signal to generate a second beamformer signal; and The first beamformer signal and the second beamformer signal are stored as a dual-channel file by a storage device.

Images