Apparatus worn on head of user, method for selecting sound signal received on device worn on head of user

Abstract

Systems and methods for extracting a desired sound signal from a device worn on the head of a user are described. An apparatus includes a head-mounted device and an array having at least three microphones. The at least three microphones are disposed along a plurality of at least two non-parallel axes. The selection logic is configured to identify a selected axis from the plurality of non-parallel axes and to identify two microphones from an array forming the selected axis. The beamformer is configured to receive as an input signals from the two microphones and to output a primary microphone channel and a reference microphone channel.

Description

[0001] This patent application is a continuation-in-part of U.S. non-provisional patent application serial number 14 / 886,077, filed on October 18, 2015, and entitled "Head-mounted acoustic system and method with noise-canceling microphone geometry device", which is a continuation-in-part of U.S. non-provisional patent application serial number 14 / 207,163, filed on March 12, 2014, and entitled "Dual-stage noise reduction structure for desired signal extraction", which U.S. non-provisional patent application serial number 14 / 207,163 claims priority to U.S. provisional patent application serial number 61 / 780,108, filed on March 13, 2013, and entitled "Noise-canceling microphone device" and priority to U.S. provisional patent application serial number 61 / 941,088, filed on February 18, 2014, and entitled "System and method for processing sound signals".

[0002] Patent application serial number 14 / 886,077, also filed on February 14, 2014, is entitled "Glasses with Microphone Array" and is a part-continuation of U.S. non-provisional patent application serial number 14 / 180,994, which claims priority to U.S. provisional patent application serial number 61 / 780,108 filed on March 13, 2013 and priority to U.S. provisional patent application serial number 61 / 839,211 filed on June 25, 2013 and priority to U.S. provisional patent application serial number 61 / 839,227 filed on June 25, 2013 and priority to U.S. provisional patent application serial number 61 / 912,844 filed on December 6, 2013.

[0003] This patent application also claims priority to U.S. provisional patent application serial number 62 / 801,618, filed on February 5, 2019, entitled "Microphone Configuration for Eyeglass Device Apparatus and Methods."

[0004] The documents of U.S. Provisional Patent Application Serial No. 62 / 801,618 are hereby incorporated by reference. The documents of U.S. Provisional Patent Application Serial No. 61 / 780,108 are hereby incorporated by reference. The documents of U.S. Provisional Patent Application Serial No. 61 / 941,088 are hereby incorporated by reference. The documents of U.S. Non-Provisional Patent Application Serial No. 14 / 207,163 are hereby incorporated by reference. The documents of U.S. Non-Provisional Patent Application Serial No. 14 / 180,994 are hereby incorporated by reference. The documents of U.S. Provisional Patent Application Serial No. 61 / 839,211 are hereby incorporated by reference. The documents of U.S. Provisional Patent Application Serial No. 61 / 839,227 are hereby incorporated by reference. The documents of U.S. Provisional Patent Application Serial No. 61 / 912,844 are hereby incorporated by reference. Technical Field

[0005] The present invention relates generally to wearable devices that detect and process sound signal data, and more particularly to reducing noise in head-worn acoustic systems and assisting a user's hearing. Background Art

[0006] Acoustic systems use sound sensors such as microphones to receive audio signals. Typically, these systems are used in real-world environments that provide both desired sound signals and undesired sound signals (also known as noise) to the receiving microphone. Such receiving microphones are part of various systems such as mobile phones, handheld microphones, hearing aids, etc. These systems typically perform speech recognition processing on the received sound signals. The simultaneous reception of the desired sound signal and the undesired sound signal can have a negative impact on the quality of the desired sound signal. The degradation of the quality of the desired sound signal can make the desired sound signal output to the user difficult to understand by the user. Using the degraded desired sound signal in algorithms such as speech recognition (SR) or automatic speech recognition (ASR) can lead to increased error rates, which can make the reconstructed speech difficult to understand. Any of these can cause problems.

[0007] Handheld systems require the user's fingers to grasp and / or operate the device in which the handheld system is implemented. Take a mobile phone as an example. Occupying the user's fingers prevents the user from performing mission-critical functions. This can cause some problems.

[0008] Unwanted sound signals (noise) may come from a variety of sources that are not the source of the desired sound signal. Therefore, the source of the undesired sound signal is statistically uncorrelated with the desired sound signal. These sources can be non-stationary sources or from stationary sources. The stationary situation applies to time and space where the amplitude, frequency and direction of the sound signal do not change significantly. For example, in an automotive environment, the engine noise at a constant speed is stationary, as is road noise or wind noise. In the case of a non-stationary signal, the noise amplitude, frequency distribution and direction of the sound signal change with time and / or space. Non-stationary noise originates from, for example, car audio, from fleeting noise such as collisions, opening or closing doors, background conversations such as chatting in the back seat of a vehicle, and so on. Stationary and non-stationary sources of undesired sound signals exist in office environments, concert halls, football stadiums, airplane cabins, and anywhere users use acoustic systems (e.g., mobile phones, tablets, etc. are equipped with microphones, headphones, earbud microphones, etc.) Sometimes, the environment in which the acoustic system is used is reverberant, causing noise to reverberate in the environment, using multiple paths of undesired sound signals to reach the microphone position. Any source of noise, i.e., non-stationary or stationary unwanted sound signals, can increase the error rate of speech recognition algorithms such as SR or ASR, or can easily make it difficult for the system to output an understandable wanted sound signal to the user. All of these can cause problems.

[0009] Various noise cancellation methods have been employed to reduce noise from both stable and non-stationary sources. Existing noise cancellation methods work better in environments where the intensity of the noise is less than the intensity of the desired sound signal, such as in relatively low-noise environments. Spectral subtraction is used to reduce noise in speech recognition algorithms and various acoustic systems, such as hearing aids.

[0010] When used in automatic speech recognition (ASR) applications, systems that employ spectral subtraction can produce unacceptable error rates when the strength of the undesired sound signal becomes large. This can cause problems.

[0011] In addition, existing algorithms, such as spectral subtraction, use nonlinear processing of the spectrum of the sound signal. Nonlinear processing of the sound signal causes the output to be disproportionately related to the input. Speech recognition (SR) algorithms are developed using speech signals recorded in a quiet and noise-free environment. Therefore, when nonlinear distortion is introduced into the speech processing process through nonlinear signal processing, the speech recognition algorithm (developed in a quiet and noise-free environment) will produce high error rates. Nonlinear processing of the sound signal causes nonlinear distortion of the desired sound signal, which disrupts the feature extraction required for speech recognition, which leads to high error rates. All of this can cause problems.

[0012] Various methods have been used to try to suppress or remove unwanted sound signals from acoustic systems, such as in speech recognition (SR) or automatic speech recognition (ASR) applications. One method is called a voice activity detector (VAD). VAD attempts to detect when the desired speech is present and when the unwanted speech is present. Thus, only the desired speech is accepted and the unwanted speech is treated as noise by not transmitting it. Traditional voice activity detection is only applicable to a single sound source or stationary noise (unwanted sound signal) with a small intensity relative to the intensity of the desired sound signal. Therefore, traditional voice activity detection makes VAD perform poorly in noisy environments. In addition, when the desired sound signal and the unwanted sound signal arrive at the receiving microphone at the same time, using VAD to remove the unwanted sound signal is not effective. This brings some problems.

[0013] An acoustic system with a single microphone used in a noisy environment has a problem in that the desired sound signal and the undesired sound signal are received simultaneously on a single channel. The undesired sound signal can make the desired sound signal difficult to understand for a human user or for an algorithm designed to use the received speech, such as a speech recognition (SR) or automatic speech recognition (ASR) algorithm. This can cause problems. Multiple channels have been used to solve the problem of receiving the desired and undesired sound signals simultaneously. Thus, on one channel, the desired sound signal and the undesired sound signal are received, while on another channel, a sound signal that also contains the desired sound signal and the undesired sound signal is received. Over time, the sensitivity of the individual channels may drift, causing the undesired sound signal to become unbalanced between the channels. The drifting channel sensitivity can result in the undesired sound signal not being accurately removed from the desired sound signal. Nonlinear distortion of the original desired sound signal may be caused by processing the sound signal obtained from a channel whose sensitivity has drifted over time. This can cause problems. Summary of the invention

[0014] To solve at least one of the above problems, an embodiment of the present invention provides a device worn on a user's head, comprising:

[0015] Head mounted device;

[0016] An array having at least three microphones, the at least three microphones being arranged along a plurality of at least two non-parallel axes;

[0017] selection logic configured to identify a selected axis from the plurality of non-parallel axes and to identify two microphones from the array forming the selected axis; and

[0018] A beamformer is configured to receive signals from the two microphones as input and output a main microphone channel and a reference microphone channel.

[0019] An embodiment of the present invention further provides a device worn on a user's head, comprising:

[0020] A head mounted device, the head mounted device further comprising:

[0021] an array of three microphones connected to the head-mounted device, wherein a first microphone and a second microphone of the array define a first axis; and the second microphone and a third microphone define a second axis; and

[0022] a speaker connected to the head mounted device and configured to provide a signal audible to the user; and

[0023] a selection logic that selects an active direction from among the first axis and the second axis when the active direction is the first axis;

[0024] a. the outputs from the first microphone and the second microphone are processed for transmission from the head mounted device;

[0025] When the active direction is the second axis;

[0026] b. The outputs from the second microphone and the third microphone will be processed to be used as input to the speaker.

[0027] An embodiment of the present invention further provides a method for selecting a sound signal received on a device worn on a user's head, characterized by comprising:

[0028] comparing sound signals from an array having at least three microphones, wherein positions of the at least three microphones define three non-parallel axes;

[0029] selecting a first microphone pair from the array, wherein the first microphone pair includes a first microphone and a second microphone;

[0030] forming a primary microphone signal from the first microphone pair; and

[0031] A reference microphone signal is formed from the first microphone pair; wherein the primary microphone signal and the reference microphone signal are input into a noise cancellation module to reduce noise from the primary microphone signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The best understanding of the present invention may be obtained by referring to the following description and accompanying drawings which illustrate various embodiments of the present invention. The present invention is described by way of example in various embodiments and is not limited by the accompanying drawings in which like reference numerals indicate similar elements.

[0033] Figure 1 is a schematic diagram of a general process of configuring a microphone on a head mounted device according to an embodiment of the present invention.

[0034] Figure 2 is a schematic diagram of the geometry of a microphone layout according to an embodiment of the present invention.

[0035] Figure 3A is a schematic diagram of a general microphone layout with a main microphone located at a first position according to an embodiment of the present invention.

[0036] Figure 3B According to an embodiment of the present invention Figure 3A Schematic diagram of the measured values ​​of the signal-to-noise ratio difference of the main microphone.

[0037] Figure 3C According to an embodiment of the present invention Figure 3B Schematic diagram of the signal-to-noise ratio difference of the data shown in with increasing microphone acoustic separation distance.

[0038] Figure 4A is a schematic diagram of a general microphone layout in which a main microphone is located at a second position according to an embodiment of the present invention.

[0039] Figure 4B According to an embodiment of the present invention Figure 4A Schematic diagram of the measured values ​​of the signal-to-noise ratio difference of the main microphone.

[0040] Figure 4C According to an embodiment of the present invention Figure 4B Schematic diagram of the signal-to-noise ratio difference of the data shown in with increasing microphone acoustic separation distance.

[0041] Figure 5A is a schematic diagram of a general microphone layout in which a main microphone is located at a third position according to an embodiment of the present invention.

[0042] Figure 5B According to an embodiment of the present invention Figure 5A Schematic diagram of the measured values ​​of the signal-to-noise ratio difference of the main microphone.

[0043] Figure 5C According to an embodiment of the present invention Figure 5B Schematic diagram of the signal-to-noise ratio difference of the data shown in with increasing microphone acoustic separation distance.

[0044] Figure 6 is a schematic diagram of a microphone directivity pattern according to an embodiment of the present invention.

[0045] Figure 7 is a schematic diagram of a misaligned reference microphone response axis according to an embodiment of the present invention.

[0046] Figure 8 is a schematic diagram of glasses with two embedded microphones according to an embodiment of the present invention.

[0047] Fig. 9 is a schematic diagram of glasses with three embedded microphones according to an embodiment of the present invention.

[0048] Fig.10 FIG. 4 is a schematic diagram of another embodiment of the present invention using four omnidirectional microphones used at four acoustic ports instead of two bidirectional microphones.

[0049] Fig.11 is a schematic diagram of glasses of the present invention employing two omnidirectional microphones placed at diagonally opposite corners of the lens opening defined by the front frame of the glasses.

[0050] Fig.12 is a schematic diagram of another embodiment of the present invention employing four omnidirectional microphones placed along the top and bottom of the eyeglass frame.

[0051] Fig.13 yes Fig.12 Schematic diagram of the microphone assembly of the glasses.

[0052] Fig.14 is a schematic diagram of another embodiment of the present invention, wherein microphones are placed at the temple portion of the glasses facing inward, and at the lower center corner of the front frame of the glasses facing downward.

[0053] Fig.15 is a schematic diagram of glasses with a built-in acoustic noise cancellation system according to an embodiment of the present invention.

[0054] Fig.16 According to an embodiment of the present invention, Fig.15 Schematic diagram of the location of the main microphone in a head-mounted device.

[0055] Fig.17 is a schematic diagram of goggles with a built-in acoustic noise cancellation system according to an embodiment of the present invention.

[0056] Fig.18 is a schematic diagram of a sun visor with a built-in acoustic noise cancellation system according to an embodiment of the present invention.

[0057] Fig.19 is a schematic diagram of a helmet with a built-in acoustic noise cancellation system according to an embodiment of the present invention.

[0058] Fig. 20 It is a schematic diagram of a process of extracting a required sound signal according to an embodiment of the present invention.

[0059] Fig.21is a schematic diagram of a system architecture according to an embodiment of the present invention.

[0060] Fig. 22 is a schematic diagram of filter control according to an embodiment of the present invention.

[0061] Fig.23 is another schematic diagram of a system architecture according to an embodiment of the present invention.

[0062] Fig.24A is another schematic diagram of a system architecture including automatic balancing according to an embodiment of the present invention.

[0063] Fig. 24B is a schematic diagram of a noise reduction process according to an embodiment of the present invention.

[0064] Fig.25A is a schematic diagram of beam forming according to an embodiment of the present invention.

[0065] Fig.25B is another schematic diagram of beamforming according to an embodiment of the present invention.

[0066] Fig.25C is a schematic diagram of beamforming using a shared acoustic element according to an embodiment of the present invention.

[0067] Fig.26 is a schematic diagram of multi-channel adaptive filtering according to an embodiment of the present invention.

[0068] Fig. 27 is a schematic diagram of single-channel filtering according to an embodiment of the present invention.

[0069] Fig.28A is a schematic diagram of required voice activity detection according to an embodiment of the present invention.

[0070] Fig.28B is a schematic diagram of a normalized speech threshold comparator according to an embodiment of the present invention.

[0071] Fig.28C is a schematic diagram of desired voice activity detection using multiple reference channels according to an embodiment of the present invention.

[0072] Fig.28D is a schematic diagram of a process for using compression according to an embodiment of the present invention.

[0073] Fig.28E is a schematic diagram of different functions for providing compression according to an embodiment of the present invention.

[0074] Fig.29A is a schematic diagram of an automatic balancing architecture according to an embodiment of the present invention.

[0075] Fig.29Bis a schematic diagram of automatic balancing according to an embodiment of the present invention.

[0076] Fig.29C is a schematic diagram of filtering according to an embodiment of the present invention.

[0077] Fig.30 is a schematic diagram of a process for automatic balancing according to an embodiment of the present invention.

[0078] Fig.31 is a schematic diagram of a speech signal processing system according to an embodiment of the present invention.

[0079] Fig.32A is a perspective view of a microphone configuration on a head mounted device according to an embodiment of the present invention.

[0080] Fig.32B According to an embodiment of the present invention Fig.32A A top view of the microphone configuration on the corresponding head-mounted device.

[0081] Fig.32C According to an embodiment of the present invention Fig.32A Bottom view of the microphone configuration on the corresponding head-mounted device.

[0082] Fig.32D is a perspective view of another set of microphone arrangements on a head mounted device according to an embodiment of the present invention.

[0083] Fig.32E According to an embodiment of the present invention Fig.32D Bottom view of the microphone layout on the corresponding head-mounted device.

[0084] Fig.33 According to an embodiment of the present invention, Fig.32A -D Schematic diagram of the head-mounted device relative to different sound sources.

[0085] Fig.34 is a schematic diagram of processing a sound signal from a microphone array configured with a head-mounted device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0086] In the following detailed description of multiple embodiments of the present invention, reference is made to multiple drawings, in which similar reference numerals indicate similar elements, and specific embodiments in which the present invention can be practiced are shown in the drawings by way of illustration. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention. In other examples, well-known circuits, structures, and techniques are not shown in detail to avoid blurring the understanding of this specification. Therefore, the following detailed description will not be understood as limiting, and the scope of the present invention is limited only by the appended claims.

[0087] The present invention describes an apparatus and method for detecting and processing acoustic signals including desired acoustic signals and undesired acoustic signals within a head mounted device. In one or more embodiments, a noise cancellation structure combines multi-channel noise cancellation and single-channel noise cancellation to extract desired acoustic signals from undesired acoustic signals. In one or more embodiments, multi-channel acoustic signal compression is used for desired voice activity detection. In one or more embodiments, the channels are automatically balanced. In one or more embodiments, the system automatically selects a subset of microphones from a possible microphone array for acoustic signal extraction. In one or more embodiments, a hearing aid is provided to the user to facilitate hearing sounds from the local environment.

[0088] Figure 1 According to an embodiment of the present invention, a general process of configuring a microphone on a head mounted device is shown by 100. Figure 1 The process begins at block 102. At block 104, a "primary" or "main" microphone channel is created on the headset using one or more microphones. The primary microphone is mounted to optimize reception of the desired sound signal, thereby enhancing a first signal-to-noise ratio associated with the primary microphone, denoted as SNR M At block 106, a reference microphone channel is created on the headset using one or more microphones. The reference microphone is located on the headset to provide a lower signal-to-noise ratio for detection of the desired sound signal from the user, thereby generating a signal-to-noise ratio denoted as SNR R Therefore, at block 108, a signal-to-noise ratio difference is achieved by the layout geometry of the microphones on the head-mounted device, resulting in a first signal-to-noise ratio SNR M Greater than the second signal-to-noise ratio SNR R .

[0089] At block 110, a signal-to-noise ratio difference is achieved by beamforming by creating different response patterns (directivity patterns) for the main microphone channel and the reference microphone channel. The generation of a signal-to-noise ratio difference using different directivity patterns is described in more detail below with reference to the accompanying drawings.

[0090] In various embodiments, at block 112 , the signal-to-noise ratio difference is achieved through a combination of one or more of microphone layout geometry, beamforming, and using different directivity patterns for the main channel and the reference channel. Flow ends at block 114 .

[0091] Figure 2 A microphone layout geometry is generally shown at 200 in accordance with an embodiment of the present invention. Figure 2200, a desired sound signal 204 emanates from a desired sound signal source, a user's mouth shown at 202. Source 202 provides the desired sound signal 204 to microphones mounted on a head mounted device. A first microphone 206 is located at a distance from source 202 shown by d1 208. A second microphone 210 is located at a distance from source 202 shown by d2 212. System 200 is also affected by undesired sound signals shown at 218.

[0092] With respect to source 202, first microphone 206 and second microphone 210 are at different acoustic distances from source 202, as indicated by ΔL at 214. This difference in acoustic distances ΔL 214 is given by equation 216. As used in the description of the embodiments, distances d1 and d2 represent paths that sound waves take to reach microphones 206 and 210, respectively. Thus, these distances may be linear or curved, depending on the specific locations of the microphones on the head mounted device and the sound frequencies of interest thereto. For clarity of illustration, these paths and corresponding distances are represented by straight lines, but this is not meant to be limiting.

[0093] The undesired sound signal 218 is typically generated by various sources located at greater distances than distances d1 and d2. For example, construction noise, car noise, airplane noise, etc., all of which originate at distances that are typically several orders of magnitude greater than d1 and d2. Therefore, the undesired sound signal 218 is substantially correlated with each other at the microphone locations 206 and 210, or at least received at a fairly uniform level at each location. The difference ΔL in acoustic distance relative to the first microphone 208, 214 reduces the amplitude of the desired sound signal 204 received at the second microphone 210 due to various mechanisms. One such mechanism is, for example, spherical diffusion, which causes the desired sound signal to increase with 1 / r 2 . Where r is the distance (e.g., 208 or 212) between the source (e.g., 202) and the receiving location (e.g., 206 or 210). The reduction of the desired sound signal at the second microphone location 210 reduces the signal-to-noise ratio at 210 relative to 206 because the noise amplitude is essentially the same at each location, but the signal amplitude is reduced at 210 relative to the amplitude received at 206. Another mechanism related to path length is the difference in acoustic impedance along one path and along another path, resulting in a curved acoustic path rather than a straight path. Overall, these mechanisms combine to reduce the amplitude of the desired sound signal received at the reference microphone location relative to the main microphone location. Therefore, the layout geometry is used to provide a signal-to-noise ratio difference between the two microphone locations used by the noise cancellation system, which will be further described below, to reduce undesired sound signals from the main microphone channel.

[0094] The microphone layout geometry allows for various configurations for the placement of the primary microphone and the reference microphone. FIG. 3A to FIG. 5C A general microphone placement method is described and presented that allows microphones to be placed in different locations on a headgear device.

[0095] Figure 3A According to an embodiment of the present invention, a general microphone layout is shown generally at 300 with a primary microphone located at a first position. Figure 3A , a head mounted device 302 is shown. As used in the detailed description of this embodiment, the head mounted device may be any device configured to be worn on the user's head, such as but not limited to glasses, goggles, helmets, sun hats, headbands, etc. FIG. 3A to FIG. 5C In the discussion given, it can be appreciated that the discussion is equally applicable to any head mounted device, such as Figures 8 to 19 Those shown in the drawings, as well as those head-mounted devices not specifically shown in the drawings herein. Therefore, embodiments of the present invention may be applicable to head-mounted devices that have not yet been named or invented.

[0096] Return to reference Figure 3A In one embodiment, the head mounted device has a frame 302, a lens 308, and a lens 310, the frame 302 having attached temples 304 and temples 306. In various embodiments, the head mounted device 302 is a pair of glasses worn on the user's head. A plurality of microphones are located on the head mounted device 302, such as microphone 1, microphone 2, microphone 3, microphone 4, microphone 5, microphone 6, microphone 7, microphone 8, and optional microphones 9 and 10. In various embodiments, the head mounted device including the frame 302 / template 304 and 306 as shown can be sized to include an electronic device 318 for signal processing as further described below. The electronic device 318 provides electrical connections to the microphones mounted on the head mounted device 302.

[0097] The head mounted device 302 has an internal volume defined by its structure, within which the electronics 318 may be mounted. Alternatively, the electronics 318 may be mounted external to the structure. In one or more embodiments, an entry panel is provided to access the electronics 318. In other embodiments, no entry door is explicitly provided, but the electronics 318 may be contained within the volume of the head mounted device 302. In this case, the electronics 318 may be embedded prior to assembling the head mounted device, wherein one or more components interlock together to form a housing that captures the electronics 318 therein. In other embodiments, the head mounted device is molded around the electronics 318, thereby encapsulating the electronics 318 within the volume of the head mounted device 302. In various non-limiting embodiments, the electronics 318 include an adaptive noise cancellation unit, a single channel noise cancellation unit, a filter controller, a power supply, a desired voice activity detector, a filter, and the like. Other components of the electronics 318 are described in the following figures.

[0098] The head mounted device 302 may include a switch (not shown) for powering the head mounted device 302 on or off.

[0099] The head mounted device 302 may contain a data processing system within its volume for processing sound signals received by a microphone associated therewith. The data processing system may include Fig.31 One or more elements of the system shown are further described below. FIG. 3A to FIG. 5C The illustrations do not limit the embodiments of the present invention.

[0100] Figure 3A The headgear device of the present invention illustrates that the microphone can be placed at any position on the device. The ten positions selected for illustration in the accompanying drawings are selected only to illustrate the general principles of layout geometry and do not constitute a limitation of the embodiments of the present invention. Therefore, the microphone can be used in different positions other than the positions shown, and different microphones can be used in different positions. For the purpose of illustration and without any limitation, the combination FIG. 3A to FIG. 5CThe measurements shown are made using omnidirectional microphones. In other embodiments, directional microphones are used. In an exemplary configuration for signal-to-noise ratio measurements, each microphone is mounted in a housing, each housing having a port that opens to the environment. Arrow 1b shows the direction of the port associated with microphone 1. Arrow 2b shows the direction of the port associated with microphone 2. Arrow 3b shows the direction of the port associated with microphone 3. Arrow 4b shows the direction of the port associated with microphone 4. Arrow 5b shows the direction of the port associated with microphone 5. Arrow 6b shows the direction of the port associated with microphone 6. Arrow 7b shows the direction of the port associated with microphone 7. Arrow 8b shows the direction of the port associated with microphone 8.

[0101] The user's mouth is shown at 312 and is similar to Figure 2 3. The source of the desired sound signal shown by 202 in FIG. 3. The acoustic path length (referred to herein as the acoustic distance or distance) from the user's mouth 312 to each microphone is shown by the arrows from the user's mouth 312 to the corresponding microphone position. For example, d1 represents the acoustic distance from the user's mouth 312 to microphone 1. d2 represents the acoustic distance from the user's mouth 312 to microphone 2. d3 represents the acoustic distance from the user's mouth 312 to microphone 3. d4 represents the acoustic distance from the user's mouth 312 to microphone 4. d5 represents the acoustic distance from the user's mouth 312 to microphone 5. d6 represents the acoustic distance from the user's mouth 312 to microphone 6. d7 represents the acoustic distance from the user's mouth 312 to microphone 7. d8 represents the acoustic distance from the user's mouth 312 to microphone 8. Similarly, optional microphones 9 and 10 also have acoustic distances; however, to maintain clarity in the figure, they are not labeled as such.

[0102] exist Figure 3A , microphones 1, 2, 3, and 6 and the user's mouth 312 generally lie on an XZ plane (see coordinate system 316), and the corresponding acoustic distances d1, d2, d3, and d6 are generally represented by straight lines. The paths to microphones 4, 5, 7, and 8, i.e., d4, d5, d7, and d8, are represented as curved paths, reflecting the fact that the user's head is opaque to the sound field. Therefore, in this case, there is some curvature in the acoustic path. In general, the acoustic path between the source of the desired sound signal and the microphone on the head-mounted device can be linear or curved. As long as the difference in path length between the main microphone and the reference microphone is large enough, the necessary signal-to-noise ratio difference required by the noise cancellation system will be obtained to achieve an acceptable level of noise cancellation.

[0103] In order to carry out Figure 3B and Figure 3CFor the measurements shown, an acoustic test facility was used to measure the difference in signal-to-noise ratio between the primary microphone and the reference microphone locations. The test facility included a mannequin with built-in speakers to simulate a user wearing a headset. A speaker located at the user's mouth was used to produce the desired sound signal. The mannequin was placed in an anechoic chamber of the acoustic test facility. Background noise was generated in the anechoic chamber with the speaker array. A pink noise spectrum was used during the measurements; however, other weightings in frequency could be used for the background noise field. The spectral amplitude level of the background noise was set to 75 dB / upa / Hz during these measurements. The headset was placed on the mannequin. During the tests, the microphones were located on the headset. Figure 3A The microphone for the main or primary channel is selected as Microphone 1 for the first measurement sequence as follows Figure 3B and Figure 3C shown.

[0104] The desired audio signal consists of the word "Camera". This word is delivered through a speaker in the mannequin. The received signal corresponding to the word "Camera" at microphone 1 is processed by the noise cancellation system (as described in the figure below), gated in time, and averaged to produce the amplitude of the "signal" corresponding to microphone 1. The corresponding signal corresponding to the word "Camera" is measured at each of the other microphones at positions 2, 3, 4, 5, 6, 7 and 8 in turn. Similarly, at each microphone position, the background noise spectrum level is measured. From these measurements, the signal-to-noise ratio is calculated at each microphone position, and then the signal-to-noise ratio difference of the microphone pair is calculated, as shown in the figure below.

[0105] Figure 3B According to an embodiment of the present invention, generally shown at 320 Figure 3A A measure of the difference in signal-to-noise ratio of the primary microphone in . Figure 3B and Figure 3A , at 314, microphone 1 is used as the primary or main microphone. Reference microphones are then placed using various positions, such as microphone 2, microphone 3, microphone 6, microphone 4, microphone 5, microphone 7, and microphone 8. Figure 3B , column 322 represents the microphone pairs used for a set of measurements. Column 324 represents the approximate difference in acoustic path length between a given pair of microphones in column 322. The approximate acoustic path length difference ≈ΔL is given by Figure 2344. The signal-to-noise ratio difference is given in Equation 216 in . Column 326 lists the dimensionless numbers ranging from 1 to 7 for seven different microphone pairs used for signal-to-noise ratio measurement. Column 328 lists the signal-to-noise ratio difference for a given microphone pair listed in column 322. Each row 330, 332, 334, 336, 338, 340 and 342 lists a different microphone pair where, while the main microphone 314 remains unchanged as microphone 1, the reference microphone is changed. Note that, as shown in equation 344, the approximate differences in acoustic path lengths for the various microphone pairs can be arranged in increasing order. According to equation 344, the microphone pairs have been arranged in rows 330 to 342 in a manner that increases the approximate acoustic path length difference 324. The signal-to-noise ratio difference changes from 5.55 dB for microphone 2 used as a reference microphone to 10.48 dB when microphone 8 is used as a reference microphone.

[0106] Figure 3C According to an embodiment of the present invention, a method for Figure 3B The data shown shows the difference in signal-to-noise ratio with increasing microphone acoustic separation distance. Figure 3C , the signal-to-noise ratio difference is plotted on the vertical axis indicated by 352, and the signal-to-noise ratio difference from column 326 ( Figure 3B ). Note that, as described above, the dimensionless X value represents the approximate acoustic path length difference ~ΔL. X-axis 354 does not correspond exactly to ~ΔL, but it is related to ~ΔL because the data is arranged and plotted in a manner that increases the approximate acoustic path length difference ~ΔL. This ordering of the data helps illustrate the above combination Figure 2 The characteristic of the signal-to-noise ratio difference described by , namely, that the signal-to-noise ratio difference will increase as the difference in acoustic path length between the primary microphone and the reference microphone increases. This behavior is identified by observing that the signal-to-noise ratio difference increases as a function of ≈ΔL using curve 356, which plots the data from column 328 as a function of the data from column 326 ( Figure 3B ).

[0107] Figure 4A A general microphone layout with a primary microphone at a second location is shown generally at 420 in accordance with an embodiment of the present invention. Figure 4A , the second position for the primary microphone 414 is the position of microphone 2. The above test is repeated using microphone 2 as the primary microphone and the reference microphone position can alternatively be the position of microphone 6, microphone 3, microphone 4, microphone 5, microphone 7 and microphone 8. Figure 4B and Figure 4C These data are explained.

[0108] Figure 4B According to an embodiment of the present invention, Figure 4A A measure of the difference in signal-to-noise ratio of the primary microphone in . Figure 4B and Figure 4A , microphone 2 is used as the main or primary microphone 414. Various positions are then used to place reference microphones, such as microphone 6, microphone 3, microphone 4, microphone 5, microphone 7, and microphone 8. Figure 4B , column 422 represents the microphone pairs used for a set of measurements. Column 424 represents the approximate difference in acoustic path length between a given pair of microphones in column 422. The approximate acoustic path length difference ~ΔL is given by Figure 2 4 is given by equation 216 in . Column 426 lists the dimensionless numbers ranging from 1 to 6 for six different microphone pairs used for signal-to-noise ratio measurements. Column 428 lists the signal-to-noise ratio difference for a given microphone pair listed in column 422. Each row 430, 432, 434, 436, 438 and 440 lists a different microphone pair where the reference microphone is changed while the main microphone 414 remains unchanged as microphone 2. Note that the approximate differences in acoustic path lengths for the various microphone pairs can be arranged in increasing order as shown in equation 442. According to equation 442, the microphone pairs have been arranged in rows 430 to 440 in a manner that increases the approximate acoustic path length difference 424. The signal-to-noise ratio difference changes from 1.2 dB for microphone 6 used as a reference microphone to 5.2 dB when microphone 8 is used as a reference microphone.

[0109] Figure 4C According to an embodiment of the present invention, Figure 4B The data shown in Figure 2 show the difference in signal-to-noise ratio with increasing microphone acoustic separation distance. Figure 4C , the signal-to-noise ratio difference is plotted on the vertical axis indicated by 452, and the signal-to-noise ratio difference from column 426 ( Figure 4B ). Note that, as described above, the dimensionless X value represents the approximate acoustic path length difference ~ΔL. X-axis 454 does not correspond exactly to ~ΔL, but it is related to ~ΔL because the data is arranged and plotted in a manner that increases the approximate acoustic path length difference ~ΔL. This ordering of the data helps illustrate the above combination Figure 2 The characteristic of the signal-to-noise ratio difference described by 454 is that the signal-to-noise ratio difference will increase as the difference in acoustic path length between the primary microphone and the reference microphone increases. This behavior is identified by observing that the signal-to-noise ratio difference increases as a function of ≈ΔL using curve 456, which plots the data from column 428 as a function of the data from column 426 ( Figure 4B ).

[0110] Figure 5A According to an embodiment of the present invention, a general microphone layout is shown in which the main microphone is located at the third position. Figure 5A, the third position for the main microphone 514 is the position where microphone 3 is located. The above test is repeated using microphone 3 as the main microphone and the reference microphone position can alternatively be the position where microphone 6, microphone 4, microphone 5, microphone 7, and microphone 8 are located. Figure 5B and Figure 5C These data are explained.

[0111] Figure 5B According to an embodiment of the present invention, Figure 5A A measure of the difference in signal-to-noise ratio of the primary microphone in . Figure 5B and Figure 5A , microphone 3 is used as the main or primary microphone 514. Various positions are then used to place reference microphones, such as the positions of microphone 6, microphone 4, microphone 5, microphone 7, and microphone 8. Figure 5B , column 522 represents the microphone pairs used for a set of measurements. Column 524 represents the approximate difference in acoustic path length between a given pair of microphones in column 522. The approximate acoustic path length difference ~ΔL is given by Figure 2 540. The signal-to-noise ratio difference is given in Equation 216 in Column 526. Column 526 lists the dimensionless numbers ranging from 1 to 5 for five different microphone pairs used for the signal-to-noise ratio measurement. Column 528 lists the signal-to-noise ratio difference for a given microphone pair listed in column 522. Each row 530, 532, 534, 536 and 538 lists a different microphone pair where the reference microphone is changed while the main microphone 514 remains unchanged as microphone 3. Note that the approximate differences in the acoustic path lengths of the various microphone pairs can be arranged in increasing order as shown in equation 540. According to equation 540, the microphone pairs have been arranged in rows 530-538 in a manner that increases the approximate acoustic path length difference 524. The signal-to-noise ratio difference changes from 0 dB for microphone 6 used as a reference microphone to 5.16 dB when microphone 7 is used as a reference microphone.

[0112] Figure 5C According to an embodiment of the present invention, Figure 5B The data shown in Figure 2 show the difference in signal-to-noise ratio with increasing microphone acoustic separation distance. Figure 5C , the signal-to-noise ratio difference is plotted on the vertical axis indicated by 552, and the signal-to-noise ratio difference from column 526 ( Figure 5B ). Note that, as described above, the dimensionless X value represents the approximate acoustic path length difference ~ΔL. X-axis 554 does not correspond exactly to ~ΔL, but it is related to ~ΔL because the data is arranged and plotted in a manner that increases the approximate acoustic path length difference ~ΔL. This ordering of the data helps illustrate the above combination Figure 2The characteristic of the signal-to-noise ratio difference described by , namely, that the signal-to-noise ratio difference will increase as the difference in acoustic path length between the primary microphone and the reference microphone increases. This behavior is identified by observing that the signal-to-noise ratio difference increases as a function of ~ΔL using curve 556, which plots the data from column 528 as a function of the data from column 526 ( Figure 5B ).

[0113] Please note that the specific positions of the microphones in the views presented in the above figures are chosen for illustration purposes only. These positions do not constitute limitations on the embodiments of the present invention. Other positions of the microphones on the head mounted device are used in other embodiments.

[0114] Therefore, if the above combination Figure 1 Block 108 and Figures 2 to 5C As described, in various embodiments, the microphone layout geometry is used to produce an acoustic path length difference between the two microphones and a corresponding signal-to-noise ratio difference between the primary microphone and the reference microphone. The signal-to-noise ratio difference can also be achieved by using different directivity patterns for the primary microphone and the reference microphone. In some embodiments, beamforming is used to create different directivity patterns for the primary channel and the reference channel. For example, in Figure 5A In the example, the values ​​of the acoustic path lengths d3 and d6 are too similar, so this choice of position for the main microphone and the reference microphone does not produce a sufficient signal-to-noise ratio difference (in Figure 5B In this case, changes in the microphone directivity pattern (one or both microphones) and / or beamforming may be used to produce the desired signal-to-noise ratio difference between the main channel and the reference channel.

[0115] The directional microphone may be used to reduce the reception of desired sound signals and / or increase the reception of undesired sound signals, thereby reducing the signal-to-noise ratio of the second microphone (reference microphone), which may result in an increase in the difference in signal-to-noise ratio between the primary microphone and the reference microphone. Figure 3A A method of using a second microphone (not shown) and the following Figure 6 and Figure 7An example of the technique taught in . In some embodiments, the second microphone can be substantially co-located with microphone 1. In other embodiments, the distance between the second microphone and the source 312 is equal to the distance between the first microphone and the source 312. In some embodiments, the second microphone is a directional microphone whose main response axis is substantially perpendicular to (or equivalently misaligned with) the acoustic path d1. Therefore, there is a direction in the direction of the desired sound signal d1 in which the second microphone has a zero response or a small response to the desired sound signal from 312. As a result, the signal-to-noise ratio of the second microphone is reduced, and the calculated signal-to-noise ratio difference between the first microphone and the second microphone is increased. Note that the two microphones can be placed anywhere on the head mounted device 302, including the same position as described above. In other embodiments, one or more microphone elements are used as inputs to the beamformer, resulting in the main channel and the reference channel having different directivity patterns and producing a signal-to-noise ratio difference between them.

[0116] Figure 6 A microphone directivity pattern is generally shown at 600 in accordance with an embodiment of the present invention. Figure 6 , the omnidirectional microphone directivity pattern is shown by circle 602 having a constant radius 604 representing uniform sensitivity as a function of angle α (α) represented by 608 measured from reference 606 .

[0117] Figure 6 An example of a directional microphone having a cardioid directivity pattern 622 is shown at 620 in FIG. 6 , wherein the cardioid directivity pattern 622 has a peak sensitivity axis indicated by 624 and a null indicated by 626. The cardioid directivity pattern can be formed by two omnidirectional microphones, or by one omnidirectional microphone and a suitable mounting structure for the microphone.

[0118] Figure 6 An example of a directional microphone having a bi-directional pattern 642 / 644 is shown at 640 in FIG. 1 , wherein a first lobe 642 of the bi-directional pattern has a first axis of peak sensitivity indicated by 648 and a second lobe 644 has a second axis of peak sensitivity indicated by 646. A first null exists in direction 650 and a second null exists in direction 652.

[0119] Figure 6 An example of a directional microphone having a supercardioid directivity pattern is shown in 660 of FIG, wherein the supercardioid directivity pattern 664 / 665 has a peak sensitivity axis indicated by direction 662 , a short sensitivity axis indicated by direction 666 , and nulls indicated by directions 668 and 670 .

[0120] Figure 7A misaligned reference microphone response axis is generally shown at 700 in accordance with an embodiment of the present invention. Figure 7 , microphone 702 is denoted by 702. Microphone 702 is a directional microphone having a primary response axis 706 and a null in its directivity pattern denoted by 704. The incident sound field is indicated as arriving from direction 708. In various embodiments, microphone 702 is, for example, Figure 6 A directional microphone 702, properly placed on a head-mounted device, when used as a reference microphone, reduces the signal-to-noise ratio by responding to unwanted sound signals from direction 710 while limiting the response to the desired sound signals from direction 708. As described above, the response of the directional microphone 702 will increase the signal-to-noise ratio difference.

[0121] Therefore, within the teachings of the embodiments presented herein, one or more primary microphones and one or more reference microphones are placed at multiple locations on the head mounted device to obtain a suitable signal-to-noise ratio difference between the primary microphones and the reference microphones. Such a signal-to-noise ratio difference enables the extraction of the desired sound signal from the acoustic signal containing the desired sound signal and the undesired sound signal, as described below in conjunction with the following figures. The microphones can be placed at different locations on the head mounted device, including placing the primary microphone and the reference microphone together at a common location on the head mounted device.

[0122] In some embodiments, according to block 112 ( Figure 1 ), which combines the technique of microphone layout geometry with different directivity patterns obtained at the microphone level or through beamforming to produce a signal-to-noise ratio difference between the main channel and the reference channel.

[0123] In various embodiments, the head mounted device is an eyeglass device as described below in conjunction with the following figures. Figure 8800 is an example diagram of an eyeglass device in an embodiment of the present invention. As shown in the figure, the eyeglass device 800 includes eyeglasses 802 with embedded microphones. Eyeglasses 802 have two microphones 804 and 806. The first microphone 804 is arranged in the middle of the frame of the eyeglasses 802. The second microphone 806 is arranged on the side of the frame of the eyeglasses 802. Microphones 804 and 806 can be bidirectional or unidirectional pressure gradient microphone elements. In one or more embodiments, each microphone 804 and 806 is a microphone assembly in a rubber sleeve. The rubber sleeve provides an acoustic port with an acoustic duct on the front and back of the microphone. The two microphones 804 and 806 and their respective sheaths can be the same. Microphones 804 and 806 can be hermetically sealed (e.g., fully sealed). The acoustic duct is filled with windshield material. The port is sealed with a woven fabric layer. The lower and upper acoustic ports are sealed with a waterproof membrane. The microphone can be built into the structure of the eyeglass frame. Each microphone has a top hole and a bottom hole that serve as an acoustic port. In one embodiment, microphones 804 and 806, which may be pressure gradient microphone elements, may each be replaced by two omnidirectional microphones.

[0124] Fig. 9 FIG. 1 is a schematic diagram of another example of an embodiment of the present invention. Fig. 9 As shown, the eyewear apparatus 900 includes eyewear 952 having three embedded microphones. Fig. 9 The glasses in 952 are similar to Figure 8 , but instead employing three microphones instead of two. Fig. 9 The glasses 952 in the embodiment have a first microphone 954 disposed in the middle of the glasses 952, a second microphone 956 disposed on the left side of the glasses 952, and a third microphone 958 disposed on the right side of the glasses 952. These three microphones can be used in the above-mentioned three-microphone embodiment.

[0125] Fig.10 is a schematic diagram of glasses 1000 according to an embodiment of the present invention, wherein for example Figure 8The two bidirectional microphones shown in are replaced by four omnidirectional microphones 1002, 1004, 1006, 1008 and electronic beam steering. Replacing the two bidirectional microphones with four omnidirectional microphones provides more flexibility and manufacturability for designers of eyeglass frames. In an exemplary embodiment with four omnidirectional microphones, the four omnidirectional microphones can be located anywhere on the eyeglass frame, and preferably, the microphone pairs are arranged vertically around the lenses. In this embodiment, omnidirectional microphones 1002 and 1004 are main microphones for detecting the main sound to be separated from interference; microphones 1004 and 1008 are reference microphones for detecting background noise to be separated from the main sound. The microphone array can be an omnidirectional microphone, wherein the omnidirectional microphone can be any combination of the following microphones: an electret condenser microphone, an analog microelectromechanical system (MEMS) microphone, or a digital MEMS microphone.

[0126] Another exemplary embodiment of the present invention is as follows Fig.11 As shown, a glasses device with a noise cancellation microphone array includes a frame 1100, a microphone array connected to the frame, the microphone array includes at least one first microphone 1102 and a second microphone 1104, the first microphone is connected to the frame near the temple area, the temple area can be roughly located between the top corner of the lens opening and the support arm, and provides a first channel output, the second microphone is connected to the frame near the inner lower corner of the lens opening, and provides a second channel output. The second microphone is located on the diagonal of the lens opening 1106, but it can be set anywhere along the inner frame of the lens, such as the lower corner, upper corner or inner frame edge. Further, the second microphone can be set along the inner edge of the lens on the left or right side of the nose bridge.

[0127] In yet another embodiment of the present invention, the microphone array may be connected to the frame using at least one flexible printed circuit board (PCB) strip, such as Fig.12 In this embodiment, the eyeglass device 1200 of the present invention includes: an upper flexible PCB strip 1202 including a first microphone 1204 and a fourth microphone 1206 , and a lower flexible PCB strip 1208 including a second microphone 1210 and a third microphone 1212 .

[0128] In a further exemplary embodiment, the frame may also include an array of vents corresponding to the microphone array. The microphone array may be a bottom-port or top-port micro-electromechanical system (MEMS) microphone. Fig.13 As shown, Fig.13 yes Fig.12The microphone assembly of the glasses, MEMS microphone assembly 1300 includes a MEMS microphone 1302 attached to a flexible printed circuit board (PCB) 1304. A gasket 1306 separates the flexible PCB 1304 from a device housing 1308. A vent 1310 is defined by the flexible PCB 1304, the gasket 1306, and the device housing 1308, and the vent 1310 is an audio channel that guides sound waves to the MEMS microphone 1302. The first and fourth MEMS microphones can be connected to the upper flexible PCB strip, the second and third MEMS microphones can be connected to the lower flexible PCB strip, and the MEMS microphone array can be arranged so that the bottom port or the top port receives the sound signal through the corresponding vent.

[0129] Fig.14 Another alternative embodiment of the glasses 1400 is shown in which the microphones 1402, 1404 are placed at the temple area 1406 and the front frame 1408, respectively.

[0130] Fig.15 According to an embodiment of the present invention, glasses with a built-in acoustic noise cancellation system are generally shown at 1500. Fig.15 , the head mounted device 1502 includes one or more microphones for the main channel and one or more microphones for the reference channel. The head mounted device 1502 is configured to function as a wearable computer with an information display 1504. In various embodiments, electronics are included at 1506 and / or 1508. In various embodiments, the electronics may include noise cancellation electronics, which will be described more fully below in conjunction with the accompanying drawings. In other embodiments, the noise cancellation electronics are not in the same location as the head mounted device 1502, but are located external to the head mounted device 1502. In these embodiments, a wireless communication link is provided, such as with protocol, to send the sound signals received from the microphone to an external location for processing by the noise cancellation electronics.

[0131] Fig.16 According to an embodiment of the present invention, generally shown at 1600 is a diagram of Fig.15 The main microphone position in the headset. Fig.16 , the main microphone position is shown by 1602.

[0132] Fig.17 A pair of goggles with a built-in acoustic noise cancellation system is shown generally at 1700 according to an embodiment of the present invention. Fig.17, a head mounted device in the form of goggles 1702 is provided with a primary microphone at position 1704 and a reference microphone at position 1706. In various embodiments, the goggles 1702 include noise cancellation electronics. The noise cancellation electronics are described more fully below in conjunction with the accompanying drawings. In other embodiments, the noise cancellation electronics are not located in the same location as the head mounted device 1702, but are located external to the head mounted device 1702. In these embodiments, a wireless communication link is provided, such as with protocol, protocol to send the sound signals received from the microphone to an external location for processing by the noise cancellation electronics.

[0133] Fig.18 A sun visor with a built-in acoustic noise cancellation system is shown generally at 1800 according to an embodiment of the present invention. Fig.18 , a head mounted device in the form of a sun visor 1802 having a primary microphone 1804 and a reference microphone 1806. In various embodiments, the sun visor 1802 includes noise cancellation electronics. The noise cancellation electronics are described more fully below in conjunction with the accompanying drawings. In other embodiments, the noise cancellation electronics are not located in the same location as the head mounted device 1802, but are located external to the head mounted device 1802. In these embodiments, a wireless communication link is provided, such as with protocol, protocol to send the sound signals received from the microphone to an external location for processing by the noise cancellation electronics.

[0134] Fig.19 A helmet having a built-in acoustic noise cancellation system is shown generally at 1900 according to an embodiment of the present invention. Fig.19 , a head mounted device in the form of a helmet 1902 having a primary microphone 1904 and a reference microphone 1906. In various embodiments, the helmet 1902 includes noise cancellation electronics. The noise cancellation electronics are described more fully below in conjunction with the accompanying drawings. In other embodiments, the noise cancellation electronics are not co-located with the head mounted device 1902, but are located external to the head mounted device 1902. In these embodiments, a wireless communication link is provided, for example with protocol, protocol to send the sound signals received from the microphone to an external location for processing by the noise cancellation electronics.

[0135] Fig. 20 According to an embodiment of the present invention, a process of extracting a desired sound signal is generally shown by 2000. Fig. 20, the process begins at block 2002. At block 2004, a primary sound signal is received from a primary microphone located on the head mounted device. At block 2006, a reference sound signal is received from a reference microphone located on the head mounted device. At block 2008, a normalized primary sound signal is formed. In various embodiments, as described in the following figure, the normalized primary sound signal is formed using one or more reference sound signals. At block 2010, the normalized primary sound signal is used to control noise cancellation using a sound signal processing system included in the head mounted device. The process stops at block 2012.

[0136] Fig.21 A system architecture is generally shown at 2100 according to an embodiment of the present invention. Fig.21 , two channels are input to an adaptive noise cancellation unit 2106. The first channel, referred to herein as the main channel 2102, is synonymously referred to as the "main" or "main" channel in the description of this embodiment. The main channel 2102 contains a desired sound signal and an undesired sound signal. As described in detail in the following figures, the sound signal input on the main channel 2102 is generated by the presence of the desired sound signal and the undesired sound signal on one or more acoustic elements. According to the configuration of one or more microphones for the main channel, the microphone element can output an analog signal. The analog signal is converted into a digital signal using an analog-to-digital converter (AD) (not shown). In addition, the amplifier can be located near the microphone element or the AD converter. The second channel, referred to herein as the reference channel 2104, provides a sound signal also generated by the presence of the desired sound signal and the undesired sound signal. Optionally, the second reference channel 2104b can be input to the adaptive noise cancellation unit 2106. Similar to the main channel, according to the configuration of one or more microphones for the reference channel, the microphone element can output an analog signal. The analog signal is converted into a digital signal using an analog-to-digital converter (AD) (not shown). Furthermore, the amplifier may be located near the microphone element or the AD converter.In some embodiments, the microphone is implemented as a digital microphone.

[0137] In some embodiments, the main channel 2102 has an omnidirectional response and the reference channel 2104 has an omnidirectional response. In some embodiments, the acoustic beam patterns of the acoustic elements of the main channel 2102 and the reference channel 2104 are different. In other embodiments, the beam patterns of the acoustic elements of the main channel 2102 and the reference channel 2104 are the same. However, the desired sound signal received on the main channel 2102 is different from the desired sound signal received on the reference channel 2104. Therefore, the signal-to-noise ratio of the main channel 2102 is different from the signal-to-noise ratio of the reference channel 2104. In general, the signal-to-noise ratio of the reference channel is less than the signal-to-noise ratio of the main channel. In various embodiments, as a non-limiting example, the difference between the main channel signal-to-noise ratio and the reference channel signal-to-noise ratio is about 1 or 2 decibels (dB) or more. In other non-limiting embodiments, the difference between the main channel signal-to-noise ratio and the reference channel signal-to-noise ratio is about 1 decibel (dB) or less. Therefore, the embodiments of the present invention are applicable to high noise environments that may result in a low signal-to-noise ratio relative to the desired sound signal, and low noise environments that may have a higher signal-to-noise ratio. As used in the description of the embodiments, the signal-to-noise ratio means the ratio of the desired sound signal to the undesired sound signal in a channel. Further, the term "main channel signal-to-noise ratio" is used interchangeably with the term "main signal-to-noise ratio". Similarly, the term "reference channel signal-to-noise ratio" is used interchangeably with the term "reference signal-to-noise ratio".

[0138] The main channel 2102, the reference channel 2104, and the optional second reference channel 2104b provide inputs to the adaptive noise cancellation unit 2106. Although the second reference channel is shown in the figure, in various embodiments, more than two reference channels are used. The adaptive noise cancellation unit 2106 filters the undesired sound signals from the main channel 2102, thereby providing a first stage of filtering with multiple channel inputs. In various embodiments, the adaptive noise cancellation unit 2106 utilizes an adaptive finite impulse response (FIR) filter. The environment in the embodiment of the present invention can present a reverberant sound field. Therefore, the adaptive noise cancellation unit 2106 includes a delay for the main channel that is sufficient to approximate the impulse response of the environment in which the system is used. The size of the delay used depends on the specific application of the system design, including whether reverberation must be considered in the design. In some embodiments, for microphone channels that are placed very closely together (where reverberation is not obvious), the size of the delay can be a fraction of a millisecond. It should be noted that at the low end of the range of values ​​that can be used for delay, the acoustic propagation time between channels can represent the minimum delay value. Therefore, in various embodiments, however, the delay value may range from about a fraction of a millisecond to about 500 milliseconds or longer, depending on the application.A further description of the adaptive noise cancellation unit 1106 and components associated therewith is provided below in conjunction with the following figures.

[0139] The output 2107 of the adaptive noise elimination unit 2106 is input to a single channel noise elimination unit 2118. The single channel noise elimination unit 2118 filters the output 2107 and further reduces the unwanted sound signal from the output 2107, thereby providing a second stage of filtering. The filtering of the unwanted sound signal by the single channel noise elimination unit 2118 is mainly a stabilizing effect. The single channel noise elimination unit 2118 includes a linear filter, such as a Wiener filter, a minimum mean square error (MMSE) filter implementation, a linear stationary noise filter, or other Bayesian filtering methods using prior information about the parameters to be estimated. The filter used by the single channel noise elimination unit 2118 will be described more fully below in conjunction with the accompanying drawings.

[0140] The sound signal from the main channel 2102 is input to the filter controller 2112 at 2108. Similarly, the sound signal from the reference channel 2104 is input to the filter controller 2112 at 2110. The optional second reference channel is input to the filter control 2112 at 2108b. The filter control 2112 provides a control signal 2114 to the adaptive noise cancellation unit 2106 and a control signal 2116 to the single channel noise cancellation unit 2118. In various embodiments, the operation of the filter controller 2112 will be described more fully below in conjunction with the accompanying drawings. The output terminal 2120 of the single channel noise cancellation unit 2118 provides a sound signal that mainly contains a majority of the desired sound signal and a reduced amount of undesired sound signals.

[0141] According to various embodiments of the present invention, Fig.21 The system architecture shown can be applied in a variety of different systems for processing sound signals. Some examples of different acoustic systems are, but are not limited to, mobile communications, handheld microphones, hanging microphones, microphone headset hearing aids, hands-free microphone devices, wearable systems embedded in eyeglass frames, near-eye (NTE) headset displays or headset computing devices, generally configured head-mounted devices, such as but not limited to eyeglass goggles, sun hats, headbands, helmets, etc. The environment in which these acoustic systems are used can have multiple sound energy sources incident on the acoustic elements, which provide sound signals for the main channel 2102 and the reference channel 2104. In various embodiments, the desired sound signal is typically a result of the user's own voice (see above Figure 2). In various embodiments, the undesired sound signal is typically the result of a combination of undesired sound energy from multiple sound sources incident on the acoustic elements of the main channel and the reference channel. Therefore, the undesired sound signal is statistically uncorrelated with the desired sound signal. In addition, there is a non-causal relationship between the undesired sound signal in the main channel and the undesired sound signal in the reference channel. In this case, due to the non-causal relationship, and because there is no measurement of a pure noise signal (undesired sound signal) in addition to the signal of interest (the desired sound signal), echo cancellation does not work. In an echo cancellation noise reduction system, the speaker that produces the sound signal provides a measure of the pure noise signal. In the context of the system embodiments described herein, there is no speaker or noise source from which a pure noise signal can be extracted.

[0142] Fig. 22 A filter controller is generally shown at 2112 according to an embodiment of the present invention. Fig. 22 , the sound signal from the main channel 2102 is input into the desired voice activity detection unit 2202 at 2108. The main channel activity detector 2206 monitors the sound signal at 2108 to create a flag associated with the activity on the main channel 2102 ( Fig.21 Optionally, a second reference channel activity detector (not shown) monitors the acoustic signal at 2110b to create a flag associated with activity on the second reference channel. Optionally, the output of the second reference channel activity detector is connected to the disable control logic 2214. The reference channel activity detector 2208 monitors the acoustic signal at 2110 to create a flag associated with activity on the reference channel 2104 ( Fig.21 ). The desired voice activity detection unit 2202 utilizes the sound signal inputs from 2110, 2108 and optionally 2110b to generate a desired voice activity signal 2204. The operation of the desired voice activity detection unit 2202 will be described more fully in the following figures.

[0143] In various embodiments, the inhibit logic unit 2214 receives as input information about the main channel activity at 2210, information about the reference channel activity at 2212, and information about whether a desired sound signal is present at 2204. In various embodiments, the inhibit logic 2214 outputs a filter control signal 2114 / 2116, which is sent, for example, to Fig.21The adaptive noise cancellation unit 2106 and the single channel noise cancellation unit 2118. The implementation and operation of the main channel activity detector 2206, the reference channel activity detector 2208 and the inhibit logic 2214 are more fully described in U.S. Patent No. 7,386,135 entitled "Cardioid Beam with Acoustic Devices, Systems and Methods Based on Desired Nulls", which is incorporated herein by reference.

[0144] In operation, in various embodiments, when the adaptive noise cancellation unit 2106 and the single-channel noise cancellation unit 2118 apply successive filtering stages, Fig.21 systems and Fig. 22 The filter controller provides filtering and removal of undesired sound signals from the main channel 2102. In one or more embodiments, throughout the system, the application of signal processing is a linear application. In linear signal processing, the output is linearly related to the input. Therefore, changing the input value results in a proportional change in the output. The linear application of the signal processing process to the signal maintains the quality and fidelity of the desired sound signal, thereby substantially eliminating or minimizing any nonlinear distortion of the desired sound signal. Maintaining the signal quality of the desired sound signal is useful to the user because accurate reproduction of speech helps promote accurate delivery of information.

[0145] In addition, algorithms for processing speech, such as speech recognition (SR) algorithms or automatic speech recognition (ASR) algorithms, benefit from the accurate presentation of sound signals that are substantially free of nonlinear distortion. Therefore, through embodiments of the present invention, distortion that may be generated due to the application of nonlinear signal processing processes is eliminated. The changes made to the desired sound signal by the linear noise elimination algorithm taught by embodiments of the present invention are transparent to the operation of the SR and ASR algorithms used by the speech recognition engine. Therefore, through the application of embodiments of the present invention, the error rate of the speech recognition engine is greatly reduced.

[0146] Fig.23 Another system architecture diagram is generally shown at 2300 according to an embodiment of the present invention. Fig.23 In the system architecture presented here, at 2302, a first channel provides an acoustic signal from a first microphone (nominaly labeled MIC 1 in the figure). At 2304, a second channel provides an acoustic signal from a second microphone (nominaly labeled MIC 2 in the figure). In various embodiments, one or more microphones may be used to create a signal from the first microphone 2302. In various embodiments, one or more microphones may be used to create a signal from the second microphone 2304. In some embodiments, one or more acoustic elements may be used to create a signal useful for the signal from the first microphone 2302 and the signal from the second microphone 2304 (see the description below). Fig.25C). Therefore, one acoustic element may be shared by 2302 and 2304. In various embodiments, the arrangement of acoustic elements providing signals at 2302 and 2304, the main channel, and the reference channel are described below in conjunction with the following figures.

[0147] The beamformer 2305 receives as input a signal from a first microphone 2302 and a signal from a second microphone 2304 and optionally a signal from a third microphone 2304b (nominaly labeled MIC 3 in the figure). The beamformer 2305 uses the signals 2302, 2304, and optionally 2304b to create a main channel 2308a containing a desired sound signal and an undesired sound signal. The beamformer 2305 also uses the signals 2302, 2304, and optionally 2304b to create one or more reference channels 2310a and optionally 2311a. The reference channels contain the desired sound signal and the undesired sound signal. The signal-to-noise ratio of the main channel, referred to as the "main channel signal-to-noise ratio", is greater than the signal-to-noise ratio of the reference channel, referred to herein as the "reference channel signal-to-noise ratio". The arrangement of the beamformer 2305 and / or acoustic elements for MIC1 and MIC2 provides a main channel signal-to-noise ratio that is greater than the reference channel signal-to-noise ratio.

[0148] The beamformer 2305 is connected to an adaptive noise cancellation unit 2306 and a filter control unit 2312. The main channel signal is output from the beamformer 2305 at 2308a and is input to the adaptive noise cancellation unit 2306. Likewise, the reference channel signal is output from the beamformer 2305 at 2310a and is input to the adaptive noise cancellation unit 2306. The main channel signal is also output from the beamformer 2305 and is input to the filter controller 2312 at 2308b. Likewise, the reference channel signal is output from the beamformer 2305 and is input to the filter controller 2312 at 2310b. Optionally, a second reference channel signal is output at 2311a and is input to the adaptive noise cancellation unit 2306, and an optional second reference channel signal is output at 2311b and is input to the filter controller 2312.

[0149] The filter controller 2312 uses inputs 2308b, 2310b and optionally 2311b to generate channel activity flags and required voice activity detection to provide filter control signals 2314 to the adaptive noise cancellation unit 2306 and filter control signals 2316 to the single channel noise reduction unit 2318.

[0150] At 2307, the adaptive noise cancellation unit 2306 provides multi-channel filtering and filters a first number of undesired sound signals from the main channel 2308a during the first stage of filtering to output a filtered main channel. The single channel noise reduction unit 2318 receives the filtered main channel 2307 as input and provides a second stage of filtering to further reduce the undesired sound signals from 2307. The single channel noise reduction unit 2318 outputs the most desired sound signal at 2320.

[0151] In various embodiments, different types of microphones may be used to provide the sound signals required for the embodiments of the present invention presented herein. Any transducer that converts sound waves into electrical signals is suitable for use with the embodiments of the present invention taught herein. Some non-limiting examples of microphones are, but not limited to, dynamic microphones, condenser microphones, electret condenser microphones (ECM), and micro-electromechanical systems (MEMS) microphones. In other embodiments, condenser microphones (CM) are used. In other embodiments, micromechanical microphones are used. Microphones based on piezoelectric film are used with other embodiments. Piezoelectric elements are made of ceramic materials, plastic materials, or thin films. In other embodiments, micromechanical microphone arrays are used. In other embodiments, silicon or polysilicon micromechanical microphones are used. In some embodiments, bidirectional pressure gradient microphones are used to provide multiple channels. Various microphones or microphone arrays including the system described herein can be mounted on or inside structures such as glasses or headphones.

[0152] Fig.24A Another system architecture diagram including automatic balancing is generally shown at 2400 according to an embodiment of the present invention. Fig.24A In the system architecture presented here, at 2402, the first channel provides an acoustic signal from a first microphone (nominaly labeled MIC 1 in the figure). At 2404, the second channel provides an acoustic signal from a second microphone (nominaly labeled MIC 2 in the figure). In various embodiments, one or more microphones may be used to create a signal from the first microphone 2402. In various embodiments, one or more microphones may be used to create a signal from the second microphone 2404. In some embodiments, as described above in conjunction with the previous figures, one or more acoustic elements may be used to create a signal that becomes part of the signal from the first microphone 2402 and the signal from the second microphone 2404. In various embodiments, the arrangement of acoustic elements, main channels, and reference channels that provide signals at 2302, 2304 are described below in conjunction with the following figures.

[0153] The beamformer 2405 receives as input a signal from a first microphone 2402 and a signal from a second microphone 2404. The beamformer 2405 uses the signals 2402 and 2404 to create a main channel that contains both a desired sound signal and an undesired sound signal. The beamformer 2405 also uses the signals 2402 and 2404 to create a reference channel. Optionally, at 2404b, a third channel provides a sound signal from a third microphone (nominally labeled MIC 3 in the figure). The signal is input to the beamformer 2405. In various embodiments, one or more microphones may be used to create the signal 2404b from the third microphone. The reference channel contains both a desired sound signal and an undesired sound signal. The signal-to-noise ratio of the main channel, referred to as the "main channel signal-to-noise ratio", is greater than the signal-to-noise ratio of the reference channel, referred to herein as the "reference channel signal-to-noise ratio". The beamformer 2405 and / or the arrangement of acoustic elements for MIC1, MIC2, and optionally MIC3 provide a main channel signal-to-noise ratio that is greater than the reference channel signal-to-noise ratio. In some embodiments, a bidirectional pressure gradient microphone element provides signals 2402, 2404, and optionally 2404b.

[0154] The beamformer 2405 is connected to an adaptive noise cancellation unit 2406 and a desired voice activity detector 2412 (filter controller). The main channel signal is output from the beamformer 2405 at 2408a and is input to the adaptive noise cancellation unit 2406. Similarly, the reference channel signal is output from the beamformer 2405 at 2410a and is input to the adaptive noise cancellation unit 2406. The main channel signal is also output from the beamformer 2405 and is input to the desired voice activity detector 2412 at 2408b. Similarly, the reference channel signal is output from the beamformer 2405 and is input to the desired voice activity detector 2412 at 2410b. Optionally, the second reference channel signal is output from the beamformer 2405 at 2409a and input to the adaptive noise cancellation unit 2406 , and the second reference channel signal is output from the beamformer 2405 at 2409b and input to the desired voice activity detector 2412 .

[0155] The desired voice activity detector 2412 uses inputs 2408b, 2410b, and optionally 2409b to generate a filter control signal 2414 for the adaptive noise cancellation unit 2408 and a filter control signal 2416 for the single channel noise reduction unit 2418. At 2407, the adaptive noise cancellation unit 2406 provides multi-channel filtering and filters a first number of undesired sound signals from the main channel 2408a during a first stage of filtering to output a filtered main channel. The single channel noise reduction unit 2418 receives the filtered main channel 2407 as input and provides a second stage of filtering to further reduce the undesired sound signals from 2407. The single channel noise reduction unit 2418 outputs the most desired sound signal at 2420.

[0156] The desired voice activity detector 2412 provides a control signal 2422 to an automatic balancing unit 2424. The automatic balancing unit 2424 is connected to the signal path of the first microphone 2402 at 2426. The automatic balancing unit 2424 is also connected to the signal path of the second microphone 2404 at 2428. Optionally, the automatic balancing unit 2424 is also connected to the signal path of the third microphone 2404b at 2429. The automatic balancing unit 2424 balances the response of the microphones to far-field signals over the life of the system. Maintaining microphone channel balance can improve system performance and maintain a high level of performance by preventing microphone sensitivity drift. The automatic balancing unit will be described more fully below in conjunction with the following figures.

[0157] Fig. 24B The noise reduction process is generally shown at 2450 according to an embodiment of the present invention. Fig. 24B , the process begins at block 2452. At block 2454, a main sound signal is received by the system. The main sound signal can be, for example, in various embodiments, as received by 2102 ( Fig.21 )、2302 / 2308a / 2308b( Fig.23 ), or 2402 / 2408a / 2408b( Fig.24A ) is represented by a signal. At block 2456, a reference sound signal is received by the system. The reference sound signal may be, for example, in various embodiments, as represented by 2104 and optionally 2104b ( Fig.21 ), 2304 / 2310a / 2310b and optional 2304b / 2311a / 2311b( Fig.23 ), or 2404 / 2410a / 2410b and optional 2404b / 2409a / 2409b( Fig.24A ) represented by the signal. In block 2458, adaptive filtering is performed using multiple input channels, for example using adaptive filtering unit 2106 ( Fig.21)、2306( Fig.23 ) and 2406( Fig.24A ) provides such as 2107( Fig.21 )、2307( Fig.23 ) and 2407( Fig.24A ) is a filtered sound signal represented by . At block 2460, the filtered sound signal resulting from the processing of block 2458 is filtered using a single channel unit. The single channel unit may be, for example, in various embodiments, as represented by 2118 ( Fig.21 )、2318( Fig.23 ), or 2418( Fig.24A ) The process stops at block 2462.

[0158] In various embodiments, an adaptive noise cancellation unit, such as 2106 ( Fig.21 )、2306( Fig.23 ) and 2406( Fig.24A ), implemented in an integrated circuit device, which may include: an integrated circuit package including the integrated circuit. In some embodiments, the adaptive noise cancellation unit 2106 or 2306 or 2406 is implemented in a single integrated circuit die. In other embodiments, the adaptive noise cancellation unit 2106 or 2306 or 2406 is implemented in more than one integrated circuit die of an integrated circuit device, which may include: a multi-chip package including the integrated circuit.

[0159] In various embodiments, a single channel noise cancellation unit, such as 2118 ( Fig.21 )、2318( Fig.23 ), and 2418( Fig.24A ), implemented in an integrated circuit device, which may include: an integrated circuit package including the integrated circuit. In some embodiments, the single-channel noise cancellation unit 2118 or 2318 or 2418 is implemented in a single integrated circuit die. In other embodiments, the single-channel noise cancellation unit 2118 or 2318 or 2418 is implemented in more than one integrated circuit die of an integrated circuit device, which may include: a multi-chip package including the integrated circuit.

[0160] In various embodiments, a filter controller, such as 2112 ( Fig.21 and 22 ) or 2312( Fig.23), implemented in an integrated circuit device, which may include: an integrated circuit package including the integrated circuit. In some embodiments, the filter controller 2112 or 2312 is implemented in a single integrated circuit die. In other embodiments, the filter controller 2112 or 2312 is implemented in more than one integrated circuit die of an integrated circuit device, which may include: a multi-chip package including the integrated circuit.

[0161] In various embodiments, a beamformer, such as 2305 ( Fig.23 ) or 2405( Fig.24A ), implemented in an integrated circuit device, which may include: an integrated circuit package including the integrated circuit. In some embodiments, the beamformer 2305 or 2405 is implemented in a single integrated circuit die. In other embodiments, the beamformer 2305 or 2405 is implemented in more than one integrated circuit die of an integrated circuit device, which may include: a multi-chip package including the integrated circuit.

[0162] Fig.25A Beamforming is generally shown at 2500 in accordance with an embodiment of the present invention. Fig.25A , a beamforming module is applied to two microphone inputs 2502 and 2504. In one or more embodiments, microphone input 2502 can originate from a first directional microphone, microphone input 2504 can originate from a second directional microphone, or microphone signals 2502 and 2504 can originate from an omnidirectional microphone. In other embodiments, microphone signals 2502 and 2504 are provided by the output of a bidirectional pressure gradient microphone. Various directional microphones can be used, such as, but not limited to, microphones with a cardioid beam pattern, a dipole beam pattern, an omnidirectional beam pattern, or a user-defined beam pattern. In some embodiments, one or more acoustic elements are configured to provide microphone inputs 2502 and 2504.

[0163] In various embodiments, the beamforming module 2506 includes a filter 2508. Depending on the type of microphone used and the specific application, the filter 2508 can provide a direct current (DC) blocking filter that filters the DC and very low frequency components of the microphone input 2502. After the filter 2508, in some embodiments, additional filtering is provided by a filter 2510. Some microphones have a non-flat response as a function of frequency. In this case, it may be necessary to use a de-emphasis filter to flatten the frequency response of the microphone. The filter 2510 can provide de-emphasis to flatten the frequency response of the microphone. After de-emphasis filtering by the filter 2510, the main microphone channel is provided to the adaptive noise cancellation unit at 2512a and to the desired voice activity detector at 2512b.

[0164] Microphone input 2504 is input into beamforming module 2506 and filtered by filter 2512 in some embodiments. Depending on the type of microphone used and the specific application, filter 2512 can provide a direct current (DC) blocking filter that filters the DC and very low frequency components of microphone input 2504. Filter 2514 filters the sound signal output from filter 2512. Filter 2514 adjusts gain, phase, and can also determine the formation of the frequency response of the sound signal. After filter 2514, in some embodiments, additional filtering is provided by filter 2516. Some microphones have a non-flat response as a function of frequency. In this case, it may be necessary to use a de-emphasis filter to flatten the frequency response of the microphone. Filter 2516 can provide de-emphasis to flatten the frequency response of the microphone. After de-emphasis filtering by filter 2516, reference microphone channel is provided to the adaptive noise cancellation unit at 2518a and provided to the desired voice activity detector at 2518b.

[0165] Optionally, the third microphone channel is input into the beamforming module 2506 at 2504b. Similar to the signal path for channel 2504 described above, filter 2512b filters the third microphone channel. Depending on the type of microphone used and the specific application, filter 2512b can provide a direct current (DC) blocking filter that filters the DC and very low frequency components of microphone input 2504b. Filter 2514b filters the sound signal output from filter 2512b. Filter 2514b adjusts gain, phase, and can also determine the formation of the frequency response of the sound signal. After filter 2514b, in some embodiments, additional filtering is provided by filter 2516b. Some microphones have a non-flat response as a function of frequency. In this case, it may be necessary to use a de-emphasis filter to flatten the frequency response of the microphone. Filter 2516b can provide de-emphasis to flatten the frequency response of the microphone. After de-emphasis filtering through filter 2516b, the second reference microphone channel is provided to the adaptive noise cancellation unit at 2520a and to the desired voice activity detector at 2520b.

[0166] Fig.25B Another schematic diagram of beamforming according to an embodiment of the present invention is generally shown at 2530. Fig.25B, a beam pattern is created for the main channel using the first microphone 2532 and the second microphone 2538. The signal 2534 output from the first microphone 2532 is input to the adder 2536. The amplitude of the signal 2540 output from the second microphone 2538 is adjusted at block 2542, and the phase thereof is adjusted by applying a delay at block 2544, thereby generating a signal 2546 input to the adder 2536. The adder 2536 subtracts one signal from the other signal, thereby obtaining an output signal 2548. Depending on the initial beam patterns of the microphones 2532 and 2538 and the gains applied at 2542 and the delays applied at 2544, the output signal 2548 has a beam pattern that can take on a variety of forms. As non-limiting examples, the beam pattern may include a cardioid, a dipole, and the like.

[0167] A beam pattern is created for the reference channel using the third microphone 2552 and the fourth microphone 2558. The signal 2554 output from the third microphone 2552 is input to the adder 2556. The signal 2560 output from the fourth microphone 2558, its amplitude is adjusted at block 2562, and its phase is adjusted by applying a delay at block 2564, thereby generating a signal 2566 input to the adder 2556. The adder 2556 subtracts one signal from the other signal, thereby obtaining an output signal 2568. Depending on the initial beam patterns of the microphones 2552 and 2558 and the gains applied at 2562 and the delays applied at 2564, the output signal 2568 has a beam pattern that can take on a variety of forms. As non-limiting examples, the beam pattern may include a cardioid, a dipole, etc.

[0168] Fig.25C Beamforming using shared acoustic elements is generally shown at 2570 in accordance with an embodiment of the present invention. Fig.25C , microphone 2552 is shared between the main channel and the reference channel. The output from microphone 2552 is split and sent to gain 2574 and delay 2576 at 2572, and then input to adder 2536 at 2586. The appropriate gain at 2574 and delay at 2576 can be selected to equivalently achieve output 2578 from adder 2536, which is equivalent to output 2548 from adder 2536 ( Fig.25B ). Similarly, gain 2582 and delay 2584 can be adjusted to provide an equivalent gain of 2568 ( Fig.25B ) output signal 2588. As non-limiting examples, beam patterns may include cardioid, dipole, etc.

[0169] Fig.26 Multi-channel adaptive filtering is generally shown at 2600 in accordance with an embodiment of the present invention. Fig.26, an embodiment of the adaptive filtering unit is shown as having a main channel 2604 (containing a microphone signal) input to a delay element 2606. A reference channel 2602 (containing a microphone signal) is input to an adaptive filter 2608. In various embodiments, the adaptive filter 2608 may be an adaptive FIR filter designed to implement normalized least mean square adaptation (NLMS) or another algorithm. Embodiments of the present invention are not limited to NLMS adaptation. The adaptive FIR filter filters the estimated desired sound signal from the reference signal 2602. In one or more embodiments, the output 2609 of the adaptive filter 2608 is input to an adder 2610. The delayed main channel signal 2607 is input to the adder 2610, and the output 2609 is subtracted from the delayed main channel signal 2607. The output 2616 of the adder 2610 provides a signal containing the desired sound signal with a reduced number of undesired sound signals.

[0170] Many situations in which the acoustic system of the embodiment of the present invention is used are used under the condition of reverberation. Reverberation causes a form of noise and causes undesirable sound signals, which are the objects of filtering and signal extraction described herein. In various embodiments, the dual-channel adaptive FIR filtering shown by 2600 simulates the reverberation between the two channels and the environment in which they are applied. Therefore, the undesirable sound signal propagates along the direct path and the reverberation path that requires the adaptive FIR filter to simulate the impulse response of the environment. Depending on the required accuracy, various approximate simulations of the impulse response of the environment can be performed. In a non-limiting example, the delay amount is approximately equal to the impulse response time of the environment. In a non-limiting example, the delay amount is greater than the impulse response of the environment. In one embodiment, the delay amount is approximately equal to n times the impulse response time of the environment, where n can be equal to 2 or 3 or more, for example. Alternatively, the delay amount is not an integer multiple of the impulse response time, for example, 0.5, 1.4, 2.75, etc. For example, in one embodiment, the filter length is approximately equal to twice the delay selected for 2606. Therefore, if an adaptive filter with 200 taps is used, the length of delay 2606 will be approximately equal to the time delay of 100 taps. The time delay equivalent to the propagation time through 100 taps is provided for illustration only and does not imply any form of limitation on the embodiments of the present invention.

[0171] Embodiments of the present invention may be used in a variety of environments with a range of impulse response times. Some examples of impulse response times are given as non-limiting examples for illustrative purposes only and do not constitute limitations on embodiments of the present invention. For example, an office environment typically has an impulse response time of approximately 100 milliseconds to 200 milliseconds. A vehicle interior may provide an impulse response time of 30 milliseconds to 60 milliseconds. In general, embodiments of the present invention are used in environments where the impulse response time may range from a few milliseconds to 500 milliseconds or more.

[0172] The adaptive filter unit 2600 is connected to the inhibit logic at 2614, such as the inhibit logic 2214 and the filter control signal 2114 ( Fig. 22 ) communication. Signal 2614 controlled by inhibit logic 2214 is used to control the filtering performed by filter 2608 and the adaptation of the filter coefficients. The output 2616 of adaptive filtering unit 2600 is input to a single channel noise cancellation unit such as described in the previous figures above, for example, 2118 ( Fig.21 )、2318( Fig.23 ), and 2418( Fig.24A ). The first level of undesired sound signals has been extracted from the main channel, resulting in output 2616. Under various operating conditions, the level of noise, i.e., undesired sound signals, may be very large relative to the signal of interest, i.e., the desired sound signal. Embodiments of the present invention are operable in the presence of a certain difference in signal-to-noise ratio between the main channel and the reference channel. In some embodiments, the difference in signal-to-noise ratio is on the order of about 1 decibel (dB) or less. In other embodiments, the difference in signal-to-noise ratio is on the order of about 1 decibel (dB) or greater. Output 2616 is additionally filtered to reduce the amount of undesired sound signals contained therein during subsequent use of a single-channel noise reduction unit.

[0173] above Fig. 22 The signal 2614 ( Fig.26 ) provides substantially inoperative filter 2608 and non-adaptation of filter coefficients when the main channel or reference channel is determined to be inactive. In this case, the signal present on the main channel 2604 is output at 2616.

[0174] If the main channel and the reference channel are active and the desired sound signal is detected or the pause threshold has not been reached, adaptation is disabled, the filter coefficients are frozen, and the signal on the reference channel 2602 is filtered through filter 2608 and subtracted from the main channel 2607 using adder 2610 and output at 2616.

[0175] If the primary channel and the reference channel are active and no desired sound signal is detected and the pause threshold (also called pause time) is exceeded, the filter coefficients are adjusted. The pause threshold depends on the application. For example, in a non-limiting example, in the case of automatic speech recognition (ASR), the pause threshold can be on the order of a fraction of a second.

[0176] Fig. 27 Single channel filtering is generally shown at 2700 in accordance with an embodiment of the present invention. Fig. 27 , the single channel noise reduction unit utilizes a linear filter with a single channel input. Examples of filters suitable for use therein are Wiener filters, filters using minimum mean square error (MMSE), etc. The output from the adaptive noise cancellation unit (as described above) is input to the filter 2702 at 2704. The input signal 2704 contains the desired sound signal and the noise component, i.e., the undesired sound signal, which is represented as the total power in equation 2714 Filter 2702 applies the equation shown by 2714 to input signal 2704. Total Power The estimated value of is a term in the numerator of equation 2714 and is obtained from the input of filter 2702. When the desired sound signal is missing from signal 2704, an estimated value of the noise is obtained That is, the unwanted sound signal. Noise estimation value is the numerator of the total power The total power is the term in the denominator of Equation 2714. The estimated value of the noise (obtained in the absence of the desired sound signal) is obtained from input signal 2704, which is obtained by inhibiting logic, such as inhibiting logic 2214 ( Fig. 22 ), the inhibit logic indicates when the desired sound signal is present and when the desired sound signal is not present. When the desired sound signal is not present on signal 2704, the noise estimate is updated. When the desired sound signal is present, the noise estimate is frozen and filtered using the noise estimate previously determined during the last interval when the desired sound signal was not present.

[0177] Fig.28A Desired voice activity detection according to an embodiment of the present invention is generally shown at 2800. Fig.28A, a dual input desired speech detector is shown at 2806. The sound signal from the main channel is input to the first signal path 2807a of the dual input desired speech detector 2806 from, for example, a beamformer or from the main channel described above in conjunction with the previous figures at 2802. The first signal path 2807a includes a voice band filter 2808. The voice band filter 2808 captures most of the desired speech energy in the main channel 2802. In various embodiments, the voice band filter 2808 is a bandpass filter characterized by having a lower corner frequency, an upper corner frequency, and a roll-off from the upper corner frequency. In various embodiments, depending on the application, the lower corner frequency can range from 50 to 300 Hz. For example, in a wideband phone, the lower corner frequency is about 50 Hz. In a standard phone, the lower corner frequency is about 300 Hz. The upper corner frequency is selected to allow the filter to pass most of the speech energy picked up by the relatively flat portion of the frequency response of the microphone. Therefore, depending on the application, the upper corner frequency can be set at different positions. A non-limiting example of a location is 2500 Hz. Another non-limiting location of the upper corner frequency is 4000 Hz.

[0178] The first signal path 2807a includes a short-term power calculator 2810. The short-term power calculator 2810 is implemented in various embodiments as a root mean square (RMS) measurement, a power detector, an energy detector, etc. The short-term power calculator 2810 may be synonymously referred to as a short-time power calculator 2810. The short-term power detector 2810 approximately calculates the instantaneous power in the filtered signal. The output of the short-term power detector 2810 (Y1) is input to a signal compressor 2812. In various embodiments, the compressor 2812 converts the signal to the Log2 domain, Log 10 domain, etc. In other embodiments, the compressor 2812 performs a user-defined compression algorithm on the signal Y1.

[0179] Similar to the first signal path described above, the sound signal from the reference channel is input at 2804 from, for example, a beamformer or from the reference channel as described above in connection with the previous figures to a second signal path 2807b of a dual input desired speech detector 2806. The second signal path 2807b includes a voice band filter 2816. The voice band filter 2816 captures most of the desired speech energy in the reference channel 2804. In various embodiments, the voice band filter 2816 is a bandpass filter characterized by having a lower corner frequency, an upper corner frequency, and a roll-off from the upper corner frequency as described above for the first signal path and the voice band filter 2808.

[0180] The second signal path 2807b includes a short-term power calculator 2818. The short-term power calculator 2818 is implemented in various embodiments as a root mean square (RMS) measurement, a power detector, an energy detector, etc. The short-term power calculator 2818 may be synonymously referred to as a short-time power calculator 2818. The short-term power detector 2818 approximately calculates the instantaneous power in the filtered signal. The output of the short-term power detector 2818 (Y2) is input to a signal compressor 2820. In various embodiments, the compressor 2820 converts the signal to the Log2 domain, Log 10 domain, etc. In other embodiments, the compressor 2820 performs a user-defined compression algorithm on the signal Y2.

[0181] At subtractor 2824, the compressed signal from the second signal path 2822 is subtracted from the compressed signal from the first signal path 2814, which results in a normalized main signal at 2826 (Z). In other embodiments, different compression functions are applied at 2812 and 2820, which results in different normalization of the signals at 2826. In other embodiments, when logarithmic compression is not implemented, a division operation may be applied at 2824 to accomplish the normalization, such as when compression based on a square root function is implemented.

[0182] The normalized main signal 2826 is input to a single channel normalized voice threshold comparator (SC-NVTC) 2828, resulting in a normalized desired voice activity detection signal 2830. Note that the architecture of the dual channel voice activity detector provides detection of desired speech using a normalized desired voice activity detection signal 2830 based on the overall difference in the signal to noise ratio of the two input channels. Thus, the normalized desired voice activity detection signal 2830 is based on the integral of the energy in the voice band rather than the energy in a specific frequency bin, thereby maintaining linearity within the noise cancellation unit described above. The compressed signals 2814 and 2822, using logarithmic compression, provide an input at 2826 (Z) having a noise floor that can take values ​​that vary from below zero to above zero (see below). Fig.28E 2895c, 2895d, or 2895e in the 12-bit DAC), unlike the uncompressed single-channel input, whose noise floor is always above zero (see below). Fig.28E Column 2895b).

[0183] Fig.28B A single channel normalized speech threshold comparator (SC-NVTC) is generally shown at 2850 according to an embodiment of the present invention. Fig.28B, the normalized main signal 2826 is input to a long-term normalized power estimator 2832. The long-term normalized power estimator 2832 provides a running estimate of the normalized main signal 2826. The running estimate provides a lower limit for the desired sound signal. The offset value 2834 is added to the running estimate of the output of the long-term normalized power estimator 2832 in an adder 2836. The output of the adder 2838 is input to a comparator 2840. An instantaneous estimate 2842 of the normalized main signal 2826 is input to the comparator 2840. The comparator 2840 contains logic to compare the instantaneous value at 2842 to the running ratio at 2838 plus the offset. If the value at 2842 is greater than the value at 2838, the desired sound signal is detected, and a flag is set accordingly and transmitted as part of the normalized desired voice activity detection signal 2830. If the value at 2842 is less than the value at 2838, the desired sound signal is not detected, and a flag is set accordingly and transmitted as part of the normalized desired voice activity detection signal 2830. The long-term normalized power estimator 2832 averages the normalized main signal 2826 over a sufficiently long period of time to slow down the change in the amplitude fluctuations. Thus, the amplitude fluctuations at 2833 vary slowly. The averaging time may vary from a fraction of a second to several minutes, as non-limiting examples. In various embodiments, the averaging time is selected to provide slowly varying amplitude fluctuations at the output of 2832.

[0184] Fig.28C Desired voice activity detection using multiple reference channels is generally shown at 2846 in accordance with an embodiment of the present invention. Fig.28C , the desired speech detector is shown at 2848. The desired speech detector 2848 includes as input the main channel 2802 and the first signal path 2807a (the above combined Fig.28A described above) and reference channel 2804 and second signal path 2807b (also described above in conjunction with Fig.28A In addition, there is a second reference channel 2850 that is input to the desired speech detector 2848 and is part of the third signal path 2807c. Similar to the second signal path 2807b (described above), the sound signal from the second reference channel is input to the third signal path 2807b of the multiple input desired speech detector 2848 at 2850 from, for example, a beamformer or from the second reference channel described above in conjunction with the previous figures. The third signal path 2807b includes a voice band filter 2852. The voice band filter 2852 captures most of the desired speech energy in the second reference channel 2850. In various embodiments, the voice band filter 2852 is a bandpass filter characterized by having a lower corner frequency, an upper corner frequency, and a roll-off from the upper corner frequency as described above for the second signal path and the voice band filter 2808.

[0185] The third signal path 2807c includes a short-term power calculator 2854. The short-term power calculator 2854 is implemented in various embodiments as a root mean square (RMS) measurement, a power detector, an energy detector, etc. The short-term power calculator 2854 may be synonymously referred to as a short-time power calculator 2854. The short-term power detector 2854 approximately calculates the instantaneous power in the filtered signal. The output of the short-term power detector 2854 is input to a signal compressor 2856. In various embodiments, the compressor 2856 converts the signal to the Log2 domain, Log 10 domain, etc. In other embodiments, the compressor 2854 performs a user-defined compression algorithm on the signal Y3.

[0186] At subtractor 2860, the compressed signal from the third signal path 2858 is subtracted from the compressed signal from the first signal path 2814, which results in a normalized main signal at 2862 (Z2). In other embodiments, different compression functions are applied at 2856 and 2812, which results in different normalization of the signals at 2862. In other embodiments, when logarithmic compression is not implemented, a division operation may be applied at 2860. For example, when compression based on a square root function is implemented.

[0187] The normalized main signal 2862 is input to a single channel normalized voice threshold comparator (SC-NVTC) 2864, resulting in a normalized desired voice activity detection signal 2868. Note that the architecture of the multi-channel voice activity detector uses a normalized desired voice activity detection signal 2868 based on the overall difference in the signal-to-noise ratio of the two input channels to provide detection of desired speech. Therefore, the normalized desired voice activity detection signal 2868 is based on the integral of the energy in the voice band rather than the energy in a specific frequency bin, thereby maintaining the linearity within the noise cancellation unit described above. The compressed signals 2814 and 2858, using logarithmic compression, provide an input at 2862 (Z2) having a noise floor that can take values ​​that vary from below zero to above zero (see below). Fig.28E 2895c, 2895d, or 2895e), unlike the uncompressed single-channel input, whose noise floor is always above zero (see below). Fig.28E Column 2895b).

[0188] The desired speech detector 2848, having a multi-channel input using at least two reference channel inputs, provides two normalized desired voice activity detection signals 2868 and 2870 for outputting a desired voice activity signal 2874. In one embodiment, the normalized desired voice activity detection signals 2868 and 2870 are input to a logic OR gate 2872. The logic OR gate outputs a desired voice activity signal 2874 based on its inputs 2868 and 2870. In other embodiments, additional reference channels may be added to the desired speech detector 2848. Each additional reference channel is used to create another normalized main channel, which is input to another single channel normalized voice threshold comparator (SC-NVTC) (not shown). The output from the additional single channel normalized voice threshold comparator (SC-NVTC) (not shown) is combined with 2874 through an additional XOR gate (also not shown) (in one embodiment) to provide a desired voice activity signal, which is output as described above in conjunction with the previous figure. Utilizing additional reference channels in a multi-channel wanted speech detector, as described above, results in a more robust detection of the wanted speech signal because more information about the noise field is obtained through the multiple reference channels.

[0189] Fig.28D A process for using compression according to an embodiment of the present invention is generally shown at 2880. Fig.28D , the process begins at block 2882. Using the combination of Fig.28A or Fig.28C As shown, for example, Log 10 compression or user defined compression, compress the main channel at block 2884. Use such as Fig.28A or Fig.28C As shown, for example, Log 10 Compression or user defined compression, compresses the reference channel at block 2886. A normalized main sound signal is created at block 2888. The desired speech is detected using the normalized sound signal at block 2890. The process stops at block 2892.

[0190] Fig.28E Different functions for providing compression are shown generally at 2893 according to an embodiment of the present invention. Fig.28E, for illustrative purposes, Table 2894 shows several compression functions, but this is not meant to be limiting. Column 2895a contains six sample values ​​of variable x. In this example, as shown at 2896, the value range of variable X is from 0.01 to 1000.0. Column 2895b shows no compression, where Y=X. Column 2895c shows base 10 logarithmic compression, where the compressed value Y=log10(X). Column 2895d shows ln(X) compression, where the compressed value Y=ln(X). Column 2895e shows base 2 logarithmic compression, where Y=Log2(X). User-defined compression (not shown) can also be implemented as needed to provide more or less compression than 2895c, 2895d, or 2895e. Utilizing the compression functions at 2812 and 2820 ( Fig.28A ) compresses the results of the short-term power detectors 2810 and 2818, reducing the dynamic range of the normalized main signal at 2826 (Z) input to the single-channel normalized speech threshold comparator (SC-NVTC) 2828. Similarly, the compression functions ( Fig.28C ) compresses the results of short term power detectors 2810, 2818, and 2854, reducing the dynamic range of the normalized main signals at 2826 (Z) and 2862 (Z2) input to SC-NVTC 828 and SC-NVTC 864, respectively. The reduced dynamic range achieved through compression can result in more accurate detection of the presence of a desired sound signal, and thus a greater degree of noise reduction can be achieved through the embodiments of the present invention presented herein.

[0191] In various embodiments, such as Fig.28A , Fig.28B , Fig.28C , Fig.28D as well as Fig.28E The components of the illustrated multiple-input speech required detector are implemented in an integrated circuit device, which may include: an integrated circuit package including the integrated circuit. In some embodiments, the multiple-input speech required detector is implemented in a single integrated circuit die. In other embodiments, the multiple-input speech required detector is implemented in more than one integrated circuit die of an integrated circuit device, which may include: a multi-chip package including the integrated circuit.

[0192] Fig.29A An automatic balancing architecture is generally shown at 2900 according to an embodiment of the present invention. Fig.29A, the automatic balancing component 2903 has a first signal path 2905a and a second signal path 2905b. At 2902b, the first channel 2902a (MIC 1) is connected to the first signal path 2905a. At 2904b, the second channel 2904a is connected to the second signal path 2905b. The sound signal is input to the voice band filter 2906 at 2902b. The voice band filter 2906 captures most of the desired speech energy in the first channel 2902a. In various embodiments, the voice band filter 1906 is a bandpass filter characterized by having a lower corner frequency, an upper corner frequency, and a roll-off from the upper corner frequency. In various embodiments, depending on the application, the lower corner frequency can range from 50 to 300Hz. For example, in a broadband phone, the lower corner frequency is about 50Hz. In a standard phone, the lower corner frequency is about 300Hz. The upper corner frequency is selected to allow the filter to pass most of the speech energy picked up by the relatively flat portion of the frequency response of the microphone. Therefore, depending on the application, the upper corner frequency can be set at different positions. A non-limiting example of one position is 2500 Hz. Another non-limiting position of the upper corner frequency is 4000 Hz.

[0193] The first signal path 2905a includes a long term power calculator 2908. The long term power calculator 2908 is implemented in various embodiments as a root mean square (RMS) measurement, a power detector, an energy detector, etc. The long term power calculator 2908 may be synonymously referred to as a long term power calculator 2908. The long term power detector 2908 approximately calculates a running average long term power in the filtered signal. The output 2909 of the long term power detector 2908 is input to a divider 2917. A control signal 2914 is input to the long term power calculator 2908 at 2916. The control signal 2914 provides a signal as described above in conjunction with the desired sound signal detector, such as Fig.28A , Fig.28B , Fig.28C , which indicates when the desired sound signal is present and when the desired sound signal is not present. Segments of the sound signal on the first channel 2902b where the desired sound signal is present are excluded from the long-term power average generated at 2908.

[0194] The sound signal is input to the voice band filter 2910 of the second signal path 2905b at 2904b. The voice band filter 2910 captures most of the desired speech energy in the second channel 2904a. In various embodiments, the voice band filter 2910 is a bandpass filter characterized by having a lower corner frequency, an upper corner frequency, and a roll-off from the upper corner frequency. In various embodiments, the lower corner frequency can range from 50 to 300Hz, depending on the application. For example, in a broadband phone, the lower corner frequency is about 50Hz. In a standard phone, the lower corner frequency is about 300Hz. The upper corner frequency is selected to allow the filter to pass most of the speech energy picked up by the relatively flat portion of the frequency response of the microphone. Therefore, depending on the application, the upper corner frequency can be set at different positions. A non-limiting example of a position is 2500Hz. Another non-limiting position of the upper corner frequency is 4000Hz.

[0195] The second signal path 2905b includes a long term power calculator 2912. The long term power calculator 2912 is implemented in various embodiments as a root mean square (RMS) measurement, a power detector, an energy detector, etc. The long term power calculator 2912 may be synonymously referred to as a long term power calculator 2912. The long term power detector 2912 approximately calculates a running average long term power in the filtered signal. The output 2913 of the long term power detector 2912 is input to a divider 2917. A control signal 2914 is input to the long term power calculator 2912 at 2916. The control signal 2914 provides a signal as described above in conjunction with the desired sound signal detector, such as Fig.28A , Fig.28B , Fig.28C , which indicates when the desired sound signal is present and when the desired sound signal is not present. Segments of the sound signal on the second channel 2904b where the desired sound signal is present are excluded from the long-term power average generated at 2912.

[0196] In one embodiment, output 2909 is normalized at 2917 by output 2913 to produce an amplitude correction signal 2918. In one embodiment, a divider is used at 2917. Amplitude correction signal 2918 is multiplied by the instantaneous value of the second microphone signal at 2904a at multiplier 2920 to produce a corrected second microphone signal at 2922.

[0197] In another embodiment, alternatively, output 2913 is normalized at 2917 by output 2909 to produce an amplitude correction signal 2918. In one embodiment, a divider is used at 2917. Amplitude correction signal 2918 is multiplied by the instantaneous value of the first microphone signal on 1902a using a multiplier connected to 2902a (not shown) to produce a corrected first microphone signal for first microphone channel 2902a. Thus, in various embodiments, either the second microphone signal is automatically balanced relative to the first microphone signal, or alternatively, the first microphone signal is automatically balanced relative to the second microphone signal.

[0198] It should be noted that the long term average power calculated at 2908 and 2912 is performed when the desired sound signal is not present. Thus, the average power represents the average value of the undesired sound signal typically generated in the far field. In various embodiments, as a non-limiting example, in some embodiments, the duration of the long term power calculator ranges from about a fraction of a second, e.g., one-half second to five seconds to several minutes, and depends on the application.

[0199] Fig.29B Automatic balancing is generally shown at 2950 in accordance with an embodiment of the present invention. Fig.29B , the automatic balancing component 2952 is configured to receive the main channel 2954a and the reference channel 2956a as input. The balancing function is similar to the above combined Fig.29A Execution continues with the description given using the first channel 2902a (MIC 1) and the second channel 2904a (MIC 2).

[0200] refer to Fig.29B, the automatic balancing component 2952 has a first signal path 2905a and a second signal path 2905b. At 2954b, the first channel 2954a (main) is connected to the first signal path 2905a. At 2956b, the second channel 2956a is connected to the second signal path 2905b. The sound signal is input to the voice band filter 2906 at 2954b. The voice band filter 2906 captures most of the desired speech energy in the first channel 2954a. In various embodiments, the voice band filter 2906 is a bandpass filter characterized by having a lower corner frequency, an upper corner frequency, and a roll-off from the upper corner frequency. In various embodiments, depending on the application, the lower corner frequency can range from 50 to 300Hz. For example, in a broadband phone, the lower corner frequency is about 50Hz. In a standard phone, the lower corner frequency is about 300Hz. The upper corner frequency is selected to allow the filter to pass most of the speech energy picked up by the relatively flat portion of the frequency response of the microphone. Therefore, depending on the application, the upper corner frequency can be set at different positions. A non-limiting example of one position is 2500 Hz. Another non-limiting position of the upper corner frequency is 4000 Hz.

[0201] The first signal path 2905a includes a long term power calculator 2908. The long term power calculator 2908 is implemented in various embodiments as a root mean square (RMS) measurement, a power detector, an energy detector, etc. The long term power calculator 2908 may be synonymously referred to as a long time power calculator 2908. The long term power detector 2908 approximately calculates a running average long term power in the filtered signal. The output 2909b of the long term power detector 2908 is input to a divider 2917. A control signal 2914 is input to the long term power calculator 2908 at 2916. The control signal 2914 provides a signal as described above in conjunction with the desired sound signal detector, such as Fig.28A , Fig.28B , Fig.28C , which indicates when the desired sound signal is present and when the desired sound signal is not present. Segments of the sound signal on the first channel 2954b where the desired sound signal is present are excluded from the long-term power average generated at 2908.

[0202] The sound signal is input to the voice band filter 2910 of the second signal path 2905b at 2956b. The voice band filter 2910 captures most of the desired speech energy in the second channel 2956a. In various embodiments, the voice band filter 2910 is a bandpass filter characterized by having a lower corner frequency, an upper corner frequency, and a roll-off from the upper corner frequency. In various embodiments, the lower corner frequency can range from 50 to 300Hz, depending on the application. For example, in a broadband phone, the lower corner frequency is about 50Hz. In a standard phone, the lower corner frequency is about 300Hz. The upper corner frequency is selected to allow the filter to pass most of the speech energy picked up by the relatively flat portion of the frequency response of the microphone. Therefore, depending on the application, the upper corner frequency can be set at different positions. A non-limiting example of a position is 2500Hz. Another non-limiting position of the upper corner frequency is 4000Hz.

[0203] The second signal path 2905b includes a long term power calculator 2912. The long term power calculator 2912 is implemented in various embodiments as a root mean square (RMS) measurement, a power detector, an energy detector, etc. The long term power calculator 2912 may be synonymously referred to as a long term power calculator 2912. The long term power detector 2912 approximately calculates a running average long term power in the filtered signal. The output 2913b of the long term power detector 2912 is input to a divider 2917. A control signal 2914 is input to the long term power calculator 2912 at 2916. The control signal 2916 provides a signal as described above in conjunction with the desired sound signal detector, such as Fig.28A , Fig.28B , Fig.28C , which indicates when the desired sound signal is present and when the desired sound signal is not present. Segments of the sound signal on the second channel 2956b where the desired sound signal is present are excluded from the long-term power average generated at 2912.

[0204] In one embodiment, output 2909b is normalized by output 2913b at 2917 to produce an amplitude correction signal 2918b. In one embodiment, a divider is used at 2917. Amplitude correction signal 2918b is multiplied by the instantaneous value of the second microphone signal at 2956a at multiplier 2920 to produce a corrected second microphone signal at 2922b.

[0205] In another embodiment, either output 2913b is normalized at 2917 by output 2909b to produce an amplitude correction signal 2918b. In one embodiment, a divider is used at 2917. Amplitude correction signal 2918b is multiplied by the instantaneous value of the first microphone signal at 2954a using a multiplier connected to 2954a (not shown) to produce a corrected first microphone signal for first microphone channel 2954a. Thus, in various embodiments, either the second microphone signal is automatically balanced relative to the first microphone signal, or alternatively, the first microphone signal is automatically balanced relative to the second microphone signal.

[0206] It should be noted that the long term average power calculated at 2908 and 2912 is performed when the desired sound signal is not present. Thus, the average power represents the average value of the undesired sound signal typically generated in the far field. In various embodiments, as a non-limiting example, in some embodiments, the duration of the long term power calculator ranges from about a fraction of a second, e.g., one-half second to five seconds to several minutes, and depends on the application.

[0207] Embodiments of the automatic balancing component 2902 or 2952 are configured to automatically balance multiple microphone channels, such as Fig.24A In such a configuration, multiple channels (e.g., multiple reference channels) are balanced relative to the main channel. Or multiple reference channels and one main channel are balanced relative to the above combination. Fig.29A or Fig.29B A particular reference channel is described as being balanced.

[0208] Fig.29C Filtering is shown according to an embodiment of the present invention. Fig.29C , 2960a shows two microphone signals 2966a and 2968a having amplitudes 2962 plotted as a function of frequency 2964. In some embodiments, the microphone does not have a constant sensitivity as a function of frequency. For example, microphone response 2966a may represent a microphone output (response) having a non-flat frequency response excited by a frequency-flat broadband excitation. Microphone response 2966a includes a non-flat region 2974 and a flat region 2970. For this example, the microphone that produces response 2968a has a uniform sensitivity with respect to frequency; therefore, 2968a is generally flat in response to the frequency-flat broadband excitation. In some embodiments, it is meaningful to balance the flat region 2970 of the microphone response. In this case, the non-flat region 2974 is filtered out so that the energy in the non-flat region 2974 does not affect the microphone automatic balancing procedure. What is of interest is the difference 2972 ​​between the flat regions of the two microphone responses.

[0209] In 2960b, a filter function 2978a is shown and plotted with amplitude 2976 plotted as a function of frequency 2964. In various embodiments, the filter function is selected to eliminate the non-flat portion 2974 of the microphone response. The filter function 2978a is characterized by having a lower corner frequency 2978b and an upper corner frequency 2978c. The filter function of 2960b is applied to two microphone signals 2966a and 2968a, the results of which are shown in 2960c.

[0210] In 2960c, filtered representations 2966c and 2968c of microphone signals 2966a and 2968a are plotted as a function of amplitude 2980 and frequency 2966. Difference 2972 ​​represents the sensitivity difference between the two filtered microphone signals 2966c and 2968c. Fig.29A and Fig.29B In the described system, this difference between the responses of the two microphones is balanced. Fig.29A and Fig.29B In various embodiments, voice band filters 2906 and 2910, in one non-limiting example, may apply the filter function shown in 2960b to microphone channels 2902b and 2904b ( Fig.29A ) or the main channel and reference channels 2954b and 2956b ( Fig.29B ). Through the above Fig.29A or Fig.29B The automatic balancing procedure described in 2972 ​​can minimize or eliminate the differences between the two microphone channels.

[0211] Fig.30 According to an embodiment of the present invention, an automatic balancing process is generally shown by 3000. Fig.30 , the process begins at block 3002. In block 3004, the average long-term power in the first microphone channel is calculated. The average long-term power calculated for the first microphone channel does not include segments of the microphone signal that appear when the desired sound signal is present. The input from the desired voice activity detector is used to exclude relevant portions of the desired sound signal. In block 3006, the average power in the second microphone channel is calculated. The average long-term power calculated for the second microphone channel does not include segments of the microphone signal that appear when the desired sound signal is present. The input from the desired voice activity detector is used to exclude relevant portions of the desired sound signal. In block 3008, the amplitude correction signal is calculated using the average values ​​calculated in blocks 3004 and 3006.

[0212] In various embodiments, the components of the automatic balancing component 2903 or 2952 are implemented in an integrated circuit device, which may include: an integrated circuit package containing the integrated circuit. In some embodiments, the automatic balancing component 2903 or 2952 is implemented in a single integrated circuit die. In other embodiments, the automatic balancing component 2903 or 2952 is implemented in more than one integrated circuit die of the integrated circuit device, which may include: a multi-chip package containing the integrated circuit.

[0213] Fig.31 A sound signal processing system in which embodiments of the present invention may be used is generally shown at 3100. The block diagram is a high-level conceptual representation that may be implemented in various ways and with various architectures. Fig.31 , bus system 3102 interconnects central processing unit (CPU) 3104, read-only memory (ROM) 3106, random access memory (RAM) 3108, storage 3110, display 3120, audio 3122, keyboard 3124, pointer 3126, data acquisition unit (DAU) 3128 and communication 3130. Bus system 3102 can be one or more of the following buses, such as system bus, peripheral component interconnect (PCI), advanced graphics port (AGP), small computer system interface (SCSI), Institute of Electrical and Electronics Engineers (IEEE) standard number 1394 (Firewire), universal serial bus (USB), or a dedicated bus designed for custom applications. CPU 3104 can be a single, multiple, or even distributed computing resource or digital signal processing (DSP) chip. Storage 3110 can be a compact disk (CD), digital versatile disk (DVD), hard disk (HD), optical disk, tape, flash memory, memory stick, video recorder, etc. The sound signal processing system 3100 can be used to receive sound signals input from multiple microphones (e.g., a first microphone, a second microphone, etc.), or from the main channel and multiple reference channels as described above in conjunction with the previous figures. Please note that depending on the actual implementation of the sound signal processing system, the sound signal processing system may include some, all, more, or rearrangement of the components in the block diagram. In some embodiments, various aspects of the system 3100 are implemented in software, while in some embodiments, various aspects of the system 3100 are implemented in dedicated hardware such as a digital signal processing (DSP) chip, as well as in a combination of dedicated hardware and software that is well known and understood by those of ordinary skill in the art.

[0214] Thus, in various embodiments, sound signal data is received at 3129 for processing by the sound signal processing system 3100. Such data may be transmitted at 3132 via the communication interface 3130 for further processing at a remote location. As will be appreciated by those skilled in the art, a connection to a network such as an intranet or the Internet is obtained via 3132, which enables the sound signal processing system 3100 to communicate with other data processing devices or systems at remote locations.

[0215] For example, the present invention may be implemented on a computer system 3100 configured as a desktop computer or a workstation, for example, running a computer such as XP Home or Compatibility with XP Professional, Linux, Unix and other operating systems , and computers from Apple Computer running operating systems such as OS X, etc. Alternatively, in conjunction with such an embodiment, embodiments of the present invention may be configured with devices such as speakers, headphones, video monitors, etc. configured for use with a Bluetooth communication channel. In other embodiments, embodiments of the present invention are configured to be implemented by mobile devices such as smart phones, tablet computers, wearable devices such as glasses, near-to-eye (NTE) headphones, generally configured head-mounted devices such as but not limited to glasses, goggles, visors, headbands, helmets, etc., or similar devices.

[0216] In one or more embodiments, a hearing aid is provided to the user to facilitate hearing sounds from the local environment.

[0217] Fig.32A A microphone configuration on a head mounted device is shown generally at 3200 in accordance with an embodiment of the present invention. Fig.32B According to an embodiment of the present invention, the Fig.32A A top view of the microphone configuration on the corresponding head-mounted device. Fig.32C According to an embodiment of the present invention, the Fig.32A Bottom view of the microphone configuration on the corresponding head-mounted device. Fig.33 According to an embodiment of the present invention, generally shown by 3300 is a diagram of Fig.32A of head mounted devices relative to different sound sources. Also refer to FIG. 32A to FIG. 33 , the head mounted device 3201 is presented in the shape of glasses used in a three-dimensional space. The three-dimensional space is represented by the X, Y, and Z axes at 3301 ( Fig.33). The three-dimensional space is represented by a Cartesian coordinate system known in the art. However, this is not meant to be limiting. The three-dimensional space may be represented by another coordinate system. In other embodiments, the head-mounted device is in the shape of goggles, etc., which is not meant to be limiting. Here, the term "glasses" or "glasses device" is synonymous with the head-mounted device. The head-mounted device 3201 has a front frame that includes one or more lenses made of glass or plastic, a left frame 3214, and a right frame 3212. The left frame and the right frame are also referred to as temples in the art. The head-mounted device is shown as having four microphones, microphone 0 (3202), microphone 1 (3204), microphone 2 (3206), and microphone 3 (3210). In one or more embodiments, microphone 0 (3202) is located below the bottom of the left frame 3214, and microphone 1 (3204) and microphone 2 (3206) are located at the top of the left frame 3214. Microphone 3 (3210) is located at the top of the right frame 3212. Alternatively, microphone 0 ( 3202 ), microphone 1 ( 3204 ), and microphone 2 ( 3206 ) are located on the right side frame 3212 , and microphone 3 ( 3210 ) is located on the left side frame 3214 .

[0218] In various embodiments, the eyeglass device includes a microphone array connected to at least one side frame member. The microphone array includes at least one first and second microphones. In one or more embodiments, the first microphone and the second microphone, such as 3202 and 3204, are located at the side frame member 3214 near the front frame member. As shown by L2 at 3209, the distance between the first and second microphones and the front frame member is approximately between 5 mm and 30 mm, and can be about 15 mm ( Fig.32B ). The first microphone (microphone 0 (3202)) is located on the bottom side of the side frame member 3214, and the second microphone (microphone 1 (3204)) is located on the top side of the side member 3214 and just or close to the top of the side frame member 3214. In another embodiment, the third microphone (microphone 2 (3206)) is located on the side frame member 3214 and away from the front frame member. As shown by L1 at 3208, the third microphone (microphone 2 (3206)) is located approximately between 10mm and 20mm from the first and / or second microphones (3202 / 3204), and can be about 15mm. If the distance L1 is too long, the third microphone (microphone 2 (3206)) can be close to a speaker embedded in the side frame member and located near the wearer's ear. In this case, there may be an echo from the speaker to microphone 2 (3206). For specific embodiments, this echo is resolved by reducing the distance L1. The reduction in distance L1 increases the separation distance between microphone 2 (3206) and speaker 3350, thereby reducing any echo.

[0219] In another embodiment, a fourth microphone (Microphone 3 (3210)) is located on the other side frame member 3212. Microphone 3 (3210) is shown near the front frame member, but may be located elsewhere along the frame member 3212. The distance between Microphone 1 (3204) and Microphone 3 (3210) is determined by the width of the eyeglass frame and is large enough for the system to detect the difference in signal levels from the two microphones. The distance between Microphone 1 (3204) and Microphone 3 (3210) is not a fixed number, but is typically provided by the geometry and size of the head mounted device. Similarly, the distance between Microphone 0 (3202) and Microphone 3 (3210) is not a fixed number, but is typically provided by the geometry and size of the head mounted device.

[0220] Fig.32D A perspective view of another set of microphone arrangements on a head mounted device is shown generally at 3260 in accordance with an embodiment of the present invention. Fig.32E According to an embodiment of the present invention, the Fig.32D Bottom view of the corresponding microphone layout on the headset. Fig.32D , Microphone 0 (3202) and Microphone 1 (3204) are located on the inner surface of the temple 3212. Microphone 2 (3206) is located on the bottom surface of the right temple 3212 and is set back from Microphone 0 (3202) / Microphone 1 (3204) by an amount equal to L1 as described above. The distance between Microphone 0 (3202) / Microphone 1 (3204) and the front frame is L2 ( Fig.32B ). Return to reference Fig.32D , Microphone 3 (3210) is located on the bottom side of the left temple 3214. Alternatively, one or both of Microphone 2 (3206) and Microphone 3 (3210) can be located on the top surface of their respective temples.

[0221] In an alternative embodiment, Fig.32D The microphone layout shown in / 32E can be reversed relative to the temples. For example, microphone 0 (3202), microphone 1 (3204), microphone 2 (3206) can be located on the inner surface of the left temple 3214, and microphone 3 (3210) can be located on the right temple 3212.

[0222] The four microphones support three or more microphone combinations for different usage scenarios described herein as Configuration 1 using microphone 0 and microphone 1, Configuration 2 using microphone 1 and microphone 2, and Configuration 3 using microphone 1 and microphone 3. In some embodiments, a software interface is used to control switching between these microphone combinations and sequencing between configurations.

[0223] In various embodiments, the glasses will have more than four microphones or less than four microphones. Four microphones are used to illustrate one or more embodiments described herein and do not constitute a limitation of embodiments of the invention. Three configurations of microphones are described below for use by a user of a head mounted device, receiving and processing sound signals to assist the user's hearing, and in some cases for remote use through, for example, voice recognition, command and control, reception and listening by another user, and for local use through embedded voice recognition, etc. The configurations described below can be used to provide the main sound signal and the reference sound signal used in the above-mentioned noise cancellation system.

[0224] Configuration 1

[0225] In one or more embodiments, when the user wears the head mounted device 101 and speaks at the same time, microphone 0 and microphone 1 are used to process sound signals. In configuration 1, the signals output from microphone 0 and microphone 1 are beamformed so that the main sound signal response is in a downward position along axis 3302. Axis 3302 is in the nominal direction of the user's mouth 3310, but does not need to be precisely aligned with it. Microphone 0 and microphone 1 have different acoustic distances to the user's mouth 3320, wherein the acoustic distance of microphone 0 is less than the acoustic distance of microphone 1. The sound signal 3312 emitted from the user's mouth 3310 is received with the maximum acoustic sensitivity to the direction of the user 3310 relative to the microphone pair microphone 0 and microphone 1. The sound signal obtained in this way is used as the main signal input to the multi-channel noise cancellation system. By beamforming the microphone pair microphone 0 and microphone 1 with the main response turned 180 degrees away from the sound source 3310, a reference signal mainly containing noise (mainly undesired sound signals) is obtained. Thus, the reference signal is obtained in a direction looking upward along axis 3302, away from the user's mouth 3310, and toward a potential noise source, such as the noise source represented by 3360, which emits noise 3362 (unwanted sound signal). The signal obtained away from the user's mouth 3310 like this is used as a reference signal for input to the above-mentioned multi-channel noise cancellation system. The beamforming applied to the reference signal minimizes the acoustic sensitivity 3 to the signal from the user's mouth 3310 and maximizes the sensitivity to noise generated in the direction away from the user's mouth. Therefore, the difference in signal-to-noise ratio between microphone 0 and microphone 1 is maximized, thereby being used to reduce the noise in the main signal by the subsequent application of noise cancellation.

[0226] The process of reducing noise (unwanted sound signals) in the signal of interest (wanted sound signals) allows the combination of microphone 0 and microphone 1 to help enhance the user's voice when making a call in a noisy environment. It also helps improve the command and control performance of the system when used in a noisy environment. In a noisy environment, the user's voice is buried in the background noise and is difficult to be understood by the far-end listener and difficult to be recognized by the speech engine during the call. The combination of microphone 0 and microphone 1 uses beamforming technology to improve the signal-to-noise ratio (SNR) of the user's voice relative to the background noise (as well as increase the difference in SNR between microphone 0 and microphone 1), thereby improving the accuracy of voice activity detection for noise cancellation. This combination can provide useful performance gains even in extremely noisy environments with 90-dB or greater background noise amplitudes. As described above, microphone 0 and microphone 1 can be implemented using omnidirectional microphones.

[0227] Configuration 2

[0228] In one or more embodiments, when the user is listening to a remote sound source, such as 3330, and wearing the headset 3201 at the same time, microphone 1 and microphone 2 are used to process sound signals. In configuration 2, the signals output from microphone 1 and microphone 2 are beamformed so that the main sound signal response is in a forward position along axis 3304, thereby receiving the sound signal 3332 emitted by the sound source shown in 3330 with the maximum acoustic sensitivity directed to the direction of the sound source 3330 relative to the microphone pair microphone 1 and microphone 2. The signal obtained as such is used as the main signal input to the multi-channel noise cancellation system. A reference signal mainly containing noise can be obtained from microphone 2 with or without beamforming. When an omnidirectional microphone is used for microphone 1 and microphone 2, microphone 1 and microphone 2 are beamformed to obtain the main signal, while only microphone 2 is used for the reference signal, without beamforming of microphone 1, the sensitivity of the beamformed pair in the direction of source 3330 is increased by about 6dB relative to the sensitivity of microphone 2 alone to source 3330. This treatment provides a significant signal-to-noise ratio difference between microphone 1 and microphone 2 that is beneficial to noise cancellation performance. Axis 3304 points in a nominal direction in front of the user, but does not need to be precisely aligned therewith. Microphone 1 and microphone 2 have different acoustic distances to a sound source, such as 3330, located in front of the user. The acoustic distance between sound source 3330 and microphone 1 is smaller than the acoustic distance between microphone 2 and sound source 3330. Therefore, microphone 1 and microphone 2 can be flexibly arranged on the head-mounted device to provide different acoustic distances relative to the sound source located in front of the head-mounted device, without having to be directly pointed at sound source 3330.

[0229] In an alternative embodiment, beamforming of microphone pair 1 and microphone 2 with a primary response turned 180 degrees away from the sound source 3330 can be used to provide a reference signal (primarily an undesired sound signal). Note that the reference signal obtained in combination with it preferably has the least amount of desired sound signals. Reference signals can be obtained according to two methods, compared, and then a selection can be made based on the best system performance. Therefore, the reference signal obtained by either method has a signal-to-noise ratio that is less than the signal-to-noise ratio of the primary signal. Therefore, the difference in signal-to-noise ratio of the microphone 1 / microphone 2 pair related to the signal of interest originating from the front of the head-mounted device 3201, such as 3330 / 3332, is obtained. The signal obtained by either method away from the sound source 3330 is used as a reference signal for the input multi-channel noise cancellation system. The beamforming for the reference signal is selected to provide minimum acoustic sensitivity to signals from the front of the user, such as source 3330 (desired sound signals), and maximize sensitivity to noise generated from directions other than source 3330. Thus, the difference in signal-to-noise ratio between microphone 1 and microphone 2 is maximized, thereby being used to reduce the noise in the primary signal through the subsequent application of noise cancellation.

[0230] The output of the noise cancellation system is then provided to a speaker 3350 to help the user hear the sound from the sound source 3330. The speaker 3350 is incorporated into one or both side frames of the glasses 3201. Thus, in various embodiments, the combination of microphone 1 and microphone 2 is used to enhance the user's hearing, for example, during some activities such as watching TV or talking to a person in front of the user wearing the glasses 3201. Some people with hearing difficulties cannot clearly understand sound signals, especially in noisy environments. Combination 2 applies beamforming technology to help the user focus on the sound signal of interest by spatially eliminating background noise.

[0231] Configuration 3

[0232] In one or more embodiments, when the user is listening to or interacting with a remote sound source, such as 3320 or 3340, from one side or the other, and is wearing the head mounted device 3201 at the same time, microphone 1 and microphone 3 are used to process sound signals. Alternatively, microphone 3 and microphone 2 are used to process signals for configuration 3, or microphone 3 and microphone 0 are used. The following description of configuration 3 is provided in the form of microphone 3 and microphone 1, and is not intended to be limiting. In configuration 3, the sound energy output from microphone 1 and microphone 3 is compared to determine which side of the user the loudest sound comes from. This information is useful because in a meeting, for example, people sit around a table and different people talk from time to time, resulting in different directions of arrival relative to the user wearing glasses 3201. In configuration 3, the signals output from the selected pair of microphones are processed so that the main sound signal response is in a position along axis 3306, which is in the nominal direction of the sound source, but does not need to be precisely aligned with it. One of the selected pair of microphones, for example, microphone 3 and microphone 0, microphone 3 and microphone 1, or microphone 3 and microphone 2, has a different acoustic distance to the sound source.

[0233] According to one method of operation, the primary microphone is the microphone from the microphone 1, microphone 3 pair that has the maximum acoustic energy output. The other microphone in the microphone 1, microphone 3 pair is then designated as the reference microphone. After determining which microphone outputs the maximum acoustic energy, alternating processing of the primary signal and the reference signal can be performed. For example, in one or more embodiments, beamforming is applied to the signals output by microphone 1 and microphone 3. In one example, the primary signal is obtained when the main response axis in the beamforming process is turned to the side (direction) of the maximum acoustic energy being measured. In this example, the reference signal is obtained by turning the main response axis in the beamforming process to the side opposite to the main response axis.

[0234] A variation on this process is to use beamforming to obtain the primary signal, i.e., the outputs of the beamformed microphones 1 and 3 are steered to the side of the one of microphones 1 and 3 where the maximum acoustic energy is measured, while the unbeamformed output of the microphone with lower acoustic energy is used for the reference signal.

[0235] Another variation of this process is to utilize beamforming to obtain a reference signal, i.e., the outputs of the beamformed microphones 1 and 3 are steered to the side of the one of microphones 1 and 3 where the least acoustic energy is measured, while the unbeamformed output of the microphone with the greatest acoustic energy is used for the primary signal.

[0236] In a non-limiting example, reference Fig.33, there is a hypothetical usage scenario when the sound source 3320 is louder than the sound source 3340. In one or more embodiments, the system is designed to select microphone 3 as the side to receive the primary signal. Receiving the primary signal can be accomplished by any of the methods directly described above, such as beamforming microphone 1 and microphone 3 while placing the primary response axis 3306 in the direction of the sound source 3320. Alternatively, the output of microphone 3 can be used as the primary signal without beamforming. A reference signal can be obtained by beamforming microphone 1 and microphone 3 while placing the primary response axis 3306 in the opposite direction of the sound source 3320. Alternatively, the output of microphone 1 can be used as the reference signal without beamforming.

[0237] In some embodiments, a system is implemented by the above methods to perform sorting, for example, beamforming to select a primary signal or a reference signal and using the unbeamformed output of the microphone for the primary signal or the reference signal. A performance metric is calculated for each method, such as the signal-to-noise ratio difference between the primary signal and the reference signal, and the method with the largest signal-to-noise ratio difference is the method for processing the signals from microphone 1 and microphone 3. The sorting by these methods can be performed at the beginning of signal processing, or the sorting can be performed continuously to monitor the performance metric, and then the method can be dynamically updated based on the development of the performance metric. Therefore, in the implementation of configuration 3, many different methods can be used. The output of the noise cancellation system is then provided to one or more speakers 3350 to help the user hear the sound from the sound source 3320. The speaker 3350 is incorporated into one or both side frames (temples) of the glasses 3201.

[0238] A similar process is achieved when sound source 3340 produces greater sound energy 3342 at microphone 1 relative to the sound energy level received at microphone 3. In this case, the system can use a beamforming process to adjust the direction of the main response axis of the microphone pair in the direction of sound source 3340.

[0239] The pair of microphones 1 and 3 helps the user pick up stronger sounds from around the user during a conversation, especially from the left and right sides, by comparing the sound energy picked up from microphones 1 and 3. During a group meeting or chat, the voice signal may come from different directions (right or left) of the user. Configuration 3 compares the sound signal energy on each of the two microphones to determine which side the sound signal is coming from, thereby helping the user focus on the person who is actively speaking during a conversation. The output of the noise cancellation system is then provided to the speaker 3350 to help the user hear the sound from the sound source 3320 or 3340. The speaker 3350 is incorporated into one or both side frames of the glasses 3201.

[0240] Configuring Switching and Scanning

[0241] In various embodiments, the system can be configured to switch between two, three, or more configurations. Scanning configurations or scanning different beams (or selected microphone pairs) formed by a microphone array incorporated into a head-mounted device can also be done automatically by signal processing (hardware or a combination of hardware and software) built into the head-mounted device. Thus, in some embodiments, a system is implemented that forms a beam (or processes a selected microphone pair) by scanning in multiple directions relative to the user, and provides assistance to the user using sound signals that have been received and improved by one or more of beamforming, noise cancellation, and / or volume adjustment before being presented to a local or remote user.

[0242] For example, when watching TV and talking on the phone at the same time, the system can be configured to switch between configuration 1 (making a phone call) and configuration 2 (watching TV). The metric for switching to configuration 1 (phone function) can be related to detecting a change in acoustic energy on microphone 0.

[0243] Another example of a configuration switch could be switching from configuration 3 to configuration 2 during a conversation. For example, in a meeting, the person sitting to the right of the user wearing glasses 3201 begins to speak. Such a geometry is represented by source 3320 outputting acoustic energy 3322 and the output of microphone 3, where the output of microphone 3 is greater than the output of microphone 1. At this point, the system is operating in configuration 3. As the user listens and realizes that the speaker is to the right, the user may turn his or her head to the right to face the speaker. Now facing speaker 3320, the difference between the acoustic energy received at microphone 1 and microphone 3 is reduced, while the acoustic energy at microphone 1 is increased. In this case, the system switches to configuration 2 described above.

[0244] In one mode of operation, the user does not have to turn his head from side to side to face the speaker in a meeting. As the person actively speaking changes from one position to another, for example, from position 3320 (right side relative to glasses 3201) to position 3340 (left side relative to glasses 3201) to position 3330 (front of glasses 3201) to position 3380 (back of glasses 3201). The system will switch between microphone pairs and directions to select the primary microphone (either separate or beamformed output) in the direction of the speaker and the reference microphone (separate or beamformed output) in the direction of the noise (primarily non-desired sound signals).

[0245] Therefore, an embodiment of the present invention is implemented by a system that switches between configurations 1, 2 and 3 (or any subset thereof), and the switching of configurations 1, 2 and 3 (or any subset thereof) can be operated by mechanical switching, audio switching, or by intelligent design, and the intelligent design can be operated by analyzing one or more performance metrics, such as but not limited to the maximum signal-to-noise ratio difference, the maximum acoustic energy output from a microphone or beamforming output, etc.

[0246] Three configurations using three or four microphones have been described above in conjunction with the accompanying drawings. Note that more than four microphones can be used with the head-mounted device to provide approximately n number of directions (axes) and possible configurations for processing acoustic signals. Likewise, beamforming can be performed using more than two microphones.

[0247] Fig.34 Processing sound signals from a microphone array configured with a head mounted device is generally shown at 3400 according to an embodiment of the present invention. Fig.34 , the process begins at block 3402. At block 3404, a microphone that is part of a microphone array attached to a head-mounted device is scanned. The scan includes analyzing the sound signal from the microphone to obtain the signal amplitude level and other parameters in some cases. At block 3406, a configuration is selected based on the scan from block 3404. In some embodiments, the selection logic is used to select between configurations available using a given microphone array. At block 3408, the sound signal from the configuration selected at block 3406 is processed to improve the sound signal. Improving the sound signal may include inputting the sound signal into a noise cancellation module to remove the bottom sound signal from the main channel. Improving the sound signal may include amplifying the sound signal and presenting the amplified sound signal to the user of the head-mounted device on a speaker combined with the head-mounted device. The process stops at block 3412.

[0248] For the purpose of discussing and understanding the embodiments of the present invention, it should be understood that various terms are used by those skilled in the art to describe technology and methods. In addition, in the process of description, for the purpose of explanation, many specific details are set forth, so as to provide a thorough understanding of the embodiments of the present invention. However, for those of ordinary skill in the art, it is obvious that the embodiments can be implemented without these specific details. In some instances, known structures and devices are shown in block diagram form, rather than being shown in detail, to avoid confusing the present invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention, and it should be understood that other embodiments can be utilized and can be changed logically, mechanically, electrically and other without departing from the scope of the present invention.

[0249] Some parts of the description can be represented by algorithms and symbolic representations of operations on data bits in, for example, computer memories. These algorithmic descriptions and representations are the means by which those of ordinary skill in the field of data processing most effectively convey the essence of their work to other persons of ordinary skill in the art. Algorithms herein generally refer to self-consistent sequences of behaviors that result in desired results. These behaviors require physical operations on physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated. It has been shown that it is often convenient to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, waveforms, data, time series, etc., mainly for general reasons.

[0250] It should be borne in mind, however, that all of these and similar terms will be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise noted, it will be apparent from the discussion that throughout the description, discussions using terms such as "processing" or "computing" or "calculating" or "determining" or "displaying" can refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities in computer system registers and memories and converts it to other similar data represented as physical (electronic) quantities in computer system memories or registers or other such information storage, transmission or display devices.

[0251] The invention may be implemented by a device for performing the operations described herein. This device may be specifically constructed for the desired purpose, or it may include a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. The computer program may be stored on a computer-readable storage medium, such as, but not limited to, any type of disk, including floppy disks, hard disks, optical disks, compact disk read-only memories (CD-ROMs), and magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), electrically programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), flash memory, magnetic or optical cards, etc., or any type of medium suitable for storing local or remote electronic instructions for a computer.

[0252] The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various general-purpose systems may be used with the programs as described herein, or it may prove convenient to construct more specialized devices to perform the desired methods. For example, any method according to the present invention may be implemented by a hard-wired circuit obtained by programming a general-purpose processor, or may be implemented by any combination of hardware and software. Those skilled in the art will appreciate that the present invention may be implemented by other computer system configurations other than those described, including: handheld devices, multi-processor systems, microprocessor-based or programmable consumer electronics, digital signal processing (DSP) devices, networked personal computers, minicomputers, mainframe computers, etc. These embodiments may also be implemented in a distributed computing environment, where tasks are performed by remote processing devices linked through a communication network. In other examples, a system on a chip (SOC), a Bluetooth chip, a digital signal processing (DSP) chip, a codec with an integrated circuit (ICs), or other hardware and software implementations may be used to implement the above. Figures 1 to 31 The embodiments of the present invention are described in .

[0253] The method of the present invention can be implemented using computer software. If written in a programming language that meets recognized standards, a sequence of instructions designed to implement these methods can be compiled to execute on various hardware platforms and interface with various operating systems. In addition, the present invention is not described with reference to any specific programming language. It should be understood that various programming languages ​​can be used to implement the embodiments described herein. In addition, software in one form or another (e.g., program, flow, application, driver ...) is often said to take a certain action or cause a certain result in the art. These expressions are merely a short statement that the computer executes software so that the processor of the computer performs an action or produces a result.

[0254] It should be understood that those skilled in the art use various terms and techniques to describe communications, protocols, applications, implementations, mechanisms, etc. A similar technique is to describe the implementation of a technique using an algorithm or mathematical expression. That is, although the technique can be implemented, for example, as a code on an executing computer, the expression of the technique can be more appropriately and concisely conveyed or communicated as a formula, algorithm, mathematical expression, block diagram, or flow chart. Therefore, one of ordinary skill in the art will recognize that A+B=C is represented as a block of an addition function, and its implementation in hardware and / or software will take two inputs (A and B) and produce a summed output (C). Therefore, the use of a formula, algorithm, or mathematical expression as a description should be understood to have a physical embodiment at least in hardware and / or software (e.g., a computer system, in which the technique of the present invention can be implemented and implemented as an embodiment).

[0255] A non-transitory machine-readable medium is understood to include any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, machine-readable media, synonymously referred to as computer-readable media, include read-only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; electrical, optical, acoustic or other forms of information transmission other than by propagation signals (e.g., carrier waves, infrared signals, digital signals, etc.); and the like.

[0256] As used in this specification, "one embodiment" or "an embodiment" or similar phrases mean that the described features are included in at least one embodiment of the present invention. The "one embodiment" mentioned in this description does not necessarily refer to the same embodiment, but these embodiments are not mutually exclusive. "One embodiment" also does not mean that the present invention has only one embodiment. For example, features, structures, actions, etc. described in "one embodiment" may also be included in other embodiments. Therefore, the present invention may include various combinations and / or integrations of the embodiments described herein.

[0257] Thus, embodiments of the present invention may be used to reduce or eliminate unwanted sound signals from an acoustic system that processes and transmits the sound signals. Some non-limiting examples of the system may be, but are not limited to, use in short arm headsets, such as telephone audio headsets suitable for enterprise call centers, industrial and general mobile use, in-line "earbud" headsets with input lines (wires, cables or other connectors) mounted on or within the eyeglass frame, near-to-eye (NTE) headset displays or headset computing devices, long arm headsets for high noise environments such as industrial, military and aviation applications, and gooseneck desktop microphones that can be used to provide theater or symphony hall type sound quality without structural cost. Other embodiments of the present invention are easily implemented in head-mounted devices with general configurations, such as but not limited to glasses, goggles, sun hats, headbands, helmets, etc.

[0258] Although the present invention has been described in terms of several embodiments, those skilled in the art will recognize that the present invention is not limited to the described embodiments, but can be implemented by modification and alteration within the spirit and scope of the appended claims. Therefore, this description should be regarded as illustrative rather than restrictive.

Claims

1. A device worn on a user's head, characterized in that: include: Head mounted device; An array having at least three microphones, the at least three microphones being arranged along a plurality of at least two non-parallel axes; selection logic configured to identify a selected axis from the plurality of non-parallel axes and to identify two microphones from the array forming the selected axis; as well as A beamformer is configured to receive signals from the two microphones as input and output a main microphone channel and a reference microphone channel.

2. The device according to claim 1, characterized in that The selection logic identifies the selected axis using a metric.

3. The device according to claim 2, characterized in that The metric includes a microphone that receives a maximum sound pressure level and / or a minimum sound pressure level.

4. The device according to claim 2, characterized in that The selection logic is configured to monitor the metric and select a new selected axis from the plurality of non-parallel axes based on a new value of the metric.

5. The device according to claim 2, characterized in that The metric is the maximum signal-to-noise ratio difference of the two microphones.

6. The device according to claim 1, characterized in that Also includes: A switch selects the selected axis based on a state of the switch.

7. The device according to claim 1, characterized in that The main microphone channel and the reference microphone channel are input to a two-stage noise cancellation module.

8. The device according to claim 1, characterized in that Also includes: A speaker is connected to the head mounted device and is configured to provide a signal audible to the user.

9. The device according to claim 8, characterized in that When the selected axis is not pointed toward the user's mouth, the primary microphone channel and the reference microphone channel are used to create a signal that is input to the speaker.

10. The device according to claim 1, characterized in that The first microphone, the second microphone, and the third microphone of the array are located on a first temple of the head-mounted device; the first microphone and the second microphone are located on an inner surface of the first temple.

11. The device according to claim 10, characterized in that The array further comprises: A fourth microphone, wherein the fourth microphone is located on the second temple of the head-mounted device and the first microphone and the fourth microphone form a third axis, the second microphone and the fourth microphone form a fourth axis and the third axis is different from the fourth axis, and the selection logic selects an active direction from at least one of the first axis, the second axis and the third axis.

12. A device worn on a user's head, characterized in that: include: A head mounted device, the head mounted device further comprising: an array of three microphones connected to the head-mounted device, wherein a first microphone and a second microphone of the array define a first axis; and the second microphone and a third microphone define a second axis; and a speaker connected to the head mounted device and configured to provide a signal audible to the user; and a selection logic that selects an active direction from among the first axis and the second axis when the active direction is the first axis; a. the outputs from the first microphone and the second microphone are processed for transmission from the head mounted device; When the active direction is the second axis; b. The outputs from the second microphone and the third microphone will be processed to be used as input to the speaker.

13. The device according to claim 12, characterized in that The first microphone, the second microphone and the third microphone are located on the first temple of the head-mounted device; the first microphone and the second microphone are located on the inner surface of the first temple, and the third microphone is located on the bottom surface of the first temple.

14. The device according to claim 13, characterized in that The array further comprises: A fourth microphone, wherein the fourth microphone is located on the second temple of the head-mounted device and the first microphone and the fourth microphone form a third axis, the second microphone and the fourth microphone form a fourth axis and the third axis is different from the fourth axis, and the selection logic selects an active direction from at least one of the first axis, the second axis and the third axis.

15. A method for selecting sound signals received on a device worn on a user's head, characterized in that include: comparing sound signals from an array having at least three microphones, wherein positions of the at least three microphones define three non-parallel axes; selecting a first microphone pair from the array, wherein the first microphone pair includes a first microphone and a second microphone; forming a primary microphone signal from the first microphone pair; and A reference microphone signal is formed from the first microphone pair; wherein the primary microphone signal and the reference microphone signal are input into a noise cancellation module to reduce noise from the primary microphone signal.

16. The method according to claim 15, characterized in that The comparison forms at least three microphone pairs from the array, identifies a potential primary microphone and a potential reference microphone from each microphone pair, calculates a signal-to-noise ratio difference for each microphone pair, and the first microphone pair is the microphone pair with the largest SNR difference; during the comparison of the sound signals, beamforming is performed on the microphone pairs.

17. The method according to claim 15, characterized in that The forming of the main microphone signal is achieved by beamforming the first microphone pair, and the forming of the reference microphone signal is not achieved by beamforming the first microphone pair.

18. The method according to claim 15, characterized in that The forming of the main microphone signal is not achieved by beamforming the first microphone pair, and the forming of the reference microphone signal is achieved by beamforming the first microphone pair.

Images