Head-worn device based on three-microphone directional sound recording

Abstract

The present invention discloses a head-mounted device based on three-microphone directional sound collection, including a front frame provided with three microphone modules and a signal synthesis module. When the head-mounted device is worn on the user's head, two of the microphone modules are located on the same horizontal plane, and the other microphone module is located above or below to form an isosceles triangle, and the vertex of the isosceles triangle points to the user's mouth. The signal synthesis module is used to obtain and process the sound signals generated by the three microphone modules, enhance the sound from the direction of the mouth, and obtain a first sound signal. The first sound signal is used to process with a second sound signal from a specific area in front of the user to enhance the second sound signal. The present invention can effectively separate and enhance the voices of the user and the speaker opposite, and is suitable for people with hearing impairments or real-time language translation.

Description

Technical Field

[0001] The present invention relates to a directional sound receiving device, in particular to a directional sound receiving device suitable for being worn on the head, such as glasses with sound receiving / playing functions, which can provide directional sound recording for playback or voice recognition for hearing-impaired people. Background Art

[0002] Head-worn devices with recording functions are not uncommon. In order to facilitate carrying, the prior art integrates recording elements into various portable items, such as watches, mobile phones, headphones, helmets, glasses, etc. Among them, the solution of integrating recording elements into glasses has good portability and controllability of recording direction. For example, in the US patent with the publication number US7792552B2, a pair of glasses for wireless communication is proposed. Figure 1 As shown, it includes a front frame 1 and two bendable arms located on both sides of the front frame 1, namely a left arm 2 and a right arm 3. The left arm 2 and the right arm 3 are folded together with the front frame 1 after being bent. The front frame is used to hold two lenses. A microphone 4 is provided in the middle of the two lenses of the front frame 1, and the microphone 4 is used for recording. However, since the glasses only use one microphone, it is impossible to achieve directional recording of sound, and it is also impossible to effectively remove environmental noise, which makes the recording quality low, and it is not convenient to process the sound to improve the sound quality, nor is it conducive to subsequent advanced operations (such as voice recognition, etc.).

[0003] In order to enhance the quality of recording and improve the directionality, a Chinese patent application with publication number CN114339524A proposes a head-mounted device, wherein three or four microphones are arranged on the front frame, respectively. Figure 2 and Figure 3 When three microphones are used, two microphones form a call microphone group for noise reduction processing. When four microphones are used, left and right stereo recording mode can be achieved.

[0004] However, the above-mentioned existing glasses or head-mounted device solutions cannot achieve directional sound reception, especially the directional sound recording of a specific speaker, and therefore are not suitable for occasions where directional sound reception is required. For example, for people with hearing impairments, their main requirement is to be able to hear and clearly hear the voice of the speaker on the other side without having to reproduce their own voice, and at the same time, the impact of environmental noise on the voice should be minimized as much as possible. For another example, in occasions where real-time voice translation is required, it is necessary to record the voice of the speaker on the other side in real time with high quality. However, the existing glasses or head-mounted devices with a sound reception function still cannot achieve this. Therefore, if a head-mounted device that can meet the needs of hearing-impaired people or real-time translation and its corresponding sound reception method can be realized, the glasses or head-mounted devices with a sound reception function will provide great convenience and practicality. Summary of the invention

[0005] The present invention aims to solve the problem that the existing head-mounted device with a sound receiving function cannot meet the needs of hearing-impaired people or real-time voice translation, and cannot achieve directional sound receiving or sound receiving of a specific speaker.

[0006] In order to solve the above technical problems, the present invention proposes a head-mounted device based on three-microphone directional sound collection, including a front frame. When the head-mounted device is worn on the user's head, the front frame faces the front of the user's face. A first microphone module, a second microphone module and a third microphone module are arranged in the front frame. When the head-mounted device is worn on the user's head, the second microphone module and the third microphone module are located in the same horizontal plane, the first microphone module is located above or below the line connecting the second microphone module and the third microphone module, and the three microphone modules constitute an isosceles triangle with the first microphone module as the vertex angle, and the vertex angle of the isosceles triangle points to the user's mouth; the head-mounted device also includes a signal synthesis module, which is used to obtain the sound signals generated by the first microphone module, the second microphone module and the third microphone module, and process the sound signals generated by the first microphone module, the second microphone module and the third microphone module to enhance the sound from the mouth direction to obtain a first sound signal.

[0007] According to a preferred embodiment of the present invention, the signal synthesis module is further used to process the sound signals generated by the second microphone module and the third microphone module to enhance the sound from a specific area in front of the user's face to obtain a second sound signal.

[0008] According to a preferred embodiment of the present invention, the signal synthesis module is further used to process the first sound signal and the second sound signal to remove the first sound part in the second sound signal as an output signal.

[0009] According to a preferred embodiment of the present invention, the sound signals generated by the second microphone module and the third microphone module are processed to enhance the sound from a specific area in front of the user's face, and the step of obtaining the second sound signal includes: adding the sound signals generated by the second microphone module and the third microphone module and dividing them by two to obtain a signal d(n), and subtracting them and dividing them by two to obtain a signal x(n), where n represents a time sequence number; processing the signal x(n) through an adaptive filter to obtain a signal y(n); subtracting the signal y(n) from the signal d(n) to obtain an enhanced output signal e(n) as the second sound signal, and simultaneously inputting the signal e(n) into a parameter update module of the adaptive filter to update the adaptive filter parameters.

[0010] According to a preferred embodiment of the present invention, the head-mounted device is glasses, which include two temples located on both sides of a front frame, wherein the front frame is used to fix two lenses, and a first microphone module, a second microphone module and a third microphone module are located between the two lenses, and the signal synthesis module is located in the front frame or the temples.

[0011] According to a preferred embodiment of the present invention, the head-mounted device further comprises a sound playing module, which is used to play sound to the user according to the output signal.

[0012] According to a preferred embodiment of the present invention, the sound playing module is located at a portion of the glasses leg close to the ear of the user, and is electrically connected to the signal synthesis module to receive the output signal.

[0013] According to a preferred embodiment of the present invention, the head-mounted device further comprises a speech recognition module, which is used to perform speech recognition on the output signal to convert the sound signal into text information.

[0014] According to a preferred embodiment of the present invention, the head-wearing device further comprises an information display module, which is used to display the text information.

[0015] According to a preferred embodiment of the present invention, the speech recognition module is further used to convert the recognized text information into text information in a language specified by the user, and send it to the information display module for display.

[0016] According to a preferred embodiment of the present invention, the information display module is a retinal projection device, which is used to project the text information onto the user's retina.

[0017] According to a preferred embodiment of the present invention, the speech recognition module is also used to convert the recognized text information into text information in a user-specified language, and then convert the converted text information into a sound signal, which is sent to the sound playing module for playing.

[0018] According to a preferred embodiment of the present invention, the first microphone module and the second microphone module constitute a first channel, and the first microphone module and the third microphone module constitute a second channel;

[0019] The signal synthesis module is used to: obtain the sound signals generated by the first microphone module, the second microphone module and the third microphone module; respectively calculate the delayed subtraction beamforming signals of the first channel and the second channel, which include the first channel forward forming signal, the first channel backward forming signal, the second channel forward forming signal and the second channel backward forming signal; divide each of the delayed subtraction beamforming signals into a predetermined number of spectrum subband signals, and calculate the signal energy E of each same spectrum subband of the forward forming signal and the backward forming signal of the same channel fij and E bij , f represents forward, b represents backward, i is the channel number, and j is the spectrum subband number; calculate the relative energy statistical parameter according to the energy of the spectrum subband signals of the forward forming signal and the backward forming signal; assign the first gain G to each spectrum subband signal of the first channel forward forming signal and the second channel forward forming signal according to the relative energy statistical parameter 1ij , where i is the channel number and j is the spectrum subband number; calculate the second gain G when the first channel forward forming signal and the second channel forward forming signal decay to the steady-state noise energy level 2ij , where i is the channel number and j is the spectrum subband number; for each spectrum subband signal of the first channel forward formed signal and the second channel forward formed signal, compare the first gain G 1ij and the second gain G 2ij The size of the channel and the maximum value of the two is taken as the final gain G of the spectrum sub-band signal. ij ; Calculate the energy value E' after noise reduction of each spectrum subband signal of the first channel forward forming signal and the second channel forward forming signal fij =E fij *G ij , and compare the energy values ​​E' of the spectrum sub-band signals of the same spectrum sub-band of the two forward forming signals f1j and E' f2j , the final gain G of the spectrum subband j j The gain of the channel with the smallest energy value is taken; after smoothing the gain value of the spectrum sub-band, the final complex spectrum of the spectrum sub-band is calculated, and then the inverse Fourier transform is performed to obtain the time domain signal, that is, the first sound signal.

[0020] According to a preferred embodiment of the present invention, a first-order delay subtraction beamforming algorithm is used to calculate the delay subtraction beamforming signal.

[0021] According to a preferred embodiment of the present invention, the method for calculating the relative energy statistical parameter value is: initialize the relative energy statistical parameter value to 0, compare the energy of each identical spectral sub-band signal of the forward forming signal and the backward forming signal one by one, when the energy of the spectral sub-band signal of the forward forming signal is larger, the relative energy statistical parameter value is increased by 1, otherwise it is decreased by 1.

[0022] According to a preferred embodiment of the present invention,

[0023] The signal synthesis module is also used to perform steady-state noise reduction processing on the time domain signal obtained by Fourier transform.

[0024] The present invention can effectively separate and enhance the voice of the user and the opposite speaker, and is suitable for hearing-impaired people or for real-time language translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to make the technical problems solved by the present invention, the technical means adopted and the technical effects achieved more clearly, the specific embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, it should be noted that the drawings described below are only drawings of exemplary embodiments of the present invention, and those skilled in the art can obtain drawings of other embodiments based on these drawings without creative work.

[0026] Figure 1 The present invention is a schematic structural diagram of a wireless communication glasses in the prior art.

[0027] Figure 2 It is a structural schematic diagram of a head-mounted device provided with three microphones in the prior art.

[0028] Figure 3 It is a structural schematic diagram of a head-mounted device provided with four microphones in the prior art.

[0029] Figure 4 It is a schematic diagram of the NLMS algorithm used to obtain the second sound signal in one embodiment of the present invention.

[0030] Figure 5 1 is a front view of a head-mounted device with a sound recording function according to an embodiment of the present invention.

[0031] Figure 6 The diagram is a schematic diagram of a head-mounted device with a sound recording function according to an embodiment of the present invention in a usage state when the head-mounted device is worn on a user's head.

[0032] Figure 7 1 is a side view of a head-mounted device with a sound recording function according to an embodiment of the present invention.

[0033] Figure 8 FIG. 1 is a layout diagram of three microphones for collecting sound of a head-mounted device with a sound collecting function according to an embodiment of the present invention.

[0034] Fig. 9 It is a schematic diagram of delay subtraction beamforming.

[0035] Fig.10It is the directivity diagram of the delayed subtraction beamforming signal of the two microphones.

[0036] Fig.11 It is a module structure diagram of the head-worn device with sound recording function of the present invention. DETAILED DESCRIPTION

[0037] The exemplary embodiments of the present invention will now be described more fully with reference to the accompanying drawings. Although each exemplary embodiment can be implemented in a variety of specific ways, it should not be understood that the present invention is limited to the embodiments set forth herein. On the contrary, these exemplary embodiments are provided to make the content of the present invention more complete and to facilitate the comprehensive communication of the inventive concept to those skilled in the art.

[0038] Under the premise of conforming to the technical concept of the present invention, the structure, performance, effect or other characteristics described in a specific embodiment may be combined with one or more other embodiments in any appropriate manner.

[0039] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0040] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to actual conditions without departing from the main purpose of the present invention.

[0041] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0042] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0043] The present invention proposes a head-mounted device for directional sound collection based on three microphones. The head-mounted device referred to in the present invention refers to any device that can be worn on the human head, and is not limited to the type of device. However, in order to implement the sound collection method of the present invention, the head-mounted device in the present invention must have a frame for setting the microphone. When the head-mounted device is worn on the user's head, the frame faces the front of the user's face, so it is also referred to as a front frame. In this way, when the user wears the device, the microphone is located on one side of the user's face. Therefore, typical examples of the head-mounted device in the present invention include frame glasses, VR glasses, protective shields, helmets, headbands, and hats and headphones with a portion located on the front side of the head.

[0044] According to the present invention, three microphone modules are arranged in the front frame, and the so-called microphone module refers to a microphone functional module having a sound receiving element and a signal preprocessing element. Here, the three microphones are respectively referred to as a first microphone module, a second microphone module and a third microphone module. When the head-mounted device is worn on the user's head, two of the microphone modules are located in the same horizontal plane. Here, we refer to the microphone modules located in the same horizontal plane as the second microphone module and the third microphone module, and the other microphone not located in the horizontal plane as the first microphone. The first microphone module is located above or below the line connecting the second microphone module and the third microphone module, whereby the three microphone modules constitute an isosceles triangle with the first microphone module as the vertex.

[0045] According to the present invention, the vertex of the isosceles triangle points to the user's mouth. That is, one vertex of the triangle formed by the three microphone modules points to the part where the user makes his own sound, which is specially set to cooperate with the sound processing algorithm of the present invention. According to the sound processing algorithm of the present invention, the sound recorded in the direction pointed by the vertex of the isosceles triangle can be enhanced, and at the same time, the sound recorded in the direction opposite to the vertex can be weakened.

[0046] The first microphone module, the second microphone module and the third microphone module are used to sense sound to generate sound signals. In a special case, the distances between any two of the microphone modules are equal.

[0047] According to the present invention, the head-mounted device further includes a signal synthesis module, which is used to obtain the sound signals recorded by the first microphone module, the second microphone module and the third microphone module, and process the sound signals recorded by the first microphone module, the second microphone module and the third microphone module to enhance the sound from the mouth direction of the user to obtain a first sound signal. The processed first sound signal eliminates or weakens the noise in the original sound signal, so it can also be called an enhanced sound signal. It can be seen that the present invention can clearly record the voice of the wearer (i.e. the user himself) wearing the head-mounted device of the present invention, and the surrounding or other people's talking voices will be weakened. Therefore, for glasses, helmets, etc. with recording functions, the present invention can improve the sound recording quality and improve the user experience. The signal synthesis module is formed by components with signal processing functions, such as single-chip microcomputers, DSPs, FPGAs, etc.

[0048] More preferably, the signal synthesis module of the present invention is further used to process the sound signals generated by the second microphone module and the third microphone module to enhance the sound from the specific area in front of the user's face to obtain a second sound signal. Generally, the classic NLMS (normalized least mean square adaptive filter) algorithm can be used for speech enhancement.

[0049] Figure 4 A schematic diagram of an adaptive filtering algorithm used to obtain a second sound signal in an embodiment of the present invention. Figure 4 As shown, the sound signals (time domain sampling signals) obtained by the second microphone module and the third microphone module are added and divided by two to obtain the signal d(n), and the sound signals (time domain sampling signals) obtained by the second microphone module and the third microphone module are subtracted and divided by two to obtain the signal x(n). The signal x(n) is processed by an M-order adaptive filter to obtain the signal y(n). The signal d(n) is subtracted from the signal y(n) to obtain the enhanced output signal e(n) (the second sound signal of the signal synthesis module), and the signal e(n) is input to the parameter update module of the adaptive filter to update the parameters of the M-order adaptive filter. The n in the figure represents the time sequence number, and z -1 represents the delay unit, M is the order of the filter, W 1 , W 2 ,…W M They represent the parameters of the M-order filter, x(n), x(n-1), x(n-2), ..., x(n-M+1) respectively. The M delayed signals of the signal x(n) are denoted by x 1(n), x 2 (n), ..., x M (n) In this embodiment, the adaptive filter may be an NLMS (normalized least mean square error) filter, and the algorithm adopted by the parameter updating module of the adaptive filter is the NLMS algorithm.

[0050] As an implementation mode, the signal synthesis module is further used to process the first sound signal and the second sound signal to remove the first sound part in the second sound signal as an output signal. Since the first sound signal only contains the user's own voice, as another implementation mode, a speech detection algorithm based on signal energy value can be used for the first sound signal, and the speech detection result can be applied to the second sound signal. When the user speaks, the second sound signal outputs silence, and when the user does not speak, the second sound signal outputs the voice of the opposite speaker.

[0051] Figure 5 FIG. 1 is a front view of a head-mounted device with a sound recording function according to a specific embodiment of the present invention. Figure 5 As shown, the head-mounted device of this embodiment is glasses, which includes two glasses legs located on both sides of a front frame 3, a first glasses leg 1 and a second glasses leg 2, wherein the front frame 3 is used to fix two lenses, and a first microphone module mic1, a second microphone module mic2 and a third microphone module mic3 located between the two lenses. The signal synthesis module 5 in this embodiment is located in the second glasses leg 2 ( Figure 5 not shown).

[0052] Figure 6 FIG. 1 is a schematic diagram of a head-mounted device according to an embodiment of the present invention worn on a user's head. Figure 6 As shown, when the glasses as a head-mounted device are worn on the user's head, the three microphones are located in the middle of the front frame 3 of the glasses, and thus are located just above the bridge of the nose. The second microphone module mic2 and the third microphone module mic3 are located in the same horizontal plane, that is, the line connecting the second microphone module mic2 and the third microphone module mic3 is a horizontal line. The first microphone module mic1 is located below the line connecting the second microphone module mic2 and the third microphone module mic3, and the three microphone modules form an isosceles triangle with the first microphone module as the vertex, so that the vertex of the isosceles triangle just points to the user's mouth.

[0053] In the glasses of this embodiment, a sound playback function is also added to the glasses, that is, sound is played to the user. Therefore, the head-worn device of this embodiment of the present invention also includes a sound playback module. In principle, the sound source played can be any sound source, but the present invention preferably receives the output signal output after processing the first sound signal and the second sound signal as the sound source signal, because this situation is particularly useful for people with hearing impairments. As mentioned above, the microphone module of the present invention can strengthen the voice of the user and the opposite speaker after being combined with the signal synthesis module, thereby further processing the sound signal that is later weakened or eliminated by the user's own voice. In this way, when the sound playback element of the present invention adopts a sound playback element with a hearing aid function, the signal of the user's opposite speaker's voice can be strengthened in real time and sent to the user's ear, helping the hearing-impaired user to hear and hear the opposite speaker's voice clearly.

[0054] Figure 7 FIG. 1 is a side view of glasses as a head-worn device according to an embodiment of the present invention. Figure 7 As shown, the signal synthesis module 4 is arranged in the second eyeglass leg 2, and the sound playing module 6 is arranged on the two eyeglass legs and fixed thereto, and the sound playing module 6 is located at the part of the eyeglass leg close to the ear of the user, which can be, for example, a bone conduction hearing aid element, and is electrically connected to the signal synthesis module 5 to receive the output signal. It should be understood that the present invention is not limited to the type of specific sound playing device, and any sound playing element suitable for playing the processed output signal may be used to implement the solution of the present invention and achieve corresponding beneficial effects.

[0055] Further Figure 7 As shown, the head-worn device of the present invention preferably further includes a voice recognition module 7, which is used to perform voice recognition on the output signal to convert the sound signal into text information. In particular, for occasions such as frame glasses, VR glasses, and helmets that are suitable for placing display elements in front of the user's eyes, a preferred solution of the present invention is to perform voice recognition on the output signal, and the text information after voice recognition can be used in a variety of application scenarios.

[0056] One application scenario is still for people with hearing impairments. For users with complete or almost complete hearing loss, converting the voice of the person opposite the user into text information and displaying it is a solution that must be adopted. It is also the most intuitive way to better understand what the person opposite is saying in real time. In this case, the glasses in the above embodiment also include an information display module, which is used to display the text information. Figure 7In one embodiment shown, the information display module includes a light source 8, a projection element 9 and a special lens for displaying the projection of the projection element 9. In this way, the information display module is a retinal projection device composed of the source 8, the projection element 9 and the lens, which can project the text information onto the user's retina. After the user wears the glasses, the words spoken by the opposite speaker converted by the voice recognition module 7 will be displayed in real time in front of the user. In this embodiment, the projection element 9 receives the converted voice and text information from the voice recognition module 7.

[0057] It should be understood that the present invention is not limited to the type of information display module, and any existing element suitable for displaying images in front of the user can be adopted.

[0058] Another application scenario is real-time translation. At this time, the speech recognition module is also used to convert the recognized text information into text information in the language specified by the user, and send it to the information display module for display. If the user and the speaker on the other side do not use the same language, the present invention can convert the recognized words of the speaker on the other side into text in the language specified by the user. For example, after the user puts on the glasses of the present invention, the foreign language, dialect, etc. spoken by the speaker on the other side can be displayed in real time in front of the user, converted into the language and text familiar to the user. In addition to displaying the converted language and text in front of the user through retinal projection, it can also be processed by text-to-speech and then sent to a sound playback module for the user to listen to. The text-to-speech function can be implemented in the speech recognition module, or it can be implemented by a separate module.

[0059] According to the present invention, the first microphone module and the second microphone module are referred to as the first microphone module group, and the first microphone module and the third microphone module are referred to as the second microphone module group. The angle between the two microphone modules can be varied, but is preferably 60 degrees, that is, the present invention preferably sets the first microphone module, the second microphone module and the third microphone module to be at equal distances from each other to achieve a better directional selection effect and noise reduction effect. The steps of obtaining and first sound signal are described in detail below.

[0060] The present invention first obtains the sound signals generated by the first microphone module, the second microphone module and the third microphone module, and then processes them by a beamforming method. Specifically, the present invention forms a first channel with the first microphone module and the second microphone module, the first microphone module and the second microphone module are called a first microphone module group, and the third microphone module and the first microphone module are called a first microphone module group to form a second channel, and respectively calculates the delayed subtraction beamforming signals of the first channel and the second channel, that is, including a first channel forward forming signal, a first channel backward forming signal, a second channel forward forming signal and a second channel backward forming signal.

[0061] Next, the present invention divides each of the delay subtraction beamforming signals into a predetermined number of spectrum subband signals, and calculates the signal energy E of each of the same spectrum subbands of the forward forming signal and the backward forming signal of the same channel. fij and E bij Wherein f represents forward, b represents backward, i represents the channel number, which is 1 or 2, j represents the spectrum subband number, which is 1 to n, and n represents the number of predetermined spectrum subbands, which is a natural number greater than or equal to 2. In the present invention, n is preferably 64 to 256, and more preferably 128.

[0062] The present invention innovatively proposes to compare and count the energies of the spectrum sub-band signals of the forward forming signal and the backward forming signal to obtain a relative energy statistical parameter, and assign a first gain G to each spectrum sub-band signal of the first channel forward forming signal and the second channel forward forming signal according to the relative energy statistical parameter. 1ij , where i is the channel number and j is the spectrum subband number. Assign the first gain G 1ij When the relative energy statistical parameter value is larger, the larger the gain coefficient is given, and the smaller the relative energy statistical parameter value is, the smaller the gain coefficient is given. The gain coefficient is between 0 and 1.

[0063] The relative energy statistical parameter in the present invention indicates whether the forward energy is greater or the backward energy is greater in the current period of time, or in other words, it indicates the relative size between the forward energy and the backward energy of the sound. According to the present invention, when the sound is in the front, the forward energy is greater than the backward energy, the statistical value becomes larger, the corresponding gain value is also large, and the sound is retained; when the sound is in the back, the backward energy is greater than the forward energy, the statistical value becomes smaller, the corresponding gain value also becomes smaller, and the sound is suppressed.

[0064] For each spectrum sub-band signal of the first channel forward forming signal and the second channel forward forming signal, the present invention compares the first gain G 1ij and the second gain G 2ij The size of the channel and the maximum value of the two is taken as the final gain G of the spectrum sub-band signal. ij ; The second gain is the gain when the spectrum subband signal decays to the steady-state noise energy level. That is, the present invention also needs to calculate the second gain G when the first channel forward forming signal and the second channel forward forming signal decay to the steady-state noise energy level. 2ij , which can be calculated by spectral subtraction with steady-state noise reduction for each spectral subband.

[0065] Then, the present invention calculates the energy value E' after noise reduction of each spectrum sub-band signal of the first channel forward formed signal and the second channel forward formed signal. fij =E fij *G ij, and compare the energy values ​​E' of the spectrum sub-band signals of the same spectrum sub-band of the two forward forming signals f1j and E' f2j , the final gain G of the spectrum subband j j Take the gain of the channel with the smaller energy value; take the sub-band spectrum with low energy in each spectrum sub-band of the forward signal in the two channels and combine it with the gain, and finally output a channel.

[0066] Finally, after smoothing the gain value of the spectrum sub-band, the present invention calculates the final complex spectrum of the spectrum sub-band, and then inverse Fourier transforms to obtain the time domain signal. Specifically, for each spectrum sub-band gain G j Smooth the previous and next frames. The gain after smoothing is G jn , the previous smoothing gain is G j(n-1) ,

[0067] If G jn >=G j(n-1) ,

[0068] G jn =a*G j(n-1) +(1-a)*G j ;

[0069] If G jn <G j(n-1) , G jn =b*G j(n-1) +(1-b)*G j .

[0070] Specific embodiments of the present invention are described below with reference to the accompanying drawings.

[0071] Figure 8 : is a layout diagram of the three-microphone module receiving sound according to an embodiment of the present invention. Figure 8 As shown, the first microphone module mic1, the second microphone module mic2 and the third microphone module mic3 are arranged equidistantly from each other. In this embodiment, the spacing between them is 20 mm. The first microphone module mic1, the second microphone module mic2 and the third microphone module mic3 can be integrated into one physical device, or they can be implemented in different physical devices respectively.

[0072] The above three microphone modules constitute a microphone module array, and the first microphone module mic1 and the second microphone module mic2, and the first microphone module mic1 and the third microphone module mic3 respectively constitute two first-order delay subtraction beamforming. We call the sound reception signal channel formed by the first microphone module mic1 and the second microphone module mic2 the first channel, and the sound reception signal channel formed by the first microphone module mic1 and the third microphone module mic3 the second channel. Using the delay τ=d / c, it can be seen that the sound reception direction diagrams of the first channel and the second channel are both heart-shaped. The sound reception direction of the combination of these three microphone modules is the overlapping part of the two heart-shaped diagrams. The beamforming algorithm has different gains for sounds coming from different directions. Beamforming can make good use of the spatial information of sound, so it has a better suppression effect on spatial noise.

[0073] First-order delay subtraction is a relatively simple and classic beamforming technology. The specific method is to use two microphones to collect voice, delay one input signal, and then subtract it from the other signal.

[0074] Fig. 9 It is a schematic diagram of delay subtraction beamforming. Fig. 9 As shown, let the original sound signal be y(t), and the two microphone modules input the sound signal s 1 (t), s 2 (t), the enhanced output sound signal x(t) is:

[0075] x(t)=s 1 (t)-s 2 (t-τ), τ=d / c.

[0076] Where τ is the delay value and d is the distance between microphone modules.

[0077] The input signals of the two-channel microphone module are:

[0078] s 1 (t) = y(t)

[0079] s 2 (t) = y(t-(d / c)·cosθ), where θ is the angle of sound coming from.

[0080] Substituting in, we get:

[0081] x(t)=y(t)-y(t-(d / c)·cosθ-τ)

[0082] For the convenience of analysis, It can be written as:

[0083]

[0084] Then the power spectrum output by the microphone module array is:

[0085]

[0086] When the distance between microphone modules is small, Taylor expansion is performed and the use available:

[0087]

[0088] It can be seen from the above formula that the directivity diagram of the microphone module array is related to the delay value τ.

[0089] When τ=d / c, the formula becomes:

[0090]

[0091] When θ = 0°, the gain is maximum; when θ = 180°, the gain is minimum. The directional diagram at this time is called a cardioid diagram, such as Fig.10 shown.

[0092] In this embodiment, the first microphone module mic1 and the second microphone module mic2, and the first microphone module mic1 and the third microphone module mic3 respectively form two first-order delay subtraction beamforming. Using the delay τ=d / c, the sound receiving direction patterns of the first microphone module mic1 and the second microphone module mic2, and the first microphone module mic1 and the third microphone module mic3 are all heart-shaped. The sound receiving direction of the combination of these three microphone modules is the overlapping part of the two heart-shaped patterns.

[0093] The present invention uses a first-order delay subtraction beamforming algorithm to respectively calculate the frequency band energy of the forward and backward delay subtraction beamforming signals of the first channel and the second channel.

[0094] In this embodiment, each acquired sound signal is divided into 128 spectrum sub-band signals. In other implementations, it can also be divided into other predetermined number of spectrum sub-band signals. Then, the signal energy E of each same spectrum sub-band of the forward forming signal and the backward forming signal of the same channel is calculated. fij and E bij Where E represents energy, f represents forward, b represents backward, i represents the channel number, which is 1 or 2, representing the first channel and the second channel respectively. j represents the spectrum subband number, which is 1 to 128.

[0095] A relative energy statistical parameter is calculated based on the energy of the spectrum sub-band signals of the forward forming signal and the backward forming signal, and a first gain G is assigned to each spectrum sub-band signal of the first channel forward forming signal and the second channel forward forming signal based on the relative energy statistical parameter. 1ij, where i is the channel number and j is the spectrum subband number. As mentioned above, the relative energy statistical parameter in the present invention indicates whether the forward energy is larger or the backward energy is larger in the current period of time, or in other words, it indicates the relative size between the forward energy and the backward energy of the sound. According to the present invention, when the sound is in the front, the forward energy is greater than the backward energy, the statistical value becomes larger, the corresponding gain value is also large, and the sound is retained; when the sound is in the back, the backward energy is greater than the forward energy, the statistical value becomes smaller, the corresponding gain value also becomes smaller, and the sound is suppressed.

[0096] For each spectrum sub-band signal of the first channel forward forming signal and the second channel forward forming signal, the present invention compares the first gain G 1ij and the second gain G 2ij The size of the channel and the maximum value of the two is taken as the final gain G of the spectrum sub-band signal. ij ; The second gain is the gain when the spectrum subband signal decays to the steady-state noise energy level. That is, the present invention also needs to calculate the second gain G when the first channel forward forming signal and the second channel forward forming signal decay to the steady-state noise energy level. 2ij .

[0097] Then, the present invention calculates the energy value E' after noise reduction of each spectrum sub-band signal of the first channel forward formed signal and the second channel forward formed signal. fij =E fij *G ij , and compare the energy values ​​E' of the spectrum sub-band signals of the same spectrum sub-band of the two forward forming signals f1j and E' f2j , the final gain G of the spectrum subband j j Take the gain of the channel with smaller energy value;

[0098] Finally, after smoothing the gain value of the spectrum sub-band, the present invention calculates the final complex spectrum of the spectrum sub-band, and then performs inverse Fourier transform to obtain a time domain signal.

[0099] Those skilled in the art will appreciate that all or part of the steps to implement the above embodiments are implemented as a program executed by a data processing device (including a computer), i.e., a computer program. When the computer program is executed, the above method provided by the present invention can be implemented. Moreover, the computer program can be stored in a computer-readable storage medium, which can be a readable storage medium such as a disk, an optical disk, a ROM, a RAM, or a storage array composed of multiple storage media, such as a disk or a tape storage array. The storage medium is not limited to centralized storage, and it can also be distributed storage, such as cloud storage based on cloud computing.

[0100] The following describes an apparatus embodiment of the present invention, which can be used to execute a method embodiment of the present invention. The details described in the apparatus embodiment of the present invention should be regarded as supplementary to the above method embodiment; details not disclosed in the apparatus embodiment of the present invention can be implemented with reference to the above method embodiment.

[0101] Fig.11 1 is a block diagram of a sound receiving device according to an embodiment of the present invention. Fig.11 As shown, the signal synthesis module obtains the sound signals generated by the first microphone module, the second microphone module and the third microphone module and processes the signals to generate a final processed sound signal as an output signal. The processed output signal can retain the voice of the speaker on the other side, eliminate and weaken the voice of one's own speech and environmental noise. The speech recognition module is connected to the signal synthesis module to obtain the output sound signal and then perform speech recognition. The text information obtained after recognition can be directly sent to the retinal projection module for display, or it can be sent to the sound playback module for playback after text-to-speech processing. An optional method is that the speech recognition module is also used to convert the recognized text information into text information in the user's finger language, and then send it to the retinal projection module or sound, or send it to the sound playback module for playback after text-to-speech processing.

[0102] Those skilled in the art will appreciate that the modules in the above embodiments may be distributed in the device as described, or may be changed accordingly and distributed in one or more devices different from the above embodiments. The modules in the above embodiments may be combined into one module, or may be further split into multiple submodules. For example, in the above embodiments, the text translation and text-to-speech functions of the speech recognition module may be implemented together, or may be implemented by its submodules, or by other modules independent of the speech recognition module.

[0103] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A head-mounted device based on three-microphone directional sound collection, comprising a front frame, when the head-mounted device is worn on the user's head, the front frame faces the front of the user's face, characterized in that: The front frame is provided with a first microphone module, a second microphone module and a third microphone module. When the head-mounted device is worn on the user's head, the second microphone module and the third microphone module are located in the same horizontal plane, the first microphone module is located above or below the line connecting the second microphone module and the third microphone module, and the three microphone modules form an isosceles triangle with the first microphone module as the vertex angle, and the vertex angle of the isosceles triangle points to the user's mouth; The head-mounted device further comprises a signal synthesis module, which is used for: Acquire sound signals generated by the first microphone module, the second microphone module, and the third microphone module, and process the sound signals generated by the first microphone module, the second microphone module, and the third microphone module to enhance the sound from the direction of the mouth, thereby obtaining a first sound signal; The sound signals generated by the second microphone module and the third microphone module are added and divided by two to obtain a signal d(n), and the sound signals generated by the second microphone module and the third microphone module are subtracted and divided by two to obtain a signal x(n), where n represents a time sequence number; The signal x(n) is processed by an adaptive filter to obtain the signal y(n); The enhanced output signal e(n) is obtained by subtracting the signal y(n) from the signal d(n) as the second sound signal, and the signal e(n) is input into the parameter updating module of the adaptive filter to update the adaptive filter parameters.

2. The head-mounted device according to claim 1, wherein: The signal synthesis module is further used to process the first sound signal and the second sound signal to remove the first sound part in the second sound signal as an output signal.

3. The head-mounted device according to claim 2, characterized in that: The head-mounted device is glasses, which include two temples located on both sides of a front frame, wherein the front frame is used to fix two lenses, and a first microphone module, a second microphone module and a third microphone module are located between the two lenses, and the signal synthesis module is located in the front frame or the temples.

4. The head-mounted device according to claim 3, characterized in that: The head-mounted device further comprises a sound playing module, which is used for playing sound to the user according to the output signal.

5. The head-mounted device according to claim 4, characterized in that: The sound playing module is located at a portion of the glasses leg close to the ear of the user and is electrically connected to the signal synthesis module to receive the output signal.

6. The head-mounted device according to claim 3, characterized in that: The head-mounted device further comprises a voice recognition module, which is used to perform voice recognition on the output signal to convert the sound signal into text information.

7. The head-mounted device according to claim 6, characterized in that: The head-wearing device further comprises an information display module, which is used to display the text information.

8. The head-mounted device according to claim 7, wherein: The speech recognition module is also used to convert the recognized text information into text information in a language specified by the user, and send it to the information display module for display.

9. The head-mounted device according to claim 8, characterized in that: The information display module is a retinal projection device, which is used to project the text information onto the user's retina.

10. The head-mounted device according to claim 6, wherein: The speech recognition module is also used to convert the recognized text information into text information in a language specified by the user, and then convert the converted text information into a sound signal, which is sent to the sound playing module for playing.

11. The head-mounted device according to any one of claims 1 to 10, characterized in that: The first microphone module and the second microphone module constitute a first channel, and the first microphone module and the third microphone module constitute a second channel; The signal synthesis module is used for: Acquire sound signals generated by the first microphone module, the second microphone module and the third microphone module; Calculating the delayed subtraction beamforming signals of the first channel and the second channel respectively, which include a first channel forward forming signal, a first channel backward forming signal, a second channel forward forming signal and a second channel backward forming signal; Each of the delay-subtraction beamforming signals is divided into a predetermined number of spectrum sub-band signals, and the signal energy E of each of the same spectrum sub-bands of the forward forming signal and the backward forming signal of the same channel is calculated. fij and E bij , f represents forward, b represents backward, i represents the channel number, and j represents the spectrum subband number; Calculate relative energy statistical parameters according to the energies of the spectrum sub-band signals of the forward formed signal and the backward formed signal; According to the relative energy statistical parameter, each spectrum sub-band signal of the first channel forward formed signal and the second channel forward formed signal is assigned a first gain G 1ij , where i is the channel number and j is the spectrum subband number; Calculate the second gain G when the first channel forward forming signal and the second channel forward forming signal are attenuated to the steady-state noise energy level 2ij , where i is the channel number and j is the spectrum subband number; For each spectrum sub-band signal of the first channel forward formed signal and the second channel forward formed signal, the first gain G is compared. 1ij and the second gain G 2ij The size of the channel and the maximum value of the two is taken as the final gain G of the spectrum sub-band signal. ij ; Calculate the energy value E' after noise reduction of each spectrum sub-band signal of the first channel forward formed signal and the second channel forward formed signal fij =E fij *G ij , and compare the energy values ​​E' of the spectrum sub-band signals of the same spectrum sub-band of the two forward forming signals f1j and E' f2j , the final gain G of the spectrum subband j j Take the gain of the channel with smaller energy value; After smoothing the gain value of the spectrum sub-band, the final complex spectrum of the spectrum sub-band is calculated, and then inverse Fourier transform is performed to obtain a time domain signal, that is, the first sound signal.

12. The head-mounted device according to claim 11, characterized in that: The delay subtraction beamforming signal is calculated using a first-order delay subtraction beamforming algorithm.

13. The head-mounted device according to claim 11, characterized in that: The method for calculating the relative energy statistical parameter value is: initialize the relative energy statistical parameter value to 0, compare the energy of each same spectrum sub-band signal of the forward forming signal and the backward forming signal one by one, when the spectrum sub-band signal energy of the forward forming signal is larger, the relative energy statistical parameter value is increased by 1, otherwise it is reduced by 1.

14. The head-mounted device according to claim 11, characterized in that: The signal synthesis module is also used to perform steady-state noise reduction processing on the time domain signal obtained by Fourier transform.

Images