Voice enhancement method and device for cross-device multi-microphone cooperative processing, terminal and medium

Abstract

The invention discloses a voice enhancement method and device for cross-device multi-microphone cooperative processing, a terminal and a medium, and the method comprises the steps: obtaining a host microphone audio signal and an external microphone audio signal, and carrying out the gain compensation of the external microphone audio signal; performing power spectrum calculation based on the host microphone audio signal and the external microphone audio signal after gain compensation, and determining a power spectrum difference ratio and a power spectrum difference spectrum of the host microphone audio signal and the external microphone audio signal after gain compensation at each frequency; and performing noise estimation based on the power spectrum difference ratio at each frequency and the power spectrum difference spectrum to obtain a noise power spectrum, and performing noise reduction processing on the external microphone audio signal based on the noise power spectrum to obtain a noise-reduced external microphone audio signal. According to the invention, through cooperative work of the host microphone and the external microphone, surrounding noise signals are effectively isolated, voice enhancement is realized, and voice definition is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of audio processing technology, and in particular to a speech enhancement method, device, terminal and medium for cross-device multi-microphone collaborative processing. Background Art

[0002] At present, external headphone devices usually have built-in microphones, but in some complex environments, a single microphone may not be able to effectively isolate interfering noise. Even with the deep learning noise reduction method, the interfering human voice cannot be removed for a single microphone, and the noise suppression effect is limited, resulting in reduced speech clarity. In addition, existing audio processing technologies often rely only on the microphone of the external device, while ignoring the potential value of the built-in microphone of the host device. The common external headphone or earplug devices on the market will automatically switch and enable the microphone in the external headphone device after connecting to the host device (such as laptops, mobile phones, tablets, etc.), while the built-in microphone of the host device is disabled or not enabled. This design is effective in daily audio input scenarios, but in noisy environments, the input that relies only on a single microphone is easily interfered by environmental noise, and the target speaker’s voice cannot be clearly captured, and the potential value of the built-in microphone of the host device is not utilized.

[0003] Therefore, the prior art still has defects. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method, device, terminal and medium for speech enhancement by cross-device multi-microphone collaborative processing in view of the above-mentioned defects of the prior art. The technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a method for speech enhancement by cross-device multi-microphone collaborative processing, wherein the method comprises: Acquire a host microphone audio signal and an external microphone audio signal, and perform gain compensation on the external microphone audio signal; Performing power spectrum calculation based on the host microphone audio signal and the gain-compensated external microphone audio signal, and determining a power spectrum difference ratio and a power spectrum difference spectrum at each frequency between the host microphone audio signal and the gain-compensated external microphone audio signal; Noise estimation is performed based on the power spectrum difference ratio at each frequency and the power spectrum difference spectrum to obtain a noise power spectrum, and noise reduction processing is performed on the external microphone audio signal based on the noise power spectrum to obtain a noise-reduced external microphone audio signal.

[0005] In one implementation, the method is applied to a microphone array formed by a host microphone and an external microphone, and voice enhancement is achieved by simultaneously acquiring the host microphone audio signal and the external microphone audio signal.

[0006] In one implementation, performing gain compensation on the external microphone audio signal includes: Performing time domain synchronization on the host microphone audio signal and the external microphone audio signal; Performing background noise tracking detection on the host microphone audio signal and the external microphone audio signal after time domain synchronization to obtain noise levels corresponding to the host microphone audio signal and the external microphone audio signal; Determining a gain compensation coefficient corresponding to the external microphone audio signal based on noise levels corresponding to the host microphone audio signal and the external microphone audio signal; Based on the gain compensation coefficient, gain compensation is performed on the external microphone audio signal.

[0007] In one implementation, performing gain compensation on the external microphone audio signal based on the gain compensation coefficient includes: The gain compensation coefficient is multiplied by the external microphone audio signal to obtain a gain-compensated external microphone audio signal.

[0008] In one implementation, the power spectrum calculation based on the host microphone audio signal and the gain-compensated external microphone audio signal includes: Converting the host microphone audio signal and the gain-compensated external microphone audio signal into frequency domain signals; The power spectra corresponding to the host microphone audio signal and the gain-compensated external microphone audio signal are calculated based on the frequency domain signal, and the power spectrum is used to reflect the power distribution of the host microphone audio signal and the gain-compensated external microphone audio signal at different frequencies.

[0009] In one implementation, the performing noise estimation based on the power spectrum difference ratio at each frequency and the power spectrum difference spectrum to obtain the noise power spectrum includes: Obtaining a preset noise estimation model; Acquire a first threshold, a second threshold, and a third threshold preset in the noise estimation model, wherein the first threshold is less than the second threshold; Compare the power spectrum difference ratio and the power spectrum difference spectrum at each frequency with a first threshold, a second threshold, and a third threshold, respectively; If the power spectrum difference ratio at a certain frequency is between the first threshold and the second threshold, and the corresponding power spectrum difference spectrum is smaller than the third threshold, it is determined that noise exists at the frequency, and the noise power spectrum is obtained.

[0010] In one implementation, the performing noise reduction processing on the external microphone audio signal based on the noise power spectrum to obtain the external microphone audio signal after noise reduction includes: Calculating a filter coefficient based on the noise power spectrum and the original power spectrum corresponding to the external microphone audio signal; Filtering the original frequency domain representation of the external microphone audio signal based on the filter coefficient to obtain a frequency domain representation of the signal after noise reduction; The frequency domain representation of the noise-reduced signal is converted into a time domain representation to obtain a noise-reduced external microphone audio signal.

[0011] In a second aspect, an embodiment of the present invention further includes a speech enhancement device for cross-device multi-microphone collaborative processing, wherein the device is used to implement the steps of the speech enhancement method for cross-device multi-microphone collaborative processing described in the above scheme, and the device includes: A gain compensation module, used to obtain a host microphone audio signal and an external microphone audio signal, and perform gain compensation on the external microphone audio signal; A power spectrum analysis module, configured to calculate a power spectrum based on the host microphone audio signal and the gain-compensated external microphone audio signal, and determine a power spectrum difference ratio and a power spectrum difference spectrum between the host microphone audio signal and the gain-compensated external microphone audio signal at each frequency; The noise reduction processing module is used to perform noise estimation based on the power spectrum difference ratio at each frequency and the power spectrum difference spectrum to obtain a noise power spectrum, and perform noise reduction processing on the external microphone audio signal based on the noise power spectrum to obtain a noise-reduced external microphone audio signal.

[0012] In the third aspect, an embodiment of the present invention further provides a terminal, wherein the terminal includes a memory, a processor, and a speech enhancement program for cross-device multi-microphone collaborative processing stored in the memory and executable on the processor, and when the processor executes the speech enhancement program for cross-device multi-microphone collaborative processing, the steps of the speech enhancement method for cross-device multi-microphone collaborative processing of any one of the above-mentioned schemes are implemented.

[0013] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein a speech enhancement program for cross-device multi-microphone collaborative processing is stored on the computer-readable storage medium, and when the speech enhancement program for cross-device multi-microphone collaborative processing is executed by a processor, the steps of the speech enhancement method for cross-device multi-microphone collaborative processing described in any one of the above-mentioned schemes are implemented.

[0014] Beneficial effect: Compared with the prior art, the present invention provides a speech enhancement method for cross-device multi-microphone collaborative processing, which is applied to a microphone array formed by a host microphone and an external microphone, and achieves speech enhancement by simultaneously acquiring the host microphone audio signal and the external microphone audio signal. The present invention first acquires the host microphone audio signal and the external microphone audio signal, and performs gain compensation on the external microphone audio signal. Then, the power spectrum is calculated based on the host microphone audio signal and the gain-compensated external microphone audio signal, and the power spectrum difference ratio and the power spectrum difference spectrum of the host microphone audio signal and the gain-compensated external microphone audio signal at each frequency are determined. Finally, noise estimation is performed based on the power spectrum difference ratio at each frequency and the power spectrum difference spectrum to obtain a noise power spectrum, and noise reduction processing is performed on the external microphone audio signal based on the noise power spectrum to obtain a noise-reduced external microphone audio signal. The present invention utilizes the potential value of the host microphone, and effectively isolates the surrounding noise signal through the collaborative work of the host microphone and the external microphone, achieves speech enhancement, and effectively improves speech clarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of a preferred embodiment of a speech enhancement method for cross-device multi-microphone collaborative processing provided by an embodiment of the present invention.

[0016] Figure 2 The figure is a schematic diagram of a preferred embodiment of the present invention applied to connecting an external earphone device to a notebook computer.

[0017] Figure 3 The figure is a schematic diagram of a preferred embodiment of the present invention applied to connecting an external earphone device to a mobile phone.

[0018] Figure 4 This is a flow chart of performing gain compensation on an external microphone audio signal in the present invention.

[0019] Figure 5 The spectrum diagram of the audio signal obtained from the external microphone.

[0020] Figure 6 The obtained host microphone audio signal spectrum diagram.

[0021] Figure 7 The figure is a spectrum diagram of a speech signal after speech enhancement processing is performed using the method of the present invention.

[0022] Figure 8 A schematic diagram of the architecture of a speech enhancement apparatus for cross-device multi-microphone collaborative processing provided by an embodiment of the present invention.

[0023] Fig. 9 A functional block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0025] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations or steps, nor must they be executed in the order described. For example, some operations or steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0026] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms. It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish between identical or similar items with substantially identical functions and effects. For example, the first control information and the second control information are only used to distinguish different control information, and their order is not limited. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not necessarily limit the differences. It should be further understood that the term “and / or” used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0027] Based on the defects of the prior art, this embodiment provides a method for voice enhancement through cross-device multi-microphone collaborative processing, which can be applied to a terminal, which can be a computer, a smart TV, a mobile phone or other intelligent terminal product. Figure 1 As shown in , the speech enhancement method for cross-device multi-microphone collaborative processing of this embodiment includes the following steps: Step S100: Acquire a host microphone audio signal and an external microphone audio signal, and perform gain compensation on the external microphone audio signal.

[0028] The speech enhancement method of cross-device multi-microphone collaborative processing in this embodiment is applied to a microphone array formed by a host microphone and an external microphone, and speech enhancement is achieved by simultaneously acquiring the host microphone audio signal and the external microphone audio signal, which is a cross-device implementation method. Specifically, the host microphone in this embodiment is a microphone built into the host device, such as a microphone built into the edge of a laptop or a microphone built into a mobile phone. The external microphone is a microphone in an external wired headset or a Bluetooth headset. The core of the present invention is to utilize the collaborative work of the external microphone and the host microphone, and to achieve effective isolation of interference noise and enhance the target speaking voice by analyzing the stable feature of the amplitude difference of the sound signals collected by the two microphones. Since the two microphones have differences in spatial positions, the amplitude of the sound signals received by them will be different for sounds coming from different directions. The target speaking voice is usually close to the external microphone (such as a Bluetooth headset) and has a large amplitude; while the interference noise is relatively far away, and the amplitude is relatively close when it reaches the two microphones. Therefore, this embodiment can use this feature to process the signals collected by the two microphones through a specific algorithm, so as to achieve the effect of noise reduction and voice enhancement.

[0029] Specific as Figure 2 As shown in Figure 2 The schematic diagram of a preferred embodiment of the present invention applied to connecting an external headphone device to a laptop computer. In the prior art, when a user connects an external wired headset with a microphone, the system automatically switches the data source of the system audio to the external wired headset and ignores the built-in microphone device of the laptop computer. To solve this problem, in this embodiment, while the system automatically switches the audio signal source to the external wired headset, a module for acquiring audio signals from the built-in microphone of the laptop computer is constructed; when the user uses the external wired headset to make a call, the audio signals of the external wired headset and the laptop computer microphone can be acquired at the same time, thereby constructing a microphone array to collaboratively enhance the microphone audio signal, thereby ensuring that the surrounding interference signals can be effectively isolated and the call quality can be effectively improved. In another embodiment, such as Figure 3 As shown in Figure 3 The figure is a schematic diagram of a preferred embodiment of the present invention applied to an external earphone device connected to a mobile phone. Similarly, in this embodiment, the mobile phone as a host device has a built-in microphone. When the user connects to a Bluetooth earphone device with a microphone, the audio signals of the host device and the external microphone are obtained at the same time, so that the microphone data of the two devices can be processed collaboratively.

[0030] Based on this, the speech enhancement method of cross-device multi-microphone collaborative processing in this embodiment is applied to a microphone array, which includes at least two microphones, one of which is a host microphone (such as a mobile phone) and the other is an external microphone (such as a Bluetooth headset). When a user uses an external microphone to make a call, this embodiment can simultaneously obtain the host microphone audio signal and the external microphone audio signal. Next, since there are many types of external devices with microphones in the market, and the enhancement method in this embodiment mainly uses the amplitude difference of the sound signal collected by the two microphones (external device and host device) as a feature, and the microphones of different external devices have different microphone sensitivities. In order to effectively adapt to various types of microphone external devices in the market, it is necessary to perform adaptive gain compensation on the external microphone audio signal.

[0031] In one implementation, Figure 4 As shown, in this embodiment, gain compensation for the external microphone audio signal includes the following steps: Step S101, performing time domain synchronization on the host microphone audio signal and the external microphone audio signal; Step S102, performing background noise tracking detection on the host microphone audio signal and the external microphone audio signal after time domain synchronization to obtain noise levels corresponding to the host microphone audio signal and the external microphone audio signal; Step S103, determining a gain compensation coefficient corresponding to the external microphone audio signal based on the noise levels corresponding to the host microphone audio signal and the external microphone audio signal; Step S104: Perform gain compensation on the external microphone audio signal based on the gain compensation coefficient.

[0032] Specifically, this embodiment first calculates the cross-correlation function of the dual-channel signal (i.e., the host microphone audio signal and the external microphone audio signal), determines the peak position based on the cross-correlation function, and then realizes time domain synchronization through peak alignment. In specific applications, this embodiment can determine the offset of the two signals based on the determined peak position, and based on the determined offset, shift the external microphone audio signal in the time domain so that the peak of the external microphone audio signal is aligned with that of the host microphone audio signal, thereby ensuring that the external microphone audio signal is strictly synchronized with the host microphone audio signal on the time axis.

[0033] Next, in the present embodiment, when there is no voice or in a silent state, the host microphone audio signal and the external microphone audio signal after time domain synchronization are respectively subjected to background noise tracking detection to obtain the noise levels corresponding to the host microphone audio signal and the external microphone audio signal, which can be specifically achieved by voice activity detection (VAD). Then, based on the noise levels corresponding to the host microphone audio signal and the external microphone audio signal, the present embodiment determines the gain compensation coefficient corresponding to the external microphone audio signal. Specifically, the gain compensation coefficient corresponding to the external microphone audio signal is the square root of the ratio between the noise levels of the two signals. After calculating the gain compensation coefficient, the present embodiment can perform gain compensation on the external microphone audio signal based on the gain compensation coefficient. Specifically, the gain compensation coefficient can be multiplied by the external microphone audio signal to obtain the external microphone audio signal after gain compensation, so as to adjust the amplitude of the external microphone audio signal so that it is consistent with the amplitude of the host microphone audio signal (i.e., the reference signal), thereby completing the automatic adjustment of the gain of microphones of different devices. Step S200: Calculate the power spectrum based on the host microphone audio signal and the gain compensated external microphone audio signal, and determine the power spectrum difference ratio and power spectrum difference spectrum between the host microphone audio signal and the gain compensated external microphone audio signal at each frequency.

[0034] After gain compensation is performed on the external microphone audio signal, this embodiment can calculate and analyze the power spectrum of the signals of the two channels after gain compensation. The power spectrum reflects the distribution of the signals of each channel at different frequencies. Therefore, based on the calculated power spectrum, the power spectrum difference ratio and power spectrum difference spectrum of the host microphone audio signal and the external microphone audio signal after gain compensation at each frequency can be determined respectively. This embodiment can effectively distinguish the noise signal by analyzing the power spectrum difference ratio and the power spectrum difference spectrum.

[0035] Specifically, this embodiment may use a short-time Fourier transform (DTFT) algorithm to convert the host microphone audio signal and the gain-compensated external microphone audio signal into frequency domain signals, obtain the power distribution of each channel signal at different frequencies, and then calculate the power spectrum corresponding to the host microphone audio signal and the gain-compensated external microphone audio signal based on the frequency domain signals. For example, assuming that the gain-compensated external microphone audio signal is x1(t), and the host microphone audio signal is x2(t), where t represents the time corresponding to the audio signal, after short-time Fourier transform, their frequency domain signals are obtained as X1(f) and X2(f), respectively, and then the power spectrum P1(f)=|X1(f)|² of the gain-compensated external microphone audio signal and the power spectrum P2(f)=|X2(f)|² of the host microphone audio signal are calculated, where f represents the frequency.

[0036] Furthermore, in order to more accurately measure the degree of difference in power between the two microphone audio signals at different frequencies, this embodiment introduces a power spectrum difference ratio P(f), which is calculated as follows: P(f) = P1(f) / P2(f), where P1(f) is the power spectrum of the external microphone audio signal, and P2(f) is the power spectrum of the host microphone audio signal. P(f) reflects the relative magnitude relationship of the signal power collected by the two microphones at frequency f. When the target speaking voice is dominant, since it is closer to the external microphone, P1(f) will be relatively large, making the value of P(f) larger.

[0037] In addition, after obtaining the power spectrum difference ratio P(f), the present embodiment introduces the power spectrum difference spectrum D(f). The power spectrum difference spectrum is calculated as D(f)=P1(f)-P2(f), and the power spectrum difference spectrum of the present embodiment is a specific value. It can be seen that the present embodiment can calculate the power spectrum difference ratio and the power spectrum difference spectrum at each frequency, and the power spectrum difference ratio and the power spectrum difference spectrum complement each other, and jointly reflect the difference in power of the signals of the two channels at different frequencies. Since the propagation characteristics of the target speaking voice and the interfering noise at the two microphones are different, they have obvious characteristics in both the power spectrum difference ratio and the power spectrum difference spectrum, providing richer information for subsequent noise identification and processing.

[0038] Step S300: performing noise estimation based on the power spectrum difference ratio at each frequency and the power spectrum difference spectrum to obtain a noise power spectrum, and performing noise reduction processing on the external microphone audio signal based on the noise power spectrum to obtain a noise-reduced external microphone audio signal.

[0039] After calculating the power spectrum difference ratio at each frequency and the power spectrum difference spectrum, this embodiment can be combined with the noise estimation model to perform noise estimation, identify the noise distribution, and obtain the noise power spectrum. Specifically, the noise estimation model of this embodiment can preset a first threshold T1, a second threshold T2, and a third threshold T3, wherein the first threshold T1 is less than the second threshold T2. Then the power spectrum difference ratio and the power spectrum difference spectrum corresponding to each frequency are compared with the first threshold, the second threshold, and the third threshold, respectively. If the power spectrum difference ratio at a certain frequency is between the first threshold and the second threshold (ie, T1 <P (f) <T2), and the corresponding power spectrum difference spectrum is less than the third threshold (ie, D (f) <T3), it is determined that there is noise in the signal at the frequency, and the noise is identified. Based on the same method, it is possible to analyze whether there is noise at all other frequencies, and then obtain the noise power spectrum according to the power spectrum characteristics of the part of the signal where there is noise. The noise power spectrum reflects the power distribution of the noise signal at different frequencies.

[0040] The first threshold, the second threshold and the third threshold in this embodiment are set based on a large amount of experimental data and actual application scenarios, and this embodiment also collects a large amount of speech data in different noise environments, analyzes the distribution characteristics of the target speaker's voice and the noise in the power spectrum difference ratio and the power spectrum difference spectrum, so as to set various thresholds that can effectively distinguish noise from the target speaker's voice. By continuously tracking and updating the power spectrum characteristics of the noise, the noise signal can be more accurately identified.

[0041] Taking into account the time-varying characteristics of noise in the actual environment, the noise estimation model of this embodiment has adaptive adjustment capabilities. As time goes by and the signal continues to be input, the noise estimation model will update the estimate of the noise power spectrum in real time according to the new observation data. For example, when the environmental noise suddenly increases or the type changes, the model can quickly capture the changes in the power spectrum difference ratio and the power spectrum difference spectrum, and re-estimate the new noise power spectrum to ensure the stability and effectiveness of the noise reduction effect.

[0042] Furthermore, after obtaining the noise power spectrum, this embodiment can use algorithms such as Wiener filtering and deep learning to calculate the filter coefficient based on the noise power spectrum and the power spectrum corresponding to the original external microphone audio signal, and filter the original external microphone audio signal to remove the noise component and retain the target speaking voice. Specifically, for each frequency f, the filter coefficient H(f) is calculated according to the Wiener filtering formula, and then the frequency domain representation of the original external microphone audio signal is filtered based on the filter coefficient to obtain the frequency domain representation of the signal after noise reduction Y(f)=H(f)*X2(f). Then, the frequency domain representation of the noise-reduced signal is converted into a time domain representation through an inverse fast Fourier transform (IFFT) to obtain the external microphone audio signal after noise reduction.

[0043] It has been verified that the speech enhancement method of the present invention can significantly suppress interfering human voices and background noise. Figure 5 As shown, Figure 5 The obtained external microphone audio signal spectrum diagram includes the voice of the external microphone wearer, the surrounding interfering voices and environmental noise. Figure 6 The obtained host microphone audio signal spectrum diagram contains the target human voice, surrounding interfering human voices and environmental noise. Figure 7 As shown, Figure 7 is a speech signal spectrum diagram after enhancement processing by the method of the present invention, from Figure 7 It can be seen that interfering human voices and background noise are greatly suppressed, and the voice of the wearer of the external microphone is effectively enhanced.

[0044] In summary, this embodiment first obtains the host microphone audio signal and the external microphone audio signal, and performs gain compensation on the external microphone audio signal. Then, a power spectrum is calculated based on the host microphone audio signal and the external microphone audio signal after gain compensation, and the power spectrum difference ratio and the power spectrum difference spectrum at each frequency between the host microphone audio signal and the external microphone audio signal after gain compensation are determined. Finally, noise estimation is performed based on the power spectrum difference ratio at each frequency and the power spectrum difference spectrum to obtain a noise power spectrum, and noise reduction processing is performed on the external microphone audio signal based on the noise power spectrum to obtain a noise-reduced external microphone audio signal. This embodiment utilizes the potential value of the host microphone, and through the coordinated operation of the host microphone and the external microphone, it effectively isolates the surrounding noise signals, achieves voice enhancement, and effectively improves voice clarity.

[0045] The present invention can be widely used in various devices that require high-quality voice input, such as teleconferencing, voice recognition, live broadcasting and online communication, and is particularly suitable for scenarios with high demands on voice clarity in noisy environments, such as outdoor scenes, open office environments, public places, etc.

[0046] Based on the above embodiments, the present invention provides a speech enhancement device for cross-device multi-microphone collaborative processing, which can be used to implement the steps of the above-mentioned speech enhancement method for cross-device multi-microphone collaborative processing, such as Figure 8 As shown in , the device includes: a gain compensation module 10, a power spectrum analysis module 20 and a noise reduction processing module 30. Specifically, the gain compensation module 10 is used to obtain the host microphone audio signal and the external microphone audio signal, and perform gain compensation on the external microphone audio signal. The power spectrum analysis module 20 is used to perform power spectrum calculation based on the host microphone audio signal and the external microphone audio signal after gain compensation, and determine the power spectrum difference ratio and power spectrum difference spectrum of the host microphone audio signal and the external microphone audio signal after gain compensation at each frequency. The noise reduction processing module 30 is used to perform noise estimation based on the power spectrum difference ratio at each frequency and the power spectrum difference spectrum to obtain a noise power spectrum, and perform noise reduction processing on the external microphone audio signal based on the noise power spectrum to obtain a noise-reduced external microphone audio signal.

[0047] The working principles of each module in the speech enhancement system for cross-device multi-microphone collaborative processing in this embodiment are the same as the principles of each step in the above method embodiment, and will not be repeated here.

[0048] Each module in the above-mentioned cross-device multi-microphone collaborative voice enhancement system can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the terminal in the form of hardware, or can be stored in the memory in the terminal in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0049] Based on the above embodiment, the present invention further provides a terminal, the principle block diagram of the terminal can be as follows: Fig. 9 The terminal may include one or more processors 100 ( Fig. 9Only one is shown in the figure), a memory 101 and a computer program 102 stored in the memory 101 and executable on one or more processors 100. For example, a speech enhancement program for cross-device multi-microphone collaborative processing. When one or more processors 100 execute the computer program 102, the various steps in the embodiment of the speech enhancement method for cross-device multi-microphone collaborative processing can be implemented. Alternatively, when one or more processors 100 execute the computer program 102, the functions of each module / unit in the embodiment of the speech enhancement system for cross-device multi-microphone collaborative processing can be implemented, which is not limited here.

[0050] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0051] In one embodiment, the memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 101 may also include both an internal storage unit of the electronic device and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the terminal. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0052] Those skilled in the art will understand that Fig. 9 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0053] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, operating database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double operational data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech enhancement method for cross-device multi-microphone collaborative processing, characterized in that: The method comprises: Acquire a host microphone audio signal and an external microphone audio signal, and perform gain compensation on the external microphone audio signal; Performing power spectrum calculation based on the host microphone audio signal and the gain-compensated external microphone audio signal, and determining a power spectrum difference ratio and a power spectrum difference spectrum at each frequency between the host microphone audio signal and the gain-compensated external microphone audio signal; Noise estimation is performed based on the power spectrum difference ratio at each frequency and the power spectrum difference spectrum to obtain a noise power spectrum, and noise reduction processing is performed on the external microphone audio signal based on the noise power spectrum to obtain a noise-reduced external microphone audio signal.

2. The method for speech enhancement by cross-device multi-microphone collaborative processing according to claim 1, characterized in that: The method is applied to a microphone array formed by a host microphone and an external microphone, and voice enhancement is achieved by simultaneously acquiring the host microphone audio signal and the external microphone audio signal.

3. The method for speech enhancement by cross-device multi-microphone collaborative processing according to claim 1, characterized in that: The step of performing gain compensation on the external microphone audio signal comprises: Performing time domain synchronization on the host microphone audio signal and the external microphone audio signal; Performing background noise tracking detection on the host microphone audio signal and the external microphone audio signal after time domain synchronization to obtain noise levels corresponding to the host microphone audio signal and the external microphone audio signal; Determining a gain compensation coefficient corresponding to the external microphone audio signal based on noise levels corresponding to the host microphone audio signal and the external microphone audio signal; Based on the gain compensation coefficient, gain compensation is performed on the external microphone audio signal.

4. The method for speech enhancement by cross-device multi-microphone collaborative processing according to claim 3, characterized in that: The step of performing gain compensation on the external microphone audio signal based on the gain compensation coefficient includes: The gain compensation coefficient is multiplied by the external microphone audio signal to obtain a gain-compensated external microphone audio signal.

5. The method for speech enhancement by cross-device multi-microphone collaborative processing according to claim 1, characterized in that: The power spectrum calculation based on the host microphone audio signal and the gain-compensated external microphone audio signal includes: Converting the host microphone audio signal and the gain-compensated external microphone audio signal into frequency domain signals; The power spectrum corresponding to the host microphone audio signal and the gain-compensated external microphone audio signal is calculated based on the frequency domain signal, and the power spectrum is used to reflect the power distribution of the host microphone audio signal and the gain-compensated external microphone audio signal at different frequencies.

6. The method for speech enhancement by cross-device multi-microphone collaborative processing according to claim 1, characterized in that: The performing noise estimation based on the power spectrum difference ratio at each frequency and the power spectrum difference spectrum to obtain the noise power spectrum includes: Obtaining a preset noise estimation model; Acquire a first threshold, a second threshold, and a third threshold preset in the noise estimation model, wherein the first threshold is less than the second threshold; Compare the power spectrum difference ratio and the power spectrum difference spectrum at each frequency with a first threshold, a second threshold, and a third threshold, respectively; If the power spectrum difference ratio at a certain frequency is between the first threshold and the second threshold, and the corresponding power spectrum difference spectrum is smaller than the third threshold, it is determined that noise exists at the frequency, and the noise power spectrum is obtained.

7. The method for speech enhancement by cross-device multi-microphone collaborative processing according to claim 1, characterized in that: The performing noise reduction processing on the external microphone audio signal based on the noise power spectrum to obtain the external microphone audio signal after noise reduction includes: Calculating a filter coefficient based on the noise power spectrum and the original power spectrum corresponding to the external microphone audio signal; Filtering the original frequency domain representation of the external microphone audio signal based on the filter coefficient to obtain a frequency domain representation of the signal after noise reduction; The frequency domain representation of the noise-reduced signal is converted into a time domain representation to obtain a noise-reduced external microphone audio signal.

8. A speech enhancement device for cross-device multi-microphone collaborative processing, characterized in that: The device is used to implement the steps of the method for speech enhancement by cross-device multi-microphone collaborative processing according to any one of claims 1 to 7, and the device comprises: A gain compensation module, used to obtain a host microphone audio signal and an external microphone audio signal, and perform gain compensation on the external microphone audio signal; A power spectrum analysis module, configured to calculate a power spectrum based on the host microphone audio signal and the gain-compensated external microphone audio signal, and determine a power spectrum difference ratio and a power spectrum difference spectrum between the host microphone audio signal and the gain-compensated external microphone audio signal at each frequency; The noise reduction processing module is used to perform noise estimation based on the power spectrum difference ratio at each frequency and the power spectrum difference spectrum to obtain a noise power spectrum, and perform noise reduction processing on the external microphone audio signal based on the noise power spectrum to obtain a noise-reduced external microphone audio signal.

9. A terminal, characterized in that: The terminal includes a memory, a processor, and a speech enhancement program for cross-device multi-microphone collaborative processing stored in the memory and executable on the processor. When the processor executes the speech enhancement program for cross-device multi-microphone collaborative processing, the steps of the speech enhancement method for cross-device multi-microphone collaborative processing as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a speech enhancement program for cross-device multi-microphone collaborative processing. When the speech enhancement program for cross-device multi-microphone collaborative processing is executed by the processor, the steps of the speech enhancement method for cross-device multi-microphone collaborative processing as described in any one of claims 1-7 are implemented.

Images