A speech enhancement processing method of a multi-microphone array, a speech enhancement processing system, and an electronic device

Abstract

The application provides a multi-microphone array voice enhancement processing method, a voice enhancement processing system and electronic equipment, and the method comprises the following steps: collecting voice frames at each sampling time through a multi-microphone array at a preset frequency; performing voice processing on the voice frames to obtain a phase stability index and an energy fluctuation index; based on the phase stability index, the energy fluctuation index and a wind noise intensity evaluation value, determining the voice frames; when the voice frames are wind noise dominant frames, correcting the voice frames after gain adjustment through a pre-trained deep neural network to obtain repaired voice frames. The calculation complexity and hardware resource consumption are significantly reduced, the deep neural network is only used for reconstruction of the wind noise dominant frames, the voice components covered by the wind noise can be more naturally reconstructed, the voice repair is realized while the voice naturalness is maintained, and the subjective auditory experience is improved.

Description

Technical Field

[0001] This application relates to the field of speech signal processing technology, and in particular to a speech enhancement processing method, speech enhancement processing system and electronic device using a multi-microphone array. Background Technology

[0002] With the increasing prevalence of voice pickup scenarios, existing wind noise suppression methods using multi-microphone arrays rely on the energy or spectral characteristics of the speech signal for detection and suppression. For example, they construct wind noise detection models by calculating the energy difference and spectral coherence of speech frames at various sampling times across multiple channels. After identifying wind noise frames using a fixed threshold, a uniform linear attenuation strategy is employed to suppress the energy of the wind noise frequency band. Although simple inter-frame smoothing is used to reduce gain abrupt changes, the overall method still revolves around suppressing a single energy or spectral characteristic, or a combination of these characteristics. How to accurately detect wind noise and, while suppressing its energy, maximize the preservation of the naturalness and intelligibility of the speech signal has become a core technical problem urgently needing to be solved in the field of speech signal processing. Summary of the Invention

[0003] This invention provides a speech enhancement processing method and system using a multi-microphone array to solve the technical problem of low distinguishability between wind noise frames and speech frames, which easily leads to misjudgment and missed judgment.

[0004] This invention provides a speech enhancement processing method using a multi-microphone array, comprising: acquiring speech frames at various sampling times using a multi-microphone array at a preset frequency; performing speech processing on the speech frames to obtain a phase stability index and an energy fluctuation index; calculating a wind noise intensity assessment value based on the phase stability index and the energy fluctuation index; dynamically adjusting the gain of the speech frames according to the wind noise intensity assessment value to obtain a gain-adjusted speech frame; and determining the speech frame based on the phase stability index, the energy fluctuation index, and the wind noise intensity assessment value: when the speech frame is a wind noise-dominant frame, correcting the gain-adjusted speech frame using a pre-trained deep neural network to obtain a repaired speech frame.

[0005] In one embodiment of the present invention, speech processing is performed on a speech frame to obtain a phase stability index, including: extracting the instantaneous phase of the speech frame in each microphone channel of a multi-microphone array using Hilbert transform; calculating the variance corresponding to the instantaneous phase difference of the instantaneous phase within a preset time window to obtain the phase stability index of the speech frame.

[0006] In one embodiment of the present invention, speech processing is performed on a speech frame to obtain an energy fluctuation index, including: calculating the instantaneous energy of the speech frame in each microphone channel of the multi-microphone array, and the channel average energy of the speech frame in each microphone channel; and determining the energy fluctuation index of the speech frame based on the instantaneous energy and the channel average energy.

[0007] In one embodiment of the present invention, the wind noise intensity assessment value is calculated based on the phase stability index and the energy fluctuation index, including: linearly weighting and fusing the phase stability index and the energy fluctuation index to obtain the wind noise intensity assessment value at each sampling time.

[0008] In one embodiment of the present invention, the determination of a speech frame based on the phase stability index, energy fluctuation index, and wind noise intensity assessment value includes: calculating the current dynamic threshold of phase difference according to the phase stability index; wherein the dynamic threshold of phase difference is calculated according to the following formula. , m represents the sampling time of the speech frame based on the preset frequency. Indicates the first Phase stability index of speech frames at each sampling time. Indicates the first The dynamic threshold of phase difference corresponding to each sampling time point of the speech frame. Represents the threshold coefficient. The number of historical audio frames. Representing the The phase stability index of a historical speech frame; the phase stability index is determined based on the current dynamic threshold of phase difference: when At that time, the speech frame was initially determined to be the wind noise-dominated frame.

[0009] In one embodiment of the present invention, the determination of a speech frame based on a phase stability index, an energy fluctuation index, and a wind noise intensity assessment value further includes: calculating a current dynamic threshold for wind noise intensity based on the wind noise intensity assessment value, wherein the dynamic threshold for wind noise intensity is calculated according to the following formula. , m represents the sampling time of the speech frame based on the preset frequency. Indicates the first Wind noise intensity assessment value of speech frames at each sampling time. This is the moving average of the wind noise intensity assessment values ​​of the historical speech frames corresponding to the given speech frame. and To adjust parameters; the wind noise intensity assessment value is determined based on the current dynamic threshold of wind noise intensity: when At that time, the speech frame was ultimately determined to be the wind noise-dominated frame.

[0010] In one embodiment of the present invention, the gain of a speech frame is dynamically adjusted according to the wind noise intensity assessment value to obtain a gain-adjusted speech frame, including: converting the time-domain signal of the speech frame into a frequency-domain spectrum; determining a preset nonlinear attenuation curve according to the wind noise intensity assessment value of the speech frame; and dynamically adjusting the gain coefficient of each frequency band in the frequency-domain spectrum to obtain a gain-adjusted speech frame.

[0011] In one embodiment of the present invention, when the speech frame is a wind noise-dominated frame, the gain-adjusted speech frame is corrected by a pre-trained deep neural network to obtain a repaired speech frame, including: taking the gain-adjusted speech frame as input, and performing nonlinear correction of spectral distortion on the speech frame by a pre-trained deep neural network to obtain a repaired speech frame.

[0012] A second aspect of the present invention provides a speech enhancement processing system for a multi-microphone array, comprising: a multi-channel signal input module and a speech enhancement processing module, wherein the multi-channel signal input module is used to acquire speech frames at various sampling times through the multi-microphone array at a preset frequency; the speech enhancement processing module is used to perform speech processing on the speech frames to obtain a phase stability index and an energy fluctuation index, and to calculate a wind noise intensity assessment value based on the phase stability index and the energy fluctuation index; to dynamically adjust the gain of the speech frames according to the wind noise intensity assessment value to obtain a gain-adjusted speech frame; and to determine the speech frame based on the phase stability index, the energy fluctuation index, and the wind noise intensity assessment value: when the speech frame is a wind noise-dominant frame, the gain-adjusted speech frame is corrected through a pre-trained deep neural network to obtain a repaired speech frame.

[0013] A third aspect of the present invention provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing, when executing the computer program, the voice enhancement processing method of the multi-microphone array as described in the first aspect.

[0014] The beneficial effects of this invention are as follows: By employing a wind noise detection mechanism based on instantaneous phase difference, the instantaneous phase of each microphone channel corresponding to a multi-microphone array is extracted using a preset sampling frequency and Hilbert transform. The instantaneous phase difference matrix between adjacent microphone pairs is calculated, and its variance distribution characteristics are then analyzed. This method exhibits significantly higher sensitivity, effectively identifying weak wind noise components that are difficult to detect using traditional methods. An energy fluctuation index is constructed by fusing short-term energy distribution characteristics of multi-channel signals. A comprehensive discriminant is obtained through a weighted linear combination, and an adaptive threshold is set to accurately distinguish between wind noise-dominant frames and speech-dominant frames. Robust frame classification is achieved without the need for complex machine learning models, significantly reducing computational complexity and hardware resource consumption. The use of a deep neural network for reconstruction only of wind noise-dominant frames allows for a more natural reconstruction of speech components masked by wind noise, achieving speech restoration while maintaining speech naturalness and improving the subjective auditory experience. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0016] Figure 1 A phase coherence comparison diagram of clean speech and speech with wind noise provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating a speech enhancement processing method for a multi-microphone array provided in one embodiment of the present invention. Figure 3 This is a schematic diagram of a process for preliminary determination of speech frames based on phase stability index and dynamic threshold of phase difference, provided in one embodiment of the present invention. Figure 4 This is a schematic diagram of the process for determining speech frames by fusing short-term energy distribution features of each microphone channel at each acquisition time, provided in one embodiment of the present invention. Figure 5 This is a schematic diagram of the process for adjusting the gain of the identified wind noise dominant frame provided in one embodiment of the present invention; Figure 6 This is a comparison diagram of wind noise speech processing before and after one embodiment of the present invention; Figure 7 This is a schematic diagram of the module functions of a speech enhancement processing system provided in one embodiment of the present invention; Figure 8 This is a schematic diagram of an electronic device running a voice enhancement processing system provided in one embodiment of the present invention. Detailed Implementation

[0017] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0018] Before describing the implementation methods of this application in detail, we will first provide an inductive explanation of the application environment, the technical problems encountered, and the core technical logic for solving these problems.

[0019] With the rapid development of smart devices, users are increasingly using smartphones, true wireless stereo headphones, smart glasses, and in-vehicle voice interaction terminals to pick up audio in various complex acoustic environments. In these scenarios, smart devices inevitably encounter natural wind, ambient wind, or relative airflow generated by the device's movement, resulting in wind noise in the voice captured by the device. Wind noise is a non-stationary, incoherent low-frequency physical disturbance generated by air turbulence on the microphone surface. When airflow impacts the microphone's acoustic windproof mesh or diaphragm, it produces severe air pressure fluctuations, such as... Figure 1 As shown, after this fluctuation is converted into an electrical signal by the microphone, it manifests as noise with significant energy, a spectrum mainly concentrated in the low-frequency band, and extremely strong randomness.

[0020] The core objective of speech enhancement processing is to extract clean target speech from speech contaminated by noise (wind noise). However, since the energy of wind noise is often much greater than that of human speech, and its frequency characteristics highly overlap with the audio segments of human speech, traditional speech enhancement methods face intractable technical problems when dealing with wind noise. For example, relying solely on energy or spectral features is significantly less sensitive to low-frequency wind noise detection, making it difficult to identify weak low-frequency wind noise components. Furthermore, in non-stationary wind noise scenarios, the distinction between wind noise-dominant frames and speech-dominant frames is low, easily leading to false positives and false negatives. Simultaneously, in terms of wind noise suppression, existing methods lack specificity in their suppression strategies. During the attenuation of wind noise energy, unnecessary damage can be caused to the effective low-frequency components of the speech itself. Additionally, traditional speech enhancement methods can lead to the loss of spectral details, damaging the speech. Due to component damage and the loss of spectral details, the auditory comfort of the processed speech is significantly reduced, failing to balance wind noise suppression effectiveness with natural speech intelligibility.

[0021] To address the aforementioned issues, embodiments of this application provide a speech enhancement processing method, comprising: a four-layer progressive processing step, achieving high-sensitivity detection of low-frequency wind noise through phase feature analysis, accurately identifying wind noise frames by fusing multi-dimensional features, and then undergoing step-by-step processing of frequency domain gain adjustment and spectral detail restoration, thereby efficiently suppressing wind noise while preserving the original features of the speech to the greatest extent possible and ensuring the intelligibility of the speech.

[0022] In some embodiments, this application provides a speech enhancement processing method using a multi-microphone array, comprising: acquiring speech frames at each sampling time using a multi-microphone array at a preset frequency; performing speech processing on the speech frames to obtain a phase stability index and an energy fluctuation index; calculating a wind noise intensity assessment value based on the phase stability index and the energy fluctuation index; dynamically adjusting the gain of the speech frames according to the wind noise intensity assessment value to obtain a gain-adjusted speech frame; determining the speech frame based on the phase stability index, the energy fluctuation index, and the wind noise intensity assessment value; when the speech frame is a wind noise-dominant frame, correcting the gain-adjusted speech frame using a pre-trained deep neural network to obtain a repaired speech frame; and finally, integrating the speech frames at each sampling time into a speech signal.

[0023] Specifically, a multi-microphone array is used to acquire and record speech frames at each sampling time using a preset frequency (high-frequency sampling). Hilbert transform is used to extract the instantaneous phase of each microphone channel in the multi-microphone array at each sampling time (high-frequency sampling and analytical signal transformation are used to extract the instantaneous phase of each microphone channel). An instantaneous phase difference matrix between adjacent microphone channels corresponding to a preset acquisition time period is constructed, and the variance within the time window (variance distribution characteristics of the instantaneous phase difference) is calculated. A phase stability index is then established. The initial detection marker is generated by combining the first adaptive dynamic threshold (phase difference dynamic threshold). Wind noise dominant frame marker, The speech-dominant frame flag (the preliminary detection flag, also known as the first detection flag) is a preliminary detection flag.

[0024] Calculate the instantaneous energy of each microphone channel at each sampling time. and channel average energy To construct an energy fluctuation index Based on preliminary detection indicators, the phase stability index is... With energy fluctuation index The wind noise intensity assessment value is obtained by weighted fusion. Combined with the second adaptive dynamic threshold (dynamic threshold for wind noise intensity), That is, when In time, confirm whether the speech frame is a wind noise-dominated frame or a speech-dominated frame, determine the wind noise-dominated frame, and determine the final detection flag (generate the second detection flag). Wind noise dominant frame marker, (Speech-dominant frame flag).

[0025] The time-domain signal corresponding to each sampling moment of the speech frame is converted into a frequency-domain spectrum. Based on the wind noise intensity assessment value of the speech frame, a preset nonlinear attenuation curve is used to dynamically adjust the gain coefficient of each frequency band of the frequency-domain spectrum, and inter-frame smoothing is performed to obtain the gain-adjusted frequency-domain spectrum. The gain-adjusted frequency-domain spectrum of the speech frame and the second detection flag (wind noise dominant frame) are then used as the basis for the signal analysis. As input, the frequency spectrum distortion is nonlinearly corrected through a pre-trained deep neural network, and the repaired frequency domain spectrum is output and converted into the corresponding speech frame in the time domain signal, which is then integrated into a speech signal.

[0026] Please see Figure 2 , Figure 2 A speech enhancement processing method for a multi-microphone array provided in one embodiment of the present invention, such as... Figure 2 As shown, the voice enhancement processing method is applied to a smart device configured with a multi-microphone array, and the voice enhancement processing method includes the following steps.

[0027] Step S201: Acquire speech frames at each sampling time using a multi-microphone array at a preset frequency.

[0028] Step S202: Perform speech processing on the speech frame to obtain the phase stability index and energy fluctuation index. Based on the phase stability index and the energy fluctuation index, calculate the wind noise intensity assessment value. Use Hilbert transform to extract the instantaneous phase of each microphone channel in the multi-microphone array for the speech frame. Construct an instantaneous phase difference matrix between adjacent microphone channels corresponding to a preset acquisition time period and calculate the variance (variance distribution characteristics of the instantaneous phase difference) within a preset time window. Establish a phase stability index and combine it with a first adaptive dynamic threshold (phase difference dynamic threshold) to generate a preliminary detection flag (also called a frame detection flag, which can include two types: wind noise-dominated frame flag and speech-dominated frame flag). Calculate the instantaneous energy and average channel energy of each microphone channel from the speech frame to construct the energy fluctuation index. Combine the preliminary detection flag with the phase stability index... With energy fluctuation index The weighted fusion is used to obtain the comprehensive discrimination value. (Wind noise intensity assessment value).

[0029] Step S203: Dynamically adjust the gain of the speech frame according to the wind noise intensity assessment value to obtain the gain-adjusted speech frame. That is, convert the time domain signal of the multi-channel signal into the frequency domain spectrum, call the preset nonlinear attenuation curve according to the wind noise intensity assessment value to dynamically adjust the gain coefficient of each frequency band of the frequency domain spectrum, and perform inter-frame smoothing to obtain the gain-adjusted frequency domain spectrum.

[0030] Step S204: Based on the phase stability index, energy fluctuation index and wind noise intensity assessment value, the speech frame is judged: when the speech frame is a wind noise dominant frame, the gain-adjusted speech frame is corrected by a pre-trained deep neural network to obtain the repaired speech frame.

[0031] Combined with the second adaptive dynamic threshold, namely, the dynamic threshold of wind noise intensity ( This confirms the identification of the wind noise-dominant frame and the final detection flag from the speech frames. The gain-adjusted frequency domain spectrum and the final detection flag (wind noise-dominant frame) are then used. As input, the frequency spectrum distortion is nonlinearly corrected by a pre-trained deep neural network, and the repaired frequency domain spectrum is output and converted into a multi-channel signal corresponding to the time domain signal.

[0032] In some embodiments, for step S201, the preset frequency can be a high-frequency sampling method. For example, a multi-microphone array can include I microphones, and the preset frequency can be set to a sampling frequency of 48kHz, including 48,000 sampling moments per second, with the time interval between adjacent sampling moments being Δt = 1 / 48000 ≈ 20.8 microseconds. The preset frequency can be configured to be no less than 32kHz, for example, 44.1kHz or 48kHz, or, when extremely high peak wind noise suppression performance is required, it can be set to 96kHz. Assuming the frequency of wind noise is 100Hz, a suitable preset frequency allows the Hilbert transform to accurately calculate the instantaneous phase every 20.8 microseconds. Thus, in the variance calculation in step 1.3, the fluctuation of the phase difference between adjacent sampling times caused by wind noise is clearly captured.

[0033] In some embodiments, for step S202, refer to Figure 3 Step S202 can be composed of steps S2021 to S2025, wherein: Step S2021: Extract the instantaneous phase of the speech frame in each microphone channel using Hilbert transform. Hilbert transform can accurately extract the instantaneous phase of the signal from the speech frame. Step S2022: Construct the instantaneous phase difference matrix between adjacent microphone channels. Step S2023: Calculate the variance of the instantaneous phase difference within a preset time window. Step S2024: Establish a phase stability index. Step S2025: Generate a preliminary detection flag by combining the phase difference dynamic threshold.

[0034] Specifically, step S2021 can be described as extracting the instantaneous phase of each microphone channel through high-frequency sampling and analytical signal transformation. Assume the array contains I microphones, at the sampling time... With sampling frequency The (Hz) acquired signal satisfies the Nyquist sampling theorem, the... The sampled values ​​(energy) of each channel are To capture the phase nonlinear fluctuations caused by wind noise, the Hilbert transform is used to convert the real signal into an analytic signal, thereby calculating the instantaneous phase: ,in This represents the Hilbert transform operation. By extracting complex arguments and instantaneous phases, the small disturbances in sound wave propagation can be sensitively reflected, especially the incoherent phase changes caused by low-frequency wind noise.

[0035] For example, a multi-microphone array can contain I=3 microphones, with adjacent microphone pairs being (1,2) and (2,3). At sampling time n=100: the instantaneous phase of each microphone channel can be... =0.5404, =0.4200, =0.5000.

[0036] Step S2022 can be described as follows: Based on the instantaneous phase, construct the instantaneous phase difference matrix of adjacent microphone pairs. In a multi-microphone array, define the set of adjacent microphone pairs, I = 3 microphones, with adjacent microphone pairs being (1,2) and (2,3). For each sampling time n, calculate the phase difference. and form a matrix Its dimension is determined by the number of adjacent microphones. For example: =0.5404 0.4200 = 0.1204 =0.4200 0.5000= 0.0800, therefore the instantaneous phase difference matrix is: =[0.1204, 0.0800], (dimension 1×2, corresponding to two adjacent pairs).

[0037] As can be seen, since the phase difference of pure speech usually exhibits spatial coherence, while wind noise introduces random perturbations, this matrix captures phase anomalies in all directions, providing multi-dimensional features for detection.

[0038] Step S2023 can be described as analyzing the variance distribution characteristics of the instantaneous phase difference in the instantaneous phase difference matrix to quantify the nonlinear fluctuations caused by wind noise. The variance of each adjacent pair in the instantaneous phase difference matrix is ​​calculated within a time window of length L centered at the sampling time m (m ∈ n). The sampling time index of the time window is: Variance is defined as: ,in It is the average phase difference within the time window.

[0039] For example: if L=5 and sampling time m=100, the sampling time index n=98,99,100,101,102 within the time window is obtained. For adjacent pairs (1,2), the phase difference within the time window is assumed to be: [0.1204,0.1300,0.1500,0.1400,0.1600]. =0.1401, variance=0.0001968. Similarly, for adjacent pairs (2,3), variance=0.0001500.

[0040] It can be seen that the larger the variance value, the more drastic the phase difference fluctuation at sampling time m, and the higher the probability of wind noise. Compared with traditional energy detection, this method has higher sensitivity to low-frequency wind noise and can identify weak wind noise components.

[0041] Step S2024 can be described as establishing a phase stability index evaluation model based on the variance of sampling time m, and constructing the phase stability index by integrating the variances of all adjacent pairs: ,in This is a scaling parameter used to adjust the response speed of the phase stability index to fluctuation amplitude. Phase stability index The range of values ​​can be The closer the value is to 1, the more stable the phase of sampling time m is, that is, the weak wind noise component; the closer the value is to 0, the more unstable the phase of sampling time m is, and the wind noise dominates.

[0042] For example: assuming =200, sampling time m=100 =exp(-200*(0.0001968+0.0001500))=0.933, close to 1, indicating that the phase at sampling time m is relatively stable and the wind noise is weak. If the wind noise increases, causing the variance to increase to 0.010, then: =0.135, the exponent is close to 0, indicating that the phase is highly unstable and wind noise is dominant.

[0043] It can be seen that the phase stability index provides a quantitative basis for subsequent wind noise frame recognition and can effectively distinguish the phase characteristics of wind noise and speech.

[0044] Step S2025 can be described as generating wind noise detection markers by combining a dynamic phase difference threshold, and introducing a dynamic phase difference threshold: ,in, Representing the The dynamic threshold (cumulative) of the phase difference for wind noise detection corresponding to each speech frame (or time frame) is the core criterion for determining whether a speech frame is a wind noise frame. For threshold coefficient, The number of historical frames is dynamically adjusted based on the historical index average. Representing the The phase stability index of a historical speech frame, iterating through k from 1 to M, when... When a speech frame is determined to be a wind noise-dominated frame, a detection flag is generated. In other words, it's a preliminary detection marker; otherwise, it's a speech-dominant frame. .

[0045] For example: =0.8, k=3, sampling time m=100, three historical records The values ​​are 0.93, 0.94, and 0.95 respectively. The historical index mean is 0.94, and the dynamic threshold for phase difference is 0.8 * 0.94 = 0.752, corresponding to the speech frame at sampling time m = 100. =0.933, and since 0.933 ≥ 0.752, the speech frame at sampling time m=100 is determined to be the dominant speech frame, and a preliminary detection flag is generated. .

[0046] As can be seen, this indicator incorporates a dynamic phase difference threshold, enabling it to adapt to non-stationary wind noise changes and avoid misjudgments. The resulting phase difference features and detection indicator will serve as the core input for fusing multi-channel signal energy distribution and phase difference dynamics in step two. In step S202, this scheme constructs a wind noise detection mechanism based on instantaneous phase difference. It extracts the instantaneous phase of each microphone channel corresponding to the microphone array using a preset frequency (high-frequency sampling) and Hilbert transform, calculates the instantaneous phase difference matrix between adjacent microphone pairs, and then analyzes its variance distribution characteristics. Compared with traditional energy detection methods, this mechanism has significantly higher sensitivity to low-frequency wind noise and can effectively identify weak wind noise components that are difficult to detect using traditional methods.

[0047] In some embodiments, reference Figure 4 The phase stability index output in step S202 and preliminary detection marks Based on this, the short-term energy distribution characteristics of each microphone channel at each acquisition time are further integrated to improve recognition accuracy. Specifically, in the speech frame Calculate the instantaneous energy of each microphone channel at sampling time m. And construct an energy fluctuation index: ,in The average energy of the channel, this index reflects the degree of energy fluctuation caused by wind noise, will and The comprehensive discrimination value (wind noise intensity assessment value) is obtained by weighted linear combination. Weighting coefficient and The weighting coefficients were pre-calibrated based on experimental data. and Adjustments can also be made based on the preliminary detection marks, so that... It is significantly higher than that in the wind noise-dominated frame than in the speech-dominated frame.

[0048] Set the dynamic threshold for wind noise intensity: ,in For historical frames Moving average of values and To adjust the parameters, when At that time, the voice frame was finally confirmed. (Sampling time m) is the wind noise dominant frame; otherwise, it is the speech dominant frame. The wind noise dominant frame corresponds to the wind noise dominant frame flag, and the speech dominant frame corresponds to the speech dominant frame flag. The wind noise dominant frame flag and the speech dominant frame flag can also be called the second detection flag.

[0049] For example, continuing with the explanation using I=3, the weights are preset. =0.6, =0.4, adjust parameter =0.7, =0.05, k=3, sampling time m=100, E 1 (100) = 0.25, E 2 (100) = 0.30, E 3 (100) = 0.35, then =0.30, Energy Fluctuation Index =0.1111. =0.135, thus obtaining =0.6*(1-0.135)+0.4*0.1111=0.563. Assuming the moving average of the three historical frames is 0.14, then the dynamic threshold for wind noise intensity = 0.7×0.14+0.05=0.098+0.05=0.148. Because 0.563>0.148, that is, Finally, the voice frame was confirmed. (Sampling time m) is the wind noise dominant frame.

[0050] As can be seen, step S202 fully utilizes the high sensitivity of phase difference to wind noise and the response of energy distribution to sudden interference, achieving frame classification for speech-dominant and wind noise-dominant frames without the need for complex machine learning models. The identified wind noise-dominant frames are used as input for adaptive gain control in step S203. Based on the phase stability index and preliminary detection flag output from step S202, an energy fluctuation index is constructed by fusing short-term energy distribution features of multi-channel signals. A comprehensive discriminant is obtained through weighted linear combination, and a dynamic threshold for wind noise intensity is set to accurately distinguish between wind noise-dominant and speech-dominant frames. This recognition function fully utilizes the high sensitivity of phase difference to wind noise and the rapid response of energy distribution to sudden interference, achieving robust frame classification without the need for complex machine learning models, significantly reducing computational complexity and hardware resource consumption. The adaptive dynamic threshold adjustment mechanism effectively addresses the energy mutation problem in non-stationary wind noise scenarios, avoiding common misjudgment phenomena in traditional methods. The accurate frame recognition results provide a reliable target for subsequent gain control and spectrum restoration, ensuring that processing resources can be focused on the wind noise areas that truly require intervention.

[0051] In some embodiments, for step S203, refer to Figure 5 Based on the wind noise dominant frame output in step S202, gain adjustment processing is performed on the wind noise dominant frame. According to the wind noise intensity evaluation value of the speech frame, a preset nonlinear attenuation curve is used in the frequency domain to dynamically adjust the gain coefficient of each frequency band. First, the first... The time-domain signal of each microphone channel is arranged according to the... The speech frame is divided into segments, and the time-domain signal is converted into a frequency-domain spectrum using a Fast Fourier Transform (FFT) to obtain the original frequency-domain spectrum without gain adjustment. ,in For frequency band index, corresponding to the first Each frequency band; then, based on the wind noise intensity assessment value of the voice frame, a preset nonlinear attenuation curve is invoked to apply it to each frequency band. Gain coefficient Dynamic adjustment (auditory perception weight correction) is performed, including reducing the gain coefficient of the wind noise-sensitive frequency band in the wind noise-dominant frame and retaining the gain coefficient of the speech core frequency band in the speech-dominant frame, and calculating the spectrum after gain adjustment. To avoid auditory distortion caused by sudden changes in gain between frames, the gain coefficients of adjacent speech frames can also be adjusted. Inter-frame smoothing is performed, and the gain coefficient is corrected by weighted averaging and then recalculated (gain application) to finally obtain the frequency-domain adaptive gain-adjusted and inter-frame smoothed spectrum. (Using actual gain to generate enhanced frequency domain spectrum). For audio frame indexing, For frequency bands.

[0052] For example, the time-domain signal l of the i-th microphone channel corresponding to the wind noise-dominant frame (sampling time m) is subjected to a Fast Fourier Transform to obtain the frequency-domain spectrum: =FFT{l}, where k is the frequency band index. The wind noise intensity assessment value here (refer to the comprehensive discrimination value) Based on the wind noise intensity assessment value, a nonlinear attenuation curve is obtained by mapping, and the gain coefficient of the k-band is adjusted. For example, comprehensive discriminant measure =0.135 can be assumed to be the wind noise intensity assessment value = 0.135. =0.1, thus obtaining the spectrum after frequency domain adaptive gain adjustment and inter-frame smoothing. = • 0.1. In other words, the higher the wind noise intensity assessment value, the higher the gain coefficient can be, which reduces the wind noise sensitive frequency band carried in the speech frame (m).

[0053] In some embodiments, for step S204, a gain-adjusted and inter-frame-smoothed spectrum is used. Detection flags for wind noise-dominant frames As input, it is processed through a pre-trained deep neural network. To achieve nonlinear correction of spectral distortion. The mapping objective of this network is from... The core of restoring the harmonic structure of pure speech lies in learning the coherent relationships between frequency components in the speech spectrum, thereby intelligently filling in depressions or smoothing out spikes caused by wind noise. The specific restoration process is as follows: ,in, For the repaired spectrum, parameters It is trained using a large number of clean speech samples, enabling the deep neural network to... It can capture the integer harmonic relationships and energy distribution patterns of harmonics. When When wind noise dominates the current frame, the network activates the deep inpainting module to reconstruct the distorted area using adjacent frequency bands and contextual information from previous and subsequent frames; when... If so, then keep it directly. Either leave it unchanged or enhance it to preserve the naturalness of the speech.

[0054] For example: deep neural networks It can be = It can be seen that when That is, when the current frame is dominated by wind noise, the deep neural network... Perform repair. Input spectrum. When m=100, distortion occurs in specific frequency bands: for example, due to wind noise suppression, the amplitude of the 10th frequency band (corresponding to a frequency of about 500Hz) is excessively suppressed to 0.13, while the amplitude of its second harmonic in the 20th frequency band (1000Hz) is 0.8, and the amplitude of its third harmonic in the 30th frequency band (1500Hz) is 0.6.

[0055] It can be seen that the above-mentioned deep neural network is used to perform nonlinear mapping repair on the gain-adjusted and inter-frame-smoothed spectrum output in step S203, referring to... Figure 6 To address the spectral dips or spikes caused by wind noise, this method intelligently reconstructs distorted regions by learning the harmonic structure features of clean speech. Compared to traditional spectral subtraction or Wiener filtering, the deep learning-based restoration method can more naturally reconstruct speech components masked by wind noise, achieving speech restoration while maintaining naturalness and improving the subjective auditory experience. The numerical values ​​described in each step of the above multi-microphone array speech enhancement processing method are exemplary, and this application does not impose specific limitations on the numerical values.

[0056] The present application will be further described below through specific embodiments.

[0057] Example 1: Application Test of the Method of this Application in an In-Vehicle Voice Interaction Scenarios. Test Objective: To verify the wind noise suppression effect and speech preservation of the method of this application in non-stationary strong wind noise environments. Test Conditions: In a vehicle driving scenario with the windows open, the wind speed was set to 8 m / s. Audio was collected using a dual-microphone array, with a time window length L set to 15 milliseconds. Test Steps: Multi-channel audio data containing driver voice commands and strong wind noise was collected. The audio before and after processing was recorded, and quantitative evaluation was performed using PESQ (Perceptual Speech Quality Assessment) and STOI (Short-Term Objective Intelligibility) indicators. Test Results: Before processing, the original audio had a PESQ score of 1.2 and an STOI score of 0.45, indicating that the speech was masked by wind noise. After processing by the method of this application, the PESQ score increased to 3.1, and the STOI score increased to 0.85. Conclusion: This example demonstrates that in strong wind noise environments, the method of this application can effectively suppress wind noise energy while significantly improving speech clarity and intelligibility.

[0058] Example 2: Comparative test of traditional fixed threshold and full DNN inference. Test conditions: Same environment and hardware as Example 1. Test steps: 1. Disable adaptive dynamic threshold and use a fixed energy threshold for wind noise judgment. 2. Disable conditional activation mechanism and perform full inference on all audio frames using the DNN. 3. Record the average system power consumption and false positive rate over 30 minutes. Test results: The average system power consumption soared to 35mW, and the wind noise false positive rate reached 25% because the fixed threshold could not adapt to gust changes. Conclusion: Compared with Example 1, Example 2 showed a significant increase in power consumption, demonstrating that the dual adaptive dynamic threshold and DNN conditional activation mechanism of this application have significant technical advantages in reducing power consumption and improving detection accuracy.

[0059] As can be seen from Examples 1-2 above: First, the method of this application (Example 1) achieves a significant improvement in PESQ and STOI scores under strong wind noise conditions, demonstrating the significant advancement of the multi-dimensional feature fusion and nonlinear repair strategy of this application compared to existing technologies. Second, when using the DNN conditional activation mechanism (Example 2), the system power consumption is controlled at 12mW, while when using full inference (Comparative Example 1), the power consumption reaches as high as 35mW. This demonstrates the specific selection and critical significance of the conditional activation mechanism of this application in reducing power consumption. In summary, the technical solution of this application brings significant technical effects, possessing outstanding substantive features and significant progress.

[0060] The aforementioned conditional activation strategy based on detection markers applies computationally intensive AI inference to the damaged signal segments requiring intervention. In real-world outdoor audio pickup scenarios, wind noise is often intermittent, meaning that the deep network is in a dormant or bypassed state for some time. This mechanism reduces the system's average computational load and power consumption without sacrificing the final speech restoration quality, achieving a balance between algorithm performance and hardware energy efficiency. This makes it possible to deploy deep learning speech enhancement technology on low-power edge devices.

[0061] In some embodiments, this application provides a voice enhancement processing system 70, which includes a multi-channel signal input module 701 and a voice enhancement processing module 702. The voice enhancement processing system 70 can run on smart devices with various configurations of multiple microphone arrays, such as a first microphone and a second microphone. The multi-channel signal input module 701 is used to capture voice frames containing wind noise in a physical sound field in real time, and the voice enhancement processing module 702 is used to run a deep neural network and instruction code to perform various processing on the voice frames.

[0062] The speech enhancement processing module 702 can read speech frames in real time, call instructions for Hilbert transform instantaneous phase extraction and phase difference variance calculation to calculate the phase stability index; simultaneously, it calls instructions for constructing the energy fluctuation index in parallel to complete the weighted fusion of multi-dimensional features. Subsequently, based on the decision of wind noise-dominant frame recognition, it drives the nonlinear attenuation curve to adjust the frequency domain gain, and when necessary, calls the built-in deep neural network to perform nonlinear correction inference. Through this hardware-software co-processing system-level architecture design, this application transforms signal processing logic into a physical hardware product.

[0063] In some embodiments, the voice enhancement processing system can be applied to smart devices where the first microphone is a host microphone and the second microphone is an external microphone. In the smart terminal ecosystem, users often carry the host device while wearing an external device. Traditional noise reduction systems typically use only the microphone on a single device. Combining the host microphone and the external microphone would form a microphone array with a large spatial span; however, cross-device collaboration faces technical challenges: since the host microphone and the external microphone are different hardware products, their acoustic packaging structures, individual microphone sensitivities, etc., differ. If these two voice signals are directly fed into the voice enhancement processing system for calculation, the inconsistency in amplitude and phase between the two voice signals will be mistaken by the algorithm as fluctuations in the sound field environment, leading to calculation errors and affecting the accuracy of wind noise detection.

[0064] To address this technical challenge of cross-device collaboration, the speech enhancement processing system in this embodiment can be configured to perform gain compensation on the speech signals of each microphone before performing speech enhancement processing. Specifically, the noise floor energy difference or transfer function difference between the host microphone and the external microphone is calculated in real time. Based on this difference, a compensation coefficient is dynamically generated to perform amplitude scaling and phase calibration on the speech signals of the external microphones, adjusting their baseline to the same level as the host microphone. Through this gain compensation preprocessing, the system reduces the heterogeneous differences between cross-device hardware, enabling different devices to be integrated into a collaborative array, thus broadening the application scenarios of the wind noise suppression technology of this application.

[0065] In some embodiments, the voice enhancement processing system can be applied to at least one of smart home voice interaction devices, in-vehicle voice interaction terminals, hearing aid devices, or outdoor communication headphones. Leveraging its sensitivity to low-frequency weak wind noise, robustness of dual adaptive dynamic thresholds, and low-power characteristics of deep neural network conditional activation, the voice enhancement processing system meets the needs of smart hardware. Specifically, when applied to smart home voice interaction devices, the voice enhancement processing can effectively filter out interference from indoor drafts or strong winds from air conditioners and fans on the wake-up word, improving the far-field voice wake-up rate. When applied to in-vehicle voice interaction terminals, it can quickly respond to unstable wind noise entering through open windows while the vehicle is in motion, ensuring that the driver's voice commands are clearly recognized. When applied to hearing aid devices, given the stringent requirements for power consumption and latency, it filters outdoor wind noise for hearing-impaired individuals while controlling power consumption. When applied to outdoor communication headphones, it can suppress wind noise from continuous strong oncoming winds and simultaneously repair speech harmonics affected by wind noise, ensuring the quality of the collected voice.

[0066] In some embodiments, this application provides an edge AI chip with a built-in hardware accelerator configured to execute a voice enhancement processing method using a multi-microphone array as described in any of the foregoing embodiments. The aim is to solidify the core algorithm of this application at the underlying hardware level of semiconductor integrated circuits. In the consumer electronics industry chain, core algorithms are typically hardened or semi-hardened into dedicated system-on-a-chip (SoC). This ultra-low-power edge AI chip is characterized by its integrated dedicated hardware accelerator. This hardware accelerator is a dedicated integrated circuit IP core customized for the aforementioned four-layer progressive processing logic. On the physical layout, the accelerator implements mathematical operations such as Hilbert transform, phase difference matrix construction, variance calculation, and multi-dimensional feature fusion at the hardware level through customized digital logic gates and register-transfer level data paths. Because these operations are solidified into a dedicated hardware pipeline, their execution efficiency is high, and dynamic power consumption is reduced. This design, which integrates wind noise detection and suppression algorithms into the underlying silicon microarchitecture, gives the AI ​​chip the performance to process multi-channel audio in real time under a low power budget.

[0067] like Figure 8 As shown, the present invention provides a method for running Figure 7 The electronic device 80 of the speech enhancement processing system 70 for a multi-microphone array shown may include a memory 801, a processor 802, and a bus, and may also include a computer program stored in the memory 801 and executable on the processor 802, such as... Figure 7 The various functional modules of the multi-microphone array voice enhancement processing system 70 shown.

[0068] The memory 801 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 801 can be an internal storage unit of the electronic device 80, such as the portable hard drive of the electronic device 80. In other embodiments, the memory 801 can also be an external storage device of the electronic device 80, such as a plug-in portable hard drive, SmartMediaCard (SMC), SecureDigital (SD) card, FlashCard, etc., equipped on the electronic device 800. Furthermore, the memory 801 can include both internal and external storage units of the electronic device 80. The memory 801 can be used not only to store application software and various types of data installed on the electronic device 80, such as the task distribution code of a multi-microphone array voice enhancement processing system, but also to temporarily store data that has been output or will be output.

[0069] In some embodiments, the processor 802 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 802 is the control unit of the electronic device 80, connecting various components of the electronic device 80 through various interfaces and lines. It executes programs or modules stored in the memory 801 (such as the control program for a multi-microphone array voice enhancement processing system) and calls data stored in the memory 801 to perform various functions and process data of the electronic device 80.

[0070] The processor 802 executes the operating system of the electronic device 80 and various installed application programs. The processor 802 executes the application programs to implement the steps in the performance testing method of the electronic device for the aforementioned multi-microphone array voice enhancement processing system.

[0071] For example, a computer program may be divided into one or more modules, one or more of which are stored in memory 801 and executed by processor 802 to complete this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in electronic device 80.

[0072] In one embodiment, a storage medium is provided on which a computer program is stored, and when the computer program is executed by a processor, it can also perform the steps described above when the processor executes the computer program.

[0073] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A speech enhancement processing method using a multi-microphone array, characterized in that, include: The multi-microphone array is used to collect voice frames at each sampling time at a preset frequency; The speech frame is processed to obtain the phase stability index and the energy fluctuation index. Based on the phase stability index and the energy fluctuation index, the wind noise intensity assessment value is calculated. The gain of the speech frame is dynamically adjusted based on the wind noise intensity assessment value to obtain the gain-adjusted speech frame. The speech frame is determined based on the phase stability index, the energy fluctuation index, and the wind noise intensity assessment value: When the speech frame is a wind noise-dominated frame, the gain-adjusted speech frame is corrected by a pre-trained deep neural network to obtain the repaired speech frame.

2. The method according to claim 1, characterized in that, The step of performing speech processing on the speech frame to obtain the phase stability index includes: The instantaneous phase of the speech frame in each microphone channel of the multi-microphone array is extracted using Hilbert transform; The variance corresponding to the instantaneous phase difference within a preset time window is calculated to obtain the phase stability index of the speech frame.

3. The method according to claim 1, characterized in that, The step of performing speech processing on the speech frame to obtain the energy fluctuation index includes: Calculate the instantaneous energy of the speech frame in each microphone channel of the multi-microphone array, and the channel average energy of the speech frame in each microphone channel. The energy fluctuation index of the speech frame is determined based on the instantaneous energy and the channel average energy.

4. The method according to claim 1, characterized in that, The calculation of the wind noise intensity assessment value based on the phase stability index and the energy fluctuation index includes: The wind noise intensity assessment value at each sampling time is obtained by linearly weighting and fusing the phase stability index and the energy fluctuation index.

5. The method according to claim 1, characterized in that, The determination of the speech frame based on the phase stability index, the energy fluctuation index, and the wind noise intensity assessment value includes: The current dynamic threshold of phase difference is calculated based on the phase stability index; wherein the dynamic threshold of phase difference is calculated according to the following formula. m represents the sampling time of the voice frame based on the preset frequency. Indicates the first Phase stability index of speech frames at each sampling time. Indicates the first The dynamic threshold of phase difference corresponding to each sampling time point of the speech frame. Represents the threshold coefficient. The number of historical audio frames. Representing the Phase stability index of a historical speech frame; The phase stability index is determined based on the current dynamic threshold of phase difference: when At that time, it was initially determined that the voice frame was a wind noise-dominated frame.

6. The method according to claim 1, characterized in that, The determination of the speech frame based on the phase stability index, the energy fluctuation index, and the wind noise intensity assessment value further includes: The current dynamic threshold of wind noise intensity is calculated based on the wind noise intensity assessment value, wherein the dynamic threshold of wind noise intensity is calculated according to the following formula. ， m represents the sampling time of the voice frame based on the preset frequency. Indicates the first Wind noise intensity assessment value of speech frames at each sampling time. The moving average of the wind noise intensity assessment values ​​of the historical voice frames corresponding to the given voice frame is used. and To adjust the parameters; The wind noise intensity assessment value is determined based on the current dynamic threshold of wind noise intensity: when At that time, the voice frame was ultimately determined to be a wind noise-dominated frame.

7. The method according to claim 1, characterized in that, The step of dynamically adjusting the gain of the speech frame based on the wind noise intensity assessment value to obtain the gain-adjusted speech frame includes: Convert the time-domain signal of the speech frame into a frequency-domain spectrum; Based on the wind noise intensity assessment value of the speech frame, a preset nonlinear attenuation curve is determined, and the gain coefficient of each frequency band in the frequency domain spectrum is dynamically adjusted to obtain the speech frame after gain adjustment.

8. The method according to claim 1, characterized in that, When the speech frame is a wind noise-dominated frame, the gain-adjusted speech frame is corrected using a pre-trained deep neural network to obtain the repaired speech frame, including: Using the gain-adjusted speech frame as input, the pre-trained deep neural network is used to perform nonlinear correction of spectral distortion on the speech frame to obtain the repaired speech frame.

9. A speech enhancement processing system with a multi-microphone array, characterized in that, include: The system includes a multi-channel signal input module and a speech enhancement processing module. The multi-channel signal input module is used to acquire speech frames at various sampling times via a multi-microphone array at a preset frequency. The speech enhancement processing module performs speech processing on the speech frames to obtain a phase stability index and an energy fluctuation index. Based on the phase stability index and the energy fluctuation index, a wind noise intensity assessment value is calculated. The gain of the speech frames is dynamically adjusted according to the wind noise intensity assessment value to obtain a gain-adjusted speech frame. Based on the phase stability index, the energy fluctuation index, and the wind noise intensity assessment value, the speech frames are judged: if the speech frame is a wind noise-dominant frame, the gain-adjusted speech frame is corrected using a pre-trained deep neural network to obtain a repaired speech frame.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a computer program, implements the speech enhancement processing method of the multi-microphone array as described in any one of claims 1 to 8.

Images