Neural network-based dual-channel speech enhancement method, device, equipment and medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN JIAYZ PHOTO IND LTD
- Filing Date
- 2026-04-10
- Publication Date
- 2026-06-12
Smart Images
Figure CN122201325A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of speech data processing technology, and in particular to a method, apparatus, device and medium for dual-channel speech enhancement based on neural networks. Background Technology
[0002] With the widespread use of portable smart terminals, wearable audio devices and industrial voice interaction terminals, voice interaction has become one of the core human-computer interaction methods. However, environmental noise in complex usage environments can severely degrade the quality of voice signals, leading to a decrease in voice recognition rate and a worse voice communication experience.
[0003] Traditional speech enhancement methods typically employ speech enhancement algorithms for speech denoising. However, these methods have limitations in both the quality and efficiency of speech denoising and speech enhancement. This is especially true for embedded systems with limited computing power, where the deployment of network models consumes significant resources. Furthermore, for medium to large-scale networks, it is difficult to run in real-time within embedded systems, making it challenging to effectively perform speech denoising or speech enhancement in embedded system application scenarios. Summary of the Invention
[0004] This invention provides a method, apparatus, device, and medium for dual-channel speech enhancement based on neural networks, in order to improve the efficiency and quality of speech enhancement.
[0005] According to one aspect of the present invention, a dual-channel speech enhancement method based on a neural network is provided, the method comprising: Acquire noisy speech data from the first speech channel and pure noise data from the second speech channel; Determine the speech power spectrum of the noisy speech data, and generate a Mel power spectrum based on the speech power spectrum; The Mel power spectrum is input into the trained Melwiener filter generation model to obtain the Melwiener filter output by the model. Determine the noise power spectrum based on the pure noise data, and determine the reference Wiener filter based on the noise power spectrum and the speech power spectrum; A hybrid Wiener filter is generated based on the Melwiener filter and the reference Wiener filter; The hybrid Wiener filter is used to denoise the noisy speech data in order to enhance the speech of the noisy speech data.
[0006] According to another aspect of the present invention, a neural network-based dual-channel speech enhancement device is provided, the device comprising: The data acquisition module is used to acquire noisy speech data from the first speech channel and pure noise data from the second speech channel. A Mel power spectrum generation module is used to determine the speech power spectrum of the noisy speech data and generate a Mel power spectrum based on the speech power spectrum. The Wiener filter generation module is used to input the Mel power spectrum into the trained Mel-Wiener filter generation model to obtain the Mel-Wiener filter output by the model. A reference filter generation module is used to determine the noise power spectrum based on the pure noise data, and to determine a reference Wiener filter based on the noise power spectrum and the speech power spectrum. A hybrid filter generation module is used to generate a hybrid Wiener filter based on the Melwiener filter and the reference Wiener filter; The speech enhancement module is used to perform noise reduction processing on the noisy speech data using the hybrid Wiener filter, so as to achieve speech enhancement of the noisy speech data.
[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the neural network-based dual-channel speech enhancement method according to any embodiment of the present invention.
[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the neural network-based dual-channel speech enhancement method according to any embodiment of the present invention.
[0009] The technical solution of this invention determines the speech power spectrum of noisy speech data, generates a Mel power spectrum based on the speech power spectrum, inputs the Mel power spectrum into a trained Mel-Wiener filter generation model to obtain the output Mel-Wiener filter, determines the noise power spectrum based on the pure noise data, and determines a reference Wiener filter based on the noise power spectrum and the speech power spectrum. A hybrid Wiener filter is generated based on the Mel-Wiener filter and the reference Wiener filter, and the hybrid Wiener filter is used to denoise the noisy speech data to achieve speech enhancement. The above technical solution acquires noisy speech and pure noise through dual speech channels, processes the data from both channels separately, and combines the nonlinear processing capability of the model with the effective combination of the traditional Wiener filter. This retains both the denoising advantages of the network and the denoising directionality, avoiding the reliability issues caused by traditional single-channel processing that only processes human voice while ignoring directional or environmental noise. It solves the problem that existing technologies struggle to remove transient noise and introduce new speech distortion and noise residue, improving the quality of speech enhancement while ensuring the efficiency of the speech denoising process.
[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description
[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a flowchart of a dual-channel speech enhancement method based on a neural network according to Embodiment 1 of the present invention; Figure 2A This is a flowchart of a dual-channel speech enhancement method based on a neural network according to Embodiment 2 of the present invention; Figure 2B This is a schematic diagram of the model structure of a Melvina filter generation model provided in Embodiment 2 of the present invention; Figure 3 This is a flowchart of a dual-channel speech enhancement method based on a neural network according to Embodiment 3 of the present invention; Figure 4 This is a flowchart of a dual-channel speech enhancement method based on a neural network according to Embodiment 4 of the present invention; Figure 5This is a schematic diagram of the structure of a dual-channel speech enhancement device based on a neural network according to Embodiment 5 of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device that implements the neural network-based dual-channel speech enhancement method according to an embodiment of the present invention. Detailed Implementation
[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0015] Example 1 Figure 1 This is a flowchart of a neural network-based dual-channel speech enhancement method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where speech data containing environmental noise, such as human voice noise from different directions, is subject to noise reduction or enhancement. The method can be executed by a neural network-based dual-channel speech enhancement device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes: S110: Acquire noisy speech data from the first speech channel and pure noise data from the second speech channel.
[0016] S120. Determine the speech power spectrum of the noisy speech data, and generate the Mel power spectrum based on the speech power spectrum.
[0017] S130. Input the Mel power spectrum into the trained Melwiener filter generation model to obtain the Melwiener filter output by the model.
[0018] S140. Determine the noise power spectrum based on the pure noise data, and determine the reference Wiener filter based on the noise power spectrum and the speech power spectrum.
[0019] S150. Generate a hybrid Wiener filter based on the Melwiener filter and the reference Wiener filter.
[0020] S160. A hybrid Wiener filter is used to denoise the noisy speech data in order to enhance the speech of the noisy speech data.
[0021] The speech data in this embodiment includes two types of data, acquired through a dual-channel method: a first speech channel and a second speech channel. The first speech channel is used to acquire noisy speech data and can be a main microphone channel; the second speech channel is used to acquire pure noise data and can be a reference microphone channel. The pure noise data can be directional noise or ambient noise, etc. The noisy speech data can be speech data containing noise, specifically discrete audio points; the pure noise data is noise data containing only noise, such as directional noise or ambient noise, also in the form of discrete audio points.
[0022] The speech power spectrum of the noisy data is determined, and a Mel power spectrum is generated based on the speech power spectrum. Relevant parameters, such as frequency point size, frame overlap rate, and Hamming window function, can be pre-configured before generating the speech power spectrum. For example, the noisy data can be overlaid according to the frame overlap rate, and frames can be segmented based on the frequency point size to eliminate discontinuities or breaks between frames. Each frame of data can be multiplied by a Hamming window to reduce spectral leakage.
[0023] In an optional embodiment, determining the speech power spectrum of the noisy speech data and generating a Mel power spectrum based on the speech power spectrum includes: Step a1: Perform a short-time Fourier transform on the noisy speech data according to the preset frequency point to obtain the transformed frequency domain speech data.
[0024] Step a2: Generate the speech power spectrum based on the transformed frequency domain speech data.
[0025] Step a3: Determine the upper and lower limits of the Mel scale, determine the Mel interpolation amount based on the upper and lower limits of the Mel scale, and generate the Mel scale spectrum based on the Mel interpolation amount.
[0026] Step a4: Generate linear frequencies based on the Mel scale spectrum, and generate a Mel filter bank based on the linear frequencies.
[0027] Step a5: Generate the Mel power spectrum based on the speech power spectrum and the Mel filter bank.
[0028] The preset frequency k can be pre-set by relevant technical personnel. For example, it can be set to 512, meaning 512 frequency points constitute one frame. Assuming the noisy data is denoted as noisy(n), then n represents discrete data points in the time domain. If the frame overlap rate is set to 50%, then 256 constitute one frame.
[0029] Based on a preset frequency point k, a short-time Fourier transform is performed on the noisy(n) speech data to generate transformed frequency domain speech data. The specific implementation method is as follows: in, This represents the short-time Fourier transform function; k represents the frequency point, and its value can be set to 512; The transformed frequency domain speech data is obtained by short-time Fourier transform, i.e., frequency domain data points.
[0030] Based on transformed frequency domain speech data Generating speech power spectrum : Predetermined frequency resolution For example, resolution Determine the frequency f corresponding to each frequency point: Where k can take values from 1 to 512 or from 0 to 511, that is, 512 frequency points. That is, each frequency point corresponds to one frequency.
[0031] Define upper and lower limits for the frequency, and determine the upper and lower limits for the Mel scale based on these limits. Specifically, the Mel scale... The determination method can be as follows: Assuming the upper limit of frequency is set to 20Hz and the lower limit is set to 8000Hz, then the corresponding upper limit of the Mel scale is... and the lower limit of the Mel scale They are respectively: According to the upper limit of the Mel scale and the lower limit of the Mel scale Based on the preset number of filter banks n, the Mel interpolation amount is determined. : The number of filter banks, n, can be set to 32.
[0032] Based on Mel interpolation To generate a Mel-scale spectrum, assuming there are 32 filter banks, a 32-dimensional Mel-scale spectrum will be generated. . and / The absolute difference between them is the Mel difference. .
[0033] Converting from Mel scale to frequency, specifically, based on the Mel scale spectrum. Convert back to linear frequency : Where m ranges from 1 to 32, representing the number of filters.
[0034] According to linear frequency Generate filter matrix coefficients : Based on the filter matrix coefficient determination function described above, the filter matrix coefficients of 32×512 dimensions can be determined. It should be noted that, due to the symmetry of the audio data points, only the first 257 coefficients need to be considered in the calculation; the rest are symmetrical.
[0035] Power spectrum of noisy speech After the above filter matrix coefficients The transformation yields the Mel power spectrum, reduced from 257 dimensions to 32 dimensions. : The NFFT value is 257.
[0036] The above technical solution converts the noisy speech data from the time domain to the frequency domain and calculates the power spectrum by performing a short-time Fourier transform. Combined with a 32-dimensional Mel filter bank that conforms to the auditory characteristics of the human ear from 20Hz to 8000Hz, the 257-dimensional effective frequency domain power spectrum is reduced to a 32-dimensional Mel domain. Only the low-computational parameters of the 32-dimensional effective frequency points are retained, which greatly reduces the computational power and power consumption and has high single-frame execution efficiency. Combined with the Mel power spectrum generated by the filter matrix coefficients based on Mel cepstrum generation, the accuracy of subsequent model input data generation is improved.
[0037] The Mel-Wina power spectrum is input into a pre-trained Mel-Wina filter generation model to obtain the Mel-Wina filter output by the model. The Mel-Wina filter generation model, used to generate the Mel-Wina filter, can be derived from existing neural network models, such as convolutional neural networks and decision trees.
[0038] The Melvina filter output by the model Interpolating back to 257 dimensions yields a Melvina filter that, after interpolation, maps back from 32 dimensions to the full frequency band: The noise power spectrum of the pure noise data is determined, and a reference Wiener filter is determined based on the noise power spectrum and the speech power spectrum. For example, a traditional dual-channel noise reduction algorithm can be used to generate the reference Wiener filter by combining the noise power spectrum and the speech power spectrum of the pure noise data.
[0039] According to the Melvina filter and reference Wiener filter Generate hybrid Wiener filters : Understandably, this is in reference to the Wiener filter. If transient noise cannot be eliminated, then the Melvina filter... It can compensate; Melvina filter If directional noise cannot be eliminated, refer to the Wiener filter. This can be compensated for. The principle is that when there is transient noise, the Melvina filter output by the model... It will suppress, meaning the coefficient hysteresis will be very small, say 0.01. Assume a Wiener filter as a reference. When transient noise cannot be eliminated, and a result equal to 1 is obtained, Instantaneous noise is suppressed by 10dB, and vice versa. At the same time, since both filters are in the [0, 1] interval, multiplication will become smaller, so taking the square root will make their values more stable.
[0040] A hybrid Wiener filter is used to denoise the noisy speech data to enhance the speech. In one optional embodiment, the denoising process involves: generating denoised frequency domain data based on the hybrid Wiener filter and the transformed frequency domain speech data; and performing an inverse short-time Fourier transform on the denoised frequency domain data to generate enhanced speech data, thereby enhancing the speech of the noisy speech data.
[0041] According to the hybrid Wiener filter Transformed frequency domain speech data with noisy speech data Generate noise-reduced frequency domain data : For noise reduction frequency domain data Perform inverse short-time Fourier transform to generate enhanced speech data. : The technical solution of this invention determines the speech power spectrum of noisy speech data, generates a Mel power spectrum based on the speech power spectrum, inputs the Mel power spectrum into a trained Mel-Wiener filter generation model to obtain the output Mel-Wiener filter, determines the noise power spectrum based on the pure noise data, and determines a reference Wiener filter based on the noise power spectrum and the speech power spectrum. A hybrid Wiener filter is generated based on the Mel-Wiener filter and the reference Wiener filter, and the hybrid Wiener filter is used to denoise the noisy speech data to achieve speech enhancement. The above technical solution acquires noisy speech and pure noise through dual speech channels, processes the data from both channels separately, and combines the nonlinear processing capability of the model with the effective combination of the traditional Wiener filter. This retains both the denoising advantages of the network and the denoising directionality, avoiding the reliability issues caused by traditional single-channel processing that only processes human voice while ignoring directional or environmental noise. It solves the problem that existing technologies struggle to remove transient noise and introduce new speech distortion and noise residue, improving the quality of speech enhancement while ensuring the efficiency of the speech denoising process.
[0042] Example 2 Figure 2A This is a flowchart of a dual-channel speech enhancement method based on neural networks provided in Embodiment 2 of the present invention. This embodiment has been optimized and improved based on the above technical solutions.
[0043] Furthermore, the Melwiener filter generation model includes a first gated recurrent unit, a second gated recurrent unit, a third gated recurrent unit, a first residual fusion unit, a second residual fusion unit, a nonlinear activation unit, a fully connected unit, and a normalization unit. Correspondingly, the step "inputting the Melwiener power spectrum into the trained Melwiener filter generation model to obtain the Melwiener filter output by the model" is refined to "inputting the Melwiener power spectrum into the first gated recurrent unit in the Melwiener filter generation model for core feature extraction to obtain first feature extraction data; inputting the first feature extraction data and the Melwiener power spectrum into the first residual fusion unit for feature fusion to obtain first feature fusion data; inputting the first feature fusion data into..." The first feature extraction data is obtained by inputting the first feature fusion data into a second gated recurrent unit for core feature extraction. This second feature extraction data and the first feature fusion data are then input into a second residual fusion unit for feature fusion, resulting in second feature fusion data. The second feature fusion data is then input into a third gated recurrent unit for core feature extraction, resulting in third feature extraction data. The third feature extraction data is then input into a nonlinear activation unit for nonlinear transformation, resulting in nonlinear activation feature data. The nonlinear activation feature data is then input into a fully connected unit for feature mapping, resulting in feature mapping data. Finally, the feature mapping data is input into a normalization unit for gain normalization, resulting in a Melwiener filter. This process aims to improve the generation method of the Melwiener filter.
[0044] It should be noted that for parts not described in detail in the embodiments of the present invention, please refer to the descriptions in other embodiments. For example... Figure 2A As shown, the method includes the following specific steps: S210. Acquire noisy speech data from the first speech channel and pure noise data from the second speech channel, determine the speech power spectrum of the noisy speech data, and generate a Mel power spectrum based on the speech power spectrum.
[0045] S220. Input the Mel power spectrum into the first gated recurrent unit in the Melwiener filter generation model to extract core features and obtain the first feature extraction data.
[0046] S230. Input the first feature extraction data and the Mel power spectrum into the first residual fusion unit in the Melwiener filter generation model to perform feature fusion and obtain the first feature fusion data.
[0047] S240. Input the first feature fusion data into the second gated recurrent unit in the Melvina filter generation model to extract the core features and obtain the second feature extraction data.
[0048] S250. Input the second feature extraction data and the first feature fusion data into the second residual fusion unit in the Melvina filter generation model to perform feature fusion and obtain the second feature fusion data.
[0049] S260. Input the second feature fusion data into the third gated recurrent unit in the Melvina filter generation model to extract the core features and obtain the third feature extraction data.
[0050] S270. Input the third feature extraction data into the nonlinear activation unit in the Melvina filter generation model for nonlinear transformation to obtain nonlinear activation feature data.
[0051] S280. Input the nonlinear activation feature data into the fully connected unit in the Melvina filter generation model to perform feature mapping, and obtain the feature mapping data.
[0052] S290. Input the feature mapping data into the normalization unit in the Melwiener filter generation model to perform gain normalization, and obtain the Melwiener filter.
[0053] S2100. Determine the noise power spectrum based on the pure noise data, and determine the reference Wiener filter based on the noise power spectrum and the speech power spectrum.
[0054] S2110. Based on the Melwiener filter and the reference Wiener filter, a hybrid Wiener filter is generated. The hybrid Wiener filter is used to denoise the noisy speech data in order to achieve speech enhancement of the noisy speech data.
[0055] like Figure 2B The diagram shows a schematic of the model structure of a Melvina filter generation model. The Melvina filter generation model includes a first gated loop unit, a second gated loop unit, a third gated loop unit, a first residual fusion unit, a second residual fusion unit, a nonlinear activation unit, a fully connected unit, and a normalization unit.
[0056] The first, second, and third gated recurrent units have the same structure, specifically a GRU (Gated Recurrent Unit). The first and second residual fusion units also have the same structure and are used for feature fusion. The nonlinear activation units can be ReLU (Rectified Linear Unit) activation layers; the fully connected units can be FC (Fully Connected) layers; and the normalization units can be Sigmoid activation layers.
[0057] The first gated recurrent unit (GRU) is used to extract features from the input Mel power spectrum data. The GRU is a lightweight recurrent network that extracts transient noise features through three gating mechanisms: a reset gate, an update gate, and a candidate hidden state. The reset gate filters out historically useless noise features; the update gate controls the retention ratio of noise features in the current frame; and the candidate hidden state fuses the input features to generate new noise feature representations. Its core function is to accurately extract the Mel domain features of transient noise from the Mel power spectrum data, output the first feature extraction data, and input the first feature extraction data into the first residual fusion unit.
[0058] The input data of the first residual fusion unit includes the first feature extraction data and the Mel power spectrum data. The role of the first residual fusion unit is to prevent feature loss and avoid gradient vanishing by using residual connections between two gated recurrent units. The first residual fusion unit adds the first feature extraction data and the Mel power spectrum data element-wise to obtain the first feature fusion data, thus preserving the speech features of the Mel spectrum without destroying the human voice, and fusing transient noise features from the first feature fusion data to improve the fitting ability of the small network and reduce embedded inference errors.
[0059] The first feature fusion data is input to the second gated loop unit. The second gated loop unit functions the same as the first gated loop unit, extracting core features from the first feature fusion data to obtain second feature extracted data. This second feature extracted data and the first feature fusion data are then input to the second residual fusion unit. The second residual fusion unit functions the same as the first residual fusion unit, performing feature fusion while retaining transient noise features to obtain second fused feature data. The second fused feature data is then input to the third gated loop unit for core feature extraction to obtain third feature extracted data. The third, second, and first gated loop units have the same structure and function.
[0060] The third feature extraction data is input into the nonlinear activation unit (ReLU activation layer). Its function is to introduce nonlinear changes so that the network can learn more complex noise rules and filter out invalid negative features, reducing the amount of embedded computation, while retaining effective speech or noise features, improving filtering accuracy, and finally obtaining nonlinear activation feature data, that is, ReLU activation features.
[0061] The nonlinear activation feature data is input into a fully connected unit for feature mapping, which maps the extracted noise features into a prototype Wiener filter gain, completing the feature-to-gain conversion and obtaining the feature mapping data. The feature mapping data is then input into a normalization unit (Sigmoid activation layer), which strictly normalizes the gain value to 0~1, and outputs a Mel-Wiener filter that can be directly used for Mel-domain noise reduction calculations.
[0062] For the training process of the Melwiener filter generation model, sample data with standard labels can be pre-constructed. For example, 32-dimensional filter sample data for model training is generated, and label values for the standard Melwiener filters corresponding to each filter sample data are generated. The filter sample data with standard Melwiener filter labels is input into the pre-constructed Melwiener filter generation model to obtain the predicted Melwiener filter output by the model. Based on the standard Melwiener filter and the predicted Melwiener filter from the filter sample data, the Melwiener filter generation model is trained until a preset model training termination condition is met, resulting in a trained Melwiener filter. The model training termination condition can be that the loss value reaches a set threshold, the loss value tends to stabilize, or the number of iterations reaches a set threshold.
[0063] This embodiment's technical solution constructs an embedded small GRU neural network model, employing a 32-dimensional fixed-dimensional computation module that matches the features of the Mel domain and has no redundant computation. This results in high single-frame inference efficiency, adapting to the low computing power and high real-time requirements of embedded devices, thus solving the problem of real-time operation of medium-to-large-scale neural networks in embedded systems. The nonlinear modeling capability of the GRU gated recurrent unit accurately extracts transient noise features within the Mel domain. Combined with an element-wise additive residual connection layer, it effectively captures noise features while preserving the original speech features to the greatest extent. The ReLU activation layer enhances the network's nonlinear feature learning capability, avoiding feature loss and gradient vanishing, and improving the accuracy of noise feature recognition. A fully connected layer achieves accurate mapping of noise features to filter gain, and a Sigmoid activation layer strictly normalizes the output to the Wiener filter standard range, generating a 32-dimensional Mel-Wiener filter. This specifically addresses the poor transient noise suppression capability of traditional denoising algorithms, thereby improving the accuracy of predicting and generating Mel-Wiener filters based on the Mel-Wiener filter generation model.
[0064] Example 3 Figure 3 This is a flowchart of a dual-channel speech enhancement method based on neural networks provided in Embodiment 3 of the present invention. This embodiment has been optimized and improved based on the above technical solutions.
[0065] Furthermore, the step "determine the noise power spectrum based on the pure noise data, and determine the reference Wiener filter based on the noise power spectrum and the speech power spectrum" is refined to "determine the frequency domain noise data of the pure noise data, and generate the noise power spectrum based on the frequency domain noise data; determine the prior signal-to-noise ratio based on the noise power spectrum and the speech power spectrum; and generate the reference Wiener filter based on the prior signal-to-noise ratio." This refines the generation method of the reference Wiener filter.
[0066] It should be noted that for parts not described in detail in the embodiments of the present invention, please refer to the descriptions in other embodiments. For example... Figure 3 As shown, the method includes the following specific steps: S310: Acquire noisy speech data from the first speech channel and pure noise data from the second speech channel.
[0067] S320. Determine the speech power spectrum of the noisy speech data, and generate the Mel power spectrum based on the speech power spectrum.
[0068] S330. Input the Mel power spectrum into the trained Melwiener filter generation model to obtain the Melwiener filter output by the model.
[0069] S340. Determine the frequency domain noise data of the pure noise data, and generate the noise power spectrum based on the frequency domain noise data.
[0070] S350. Determine the prior signal-to-noise ratio based on the noise power spectrum and the speech power spectrum.
[0071] S360. Generate a reference Wiener filter based on the prior signal-to-noise ratio.
[0072] S370. Generate a hybrid Wiener filter based on the Melwiener filter and the reference Wiener filter.
[0073] S380 employs a hybrid Wiener filter to denoise the noisy speech data, thereby enhancing the speech of the noisy speech data.
[0074] Specifically, a frequency point k is preset, for example, it can be set to 512, meaning 512 frequency points constitute one frame, consistent with the parameter configuration of noisy speech data. Assuming the pure noise data is denoted as noise(n), then n represents discrete data points in the time domain. If the frame overlap rate is set to 50%, then 256 points constitute one frame. Based on the preset frequency point k, a short-time Fourier transform is performed on the pure noise data noise(n) to generate frequency domain noise data. The specific implementation method is as follows: in, This represents the short-time Fourier transform function; k represents the frequency point, and its value can be set to 512; This is the frequency domain noise data obtained after short-time Fourier transform.
[0075] Based on frequency domain noise data Generate noise power spectrum : According to the noise power spectrum and speech power spectrum Determine the prior signal-to-noise ratio (SNR). In one optional embodiment, determining the prior SNR based on the noise power spectrum and the speech power spectrum includes: smoothing the noise power spectrum to obtain a smoothed noise power spectrum; determining the posterior SNR based on the speech power spectrum and the smoothed noise power spectrum; and determining the prior SNR based on the speech power spectrum, the smoothed noise power spectrum, and the posterior SNR.
[0076] Smoothed noise power spectrum for the first frame Smooth the noise power spectrum in each subsequent frame starting from the second frame. The posterior signal-to-noise ratio of the first frame The posterior signal-to-noise ratio of each subsequent frame starting from the second frame .
[0077] According to speech power spectrum Smoothing noise power spectrum and posterior signal-to-noise ratio Determine the prior signal-to-noise ratio The prior signal-to-noise ratio of the first frame The reference Wiener filter in the first frame Prior signal-to-noise ratio for each frame from the second frame onwards The expression is as follows: Reference Wiener filter in every frame from the second frame onwards The expression is as follows: The reference Wiener filter is in the range of 0 to 1.
[0078] The technical solution of this invention performs short-time Fourier transform, power spectrum smoothing update, and precise determination of posterior and prior signal-to-noise ratios on the pure noise collected by the reference microphone. By utilizing the noise prior knowledge provided by the reference microphone, it achieves real-time, accurate, and stable suppression of steady-state directional noise, such as wind noise and continuous human voice. The generated reference Wiener filter can complement the model-generated filter, solving the problem that traditional single-channel algorithms cannot distinguish noise sources and have insufficient directional noise suppression, thus improving denoising performance and speech fidelity in complex mixed noise environments.
[0079] Example 4 Figure 4 This is a schematic diagram of the flowchart of a dual-channel speech enhancement method based on a neural network, provided in Embodiment 2 of the present invention. This embodiment provides a preferred example based on the above embodiments.
[0080] like Figure 4 As shown, the method includes the following steps: Step 41: Acquire noisy speech data from the first speech channel. Pure noise data of the second voice channel .
[0081] Step 42: Process the noisy speech data according to the preset frequency points. Perform a short-time Fourier transform to obtain the transformed frequency domain speech data. .
[0082] Step 43: Based on the transformed frequency domain speech data Generate speech power spectrum .
[0083] Step 44: Determine the upper and lower limits of the Mel scale, determine the Mel interpolation amount based on the upper and lower limits, and generate the Mel scale based on the Mel interpolation amount.
[0084] According to the upper limit of the Mel scale and the lower limit of the Mel scale Based on the preset number of filter banks n, the Mel interpolation amount is determined. : Based on Mel interpolation To generate a Mel-scale spectrum, assuming there are 32 filter banks, a 32-dimensional Mel-scale spectrum will be generated. . and / The absolute difference between them is the Mel difference. .
[0085] Step 45: According to the Mel scale spectrum Convert back to linear frequency And according to the linear frequency Generate Mel filter bank .
[0086] Step 45: Based on the speech power spectrum Based on Mel filter bank Generate Mel power spectrum .
[0087] Power spectrum of noisy speech After the above filter matrix coefficients The transformation yields the Mel power spectrum, reduced from 257 dimensions to 32 dimensions. : Step 46: Calculate the Mel power spectrum. Inputting the data into a pre-trained Melvina filter generation model yields the Melvina filter output by the model. .
[0088] Step 47: Convert the Melvina filter output by the model. Interpolating back to 257 dimensions yields a Melvina filter that, after interpolation, is mapped back to the full frequency band from 32 dimensions. .
[0089] Step 48: Determine the frequency domain noise data of the pure noise data. And based on frequency domain noise data Generate noise power spectrum .
[0090] Step 49: Smooth the noise power spectrum to obtain a smoothed noise power spectrum.
[0091] Smoothed noise power spectrum for the first frame Smooth the noise power spectrum in each subsequent frame starting from the second frame. .
[0092] Step 410: Determine the posterior signal-to-noise ratio based on the speech power spectrum and the smoothed noise power spectrum.
[0093] The posterior signal-to-noise ratio of the first frame The posterior signal-to-noise ratio of each subsequent frame starting from the second frame .
[0094] Step 411: Determine the prior signal-to-noise ratio based on the speech power spectrum, the smoothed noise power spectrum, and the posterior signal-to-noise ratio.
[0095] The prior signal-to-noise ratio of the first frame The reference Wiener filter in the first frame Prior signal-to-noise ratio for each frame from the second frame onwards The expression is as follows: Step 412: Generate a reference Wiener filter based on the prior signal-to-noise ratio.
[0096] Reference Wiener filter in every frame from the second frame onwards The expression is as follows: Step 413: Based on the Melvina filter and reference Wiener filter Generate hybrid Wiener filters .
[0097] Step 414: Use a hybrid Wiener filter to perform noise reduction on the noisy speech data in order to achieve speech enhancement of the noisy speech data.
[0098] According to the hybrid Wiener filter Transformed frequency domain speech data with noisy speech data Generate noise-reduced frequency domain data : For noise reduction frequency domain data Perform inverse short-time Fourier transform to generate enhanced speech data. : To ensure real-time performance, the above technical solution introduces a high-efficiency neural network, combining the nonlinear modeling capabilities of a small neural network with a traditional Wiener filter. This retains the noise reduction advantages of the neural network while also providing directional noise reduction. Furthermore, the small neural network and the traditional algorithm are both embedded-friendly and highly portable.
[0099] Example 5 Figure 5 This is a schematic diagram of a neural network-based dual-channel speech enhancement device provided in Embodiment 5 of the present invention. The neural network-based dual-channel speech enhancement device provided in this embodiment is applicable to situations requiring noise reduction or enhancement of speech data containing environmental noise, such as human voice noise from different directions. This neural network-based dual-channel speech enhancement device can be implemented in hardware and / or software, such as... Figure 5 As shown, the device includes: a data acquisition module 501, a Mel power spectrum generation module 502, a Wiener filter generation module 503, a reference filter generation module 504, a hybrid filter generation module 505, and a speech enhancement module 506. Among them, The data acquisition module 501 is used to acquire noisy speech data of the first speech channel and pure noise data of the second speech channel; Mel power spectrum generation module 502 is used to determine the speech power spectrum of the noisy speech data and generate a Mel power spectrum based on the speech power spectrum; The Wiener filter generation module 503 is used to input the Mel power spectrum into the trained Mel-Wiener filter generation model to obtain the Mel-Wiener filter output by the model. The reference filter generation module 504 is used to determine the noise power spectrum based on the pure noise data, and to determine a reference Wiener filter based on the noise power spectrum and the speech power spectrum. The hybrid filter generation module 505 is used to generate a hybrid Wiener filter based on the Melwiener filter and the reference Wiener filter; The speech enhancement module 506 is used to perform noise reduction processing on the noisy speech data using the hybrid Wiener filter, so as to achieve speech enhancement of the noisy speech data.
[0100] The technical solution of this invention determines the speech power spectrum of noisy speech data, generates a Mel power spectrum based on the speech power spectrum, inputs the Mel power spectrum into a trained Mel-Wiener filter generation model to obtain the output Mel-Wiener filter, determines the noise power spectrum based on the pure noise data, and determines a reference Wiener filter based on the noise power spectrum and the speech power spectrum. A hybrid Wiener filter is generated based on the Mel-Wiener filter and the reference Wiener filter, and the hybrid Wiener filter is used to denoise the noisy speech data to achieve speech enhancement. The above technical solution acquires noisy speech and pure noise through dual speech channels, processes the data from both channels separately, and combines the nonlinear processing capability of the model with the effective combination of the traditional Wiener filter. This retains both the denoising advantages of the network and the denoising directionality, avoiding the reliability issues caused by traditional single-channel processing that only processes human voice while ignoring directional or environmental noise. It solves the problem that existing technologies struggle to remove transient noise and introduce new speech distortion and noise residue, improving the quality of speech enhancement while ensuring the efficiency of the speech denoising process.
[0101] Optionally, the Mel power spectrum generation module 502 is specifically used for: The noisy speech data is subjected to a short-time Fourier transform based on a preset frequency point to obtain transformed frequency domain speech data. Based on the transformed frequency domain speech data, a speech power spectrum is generated; Determine the upper and lower limits of the Mel scale, determine the Mel interpolation amount based on the upper and lower limits, and generate the Mel scale spectrum based on the Mel interpolation amount; Based on the Mel scale spectrum, the frequency is converted back to linear frequency, and a Mel filter bank is generated based on the linear frequency. Based on the speech power spectrum, a Mel power spectrum is generated using the Mel filter bank.
[0102] Optionally, the Melvina filter generation model includes a first gated cyclic unit, a second gated cyclic unit, a third gated cyclic unit, a first residual fusion unit, a second residual fusion unit, a nonlinear activation unit, a fully connected unit, and a normalization unit.
[0103] Optionally, the Wiener filter generation module 503 is specifically used for: The Mel power spectrum is input into the first gated recurrent unit in the Melwiener filter generation model to extract core features, thereby obtaining the first feature extraction data. The first feature extraction data and the Mel power spectrum are input into the first residual fusion unit for feature fusion to obtain the first feature fusion data; The first feature fusion data is input into the second gated loop unit for core feature extraction to obtain the second feature extraction data; The second feature extraction data and the first feature fusion data are input into the second residual fusion unit for feature fusion to obtain the second feature fusion data; The second feature fusion data is input into the third gated loop unit for core feature extraction to obtain the third feature extraction data; The third feature extraction data is input into the nonlinear activation unit for nonlinear transformation to obtain nonlinear activation feature data; The nonlinear activation feature data is input into the fully connected unit for feature mapping to obtain feature mapping data; The feature mapping data is input into the normalization unit for gain normalization to obtain the Melvina filter.
[0104] Optionally, the reference filter generation module 504 includes: A noise power spectrum determination unit is used to determine the frequency domain noise data of the pure noise data and generate a noise power spectrum based on the frequency domain noise data. A priori signal-to-noise ratio determination unit is used to determine the priori signal-to-noise ratio based on the noise power spectrum and the speech power spectrum; A reference filter generation unit is used to generate a reference Wiener filter based on the prior signal-to-noise ratio.
[0105] Optional, a priori signal-to-noise ratio determination unit, specifically used for: The noise power spectrum is smoothed to obtain a smoothed noise power spectrum; The posterior signal-to-noise ratio is determined based on the speech power spectrum and the smoothed noise power spectrum. The prior signal-to-noise ratio is determined based on the speech power spectrum, the smooth noise power spectrum, and the posterior signal-to-noise ratio.
[0106] Optional, the voice enhancement module 506 is specifically used for: Based on the hybrid Wiener filter and the transformed frequency domain speech data, noise-reduced frequency domain data is generated; The noise-reduced frequency domain data is subjected to inverse short-time Fourier transform to generate enhanced speech data, thereby achieving speech enhancement of noisy speech data.
[0107] The neural network-based dual-channel speech enhancement device provided in this embodiment of the invention can execute the neural network-based dual-channel speech enhancement method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.
[0108] Example 6 Figure 6 A schematic diagram of an electronic device 60 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0109] like Figure 6 As shown, the electronic device 60 includes at least one processor 61 and a memory, such as a read-only memory (ROM) 62 and a random access memory (RAM) 63, communicatively connected to the at least one processor 61. The memory stores computer programs executable by the at least one processor. The processor 61 can perform various appropriate actions and processes based on the computer program stored in the ROM 62 or loaded from storage unit 68 into the RAM 63. The RAM 63 can also store various programs and data required for the operation of the electronic device 60. The processor 61, ROM 62, and RAM 63 are interconnected via a bus 64. An input / output (I / O) interface 65 is also connected to the bus 64.
[0110] Multiple components in electronic device 60 are connected to I / O interface 65, including: input unit 66, such as keyboard, mouse, etc.; output unit 67, such as various types of monitors, speakers, etc.; storage unit 68, such as disk, optical disk, etc.; and communication unit 69, such as network card, modem, wireless transceiver, etc. Communication unit 69 allows electronic device 60 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0111] Processor 61 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 61 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 61 performs the various methods and processes described above, such as a neural network-based dual-channel speech enhancement method.
[0112] In some embodiments, the neural network-based dual-channel speech enhancement method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 68. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 60 via ROM 62 and / or communication unit 69. When the computer program is loaded into RAM 63 and executed by processor 61, one or more steps of the neural network-based dual-channel speech enhancement method described above can be performed. Alternatively, in other embodiments, processor 61 can be configured to perform the neural network-based dual-channel speech enhancement method by any other suitable means (e.g., by means of firmware).
[0113] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0114] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0115] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0116] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0117] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.
[0118] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.
[0119] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.
[0120] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A dual-channel speech enhancement method based on neural networks, characterized in that, include: Acquire noisy speech data from the first speech channel and pure noise data from the second speech channel; Determine the speech power spectrum of the noisy speech data, and generate a Mel power spectrum based on the speech power spectrum; The Mel power spectrum is input into the trained Melwiener filter generation model to obtain the Melwiener filter output by the model. Determine the noise power spectrum based on the pure noise data, and determine the reference Wiener filter based on the noise power spectrum and the speech power spectrum; A hybrid Wiener filter is generated based on the Melwiener filter and the reference Wiener filter; The hybrid Wiener filter is used to denoise the noisy speech data in order to enhance the speech of the noisy speech data.
2. The method according to claim 1, characterized in that, The step of determining the speech power spectrum of the noisy speech data and generating a Mel power spectrum based on the speech power spectrum includes: The noisy speech data is subjected to a short-time Fourier transform based on a preset frequency point to obtain transformed frequency domain speech data. Based on the transformed frequency domain speech data, a speech power spectrum is generated; Determine the upper and lower limits of the Mel scale, determine the Mel interpolation amount based on the upper and lower limits, and generate the Mel scale spectrum based on the Mel interpolation amount; Based on the Mel scale spectrum, a linear frequency is generated, and a Mel filter bank is generated based on the linear frequency; Based on the speech power spectrum, a Mel power spectrum is generated using the Mel filter bank.
3. The method according to claim 1, characterized in that, The Melvina filter generation model includes a first gated cyclic unit, a second gated cyclic unit, a third gated cyclic unit, a first residual fusion unit, a second residual fusion unit, a nonlinear activation unit, a fully connected unit, and a normalization unit.
4. The method according to claim 3, characterized in that, The step of inputting the Mel power spectrum into the trained Melwiener filter generation model to obtain the Melwiener filter output by the model includes: The Mel power spectrum is input into the first gated recurrent unit in the Melwiener filter generation model to extract core features, thereby obtaining the first feature extraction data. The first feature extraction data and the Mel power spectrum are input into the first residual fusion unit for feature fusion to obtain the first feature fusion data; The first feature fusion data is input into the second gated loop unit for core feature extraction to obtain the second feature extraction data; The second feature extraction data and the first feature fusion data are input into the second residual fusion unit for feature fusion to obtain the second feature fusion data; The second feature fusion data is input into the third gated loop unit for core feature extraction to obtain the third feature extraction data; The third feature extraction data is input into the nonlinear activation unit for nonlinear transformation to obtain nonlinear activation feature data; The nonlinear activation feature data is input into the fully connected unit for feature mapping to obtain feature mapping data; The feature mapping data is input into the normalization unit for gain normalization to obtain the Melvina filter.
5. The method according to claim 1, characterized in that, The step of determining the noise power spectrum based on the pure noise data, and determining the reference Wiener filter based on the noise power spectrum and the speech power spectrum, includes: Determine the frequency domain noise data of the pure noise data, and generate a noise power spectrum based on the frequency domain noise data; The prior signal-to-noise ratio is determined based on the noise power spectrum and the speech power spectrum; A reference Wiener filter is generated based on the prior signal-to-noise ratio.
6. The method according to claim 5, characterized in that, The step of determining the prior signal-to-noise ratio based on the noise power spectrum and the speech power spectrum includes: The noise power spectrum is smoothed to obtain a smoothed noise power spectrum; The posterior signal-to-noise ratio is determined based on the speech power spectrum and the smoothed noise power spectrum. The prior signal-to-noise ratio is determined based on the speech power spectrum, the smooth noise power spectrum, and the posterior signal-to-noise ratio.
7. The method according to claim 2, characterized in that, The step of using the hybrid Wiener filter to perform noise reduction processing on the noisy speech data to achieve speech enhancement of the noisy speech data includes: Based on the hybrid Wiener filter and the transformed frequency domain speech data, noise-reduced frequency domain data is generated; The noise-reduced frequency domain data is subjected to inverse short-time Fourier transform to generate enhanced speech data, thereby achieving speech enhancement of noisy speech data.
8. A dual-channel speech enhancement device based on a neural network, characterized in that, include: The data acquisition module is used to acquire noisy speech data from the first speech channel and pure noise data from the second speech channel. A Mel power spectrum generation module is used to determine the speech power spectrum of the noisy speech data and generate a Mel power spectrum based on the speech power spectrum. The Wiener filter generation module is used to input the Mel power spectrum into the trained Mel-Wiener filter generation model to obtain the Mel-Wiener filter output by the model. A reference filter generation module is used to determine the noise power spectrum based on the pure noise data, and to determine a reference Wiener filter based on the noise power spectrum and the speech power spectrum. A hybrid filter generation module is used to generate a hybrid Wiener filter based on the Melwiener filter and the reference Wiener filter; The speech enhancement module is used to perform noise reduction processing on the noisy speech data using the hybrid Wiener filter, so as to achieve speech enhancement of the noisy speech data.
9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the neural network-based dual-channel speech enhancement method according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the neural network-based dual-channel speech enhancement method according to any one of claims 1-7.