A voice signal reconstruction method, device, equipment and medium

Abstract

The application discloses a speech signal reconstruction method and device, equipment and medium, and relates to the technical field of signal processing. The method comprises the following steps: performing half-wave rectification processing on a target narrowband speech signal to obtain a rectified signal, and performing feature extraction on the rectified signal and the target narrowband speech signal to obtain a first amplitude feature and a second amplitude feature; superimposing the two amplitude features to obtain a superimposed amplitude feature, and obtaining a reconstructed amplitude spectrum and a consonant probability corresponding to the target narrowband speech signal based on the superimposed amplitude feature; determining a target fusion ratio of preset white noise according to the consonant probability, fusing the preset white noise and the reconstructed amplitude spectrum based on the target fusion ratio to obtain a target amplitude spectrum, and performing signal reconstruction based on phase information corresponding to the rectified signal and the target amplitude spectrum. The problem of unstable consonant reconstruction in a complex scene is solved by determining the white noise fusion ratio according to the consonant probability and fusing.

Description

Technical Field

[0001] This invention relates to the field of signal processing technology, and in particular to a method, apparatus, device and medium for reconstructing speech signals. Background Technology

[0002] Currently, due to bandwidth limitations, communication systems have low sampling rates, and the collected speech signals consist only of the fundamental frequency and a small number of low-frequency harmonics, with few or no consonants, which seriously affects speech clarity and intelligibility.

[0003] Current speech signal reconstruction methods are based on harmonic multiple relationships, simulating high-frequency harmonic components by folding / shifting the low-frequency spectrum, and introducing white noise by taking advantage of the fact that the zero-crossing rate of consonants is much higher than that of vowels. However, in noisy scenarios and scenarios with overlapping speech by multiple people (especially overlapping vowel and consonant syllables), the calculated zero-crossing rate deviates significantly from the true value, reducing the accuracy and stability of consonant reconstruction. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a speech signal reconstruction method, apparatus, device, and medium that can determine and fuse white noise based on consonant probabilities, thus solving the problem of unstable consonant reconstruction in complex scenarios. The specific solution is as follows: In a first aspect, this application provides a method for reconstructing speech signals, including: A target narrowband speech signal is acquired, and the target narrowband speech signal is subjected to half-wave rectification to obtain a corresponding rectified signal. Feature extraction is performed on the rectified signal and the target narrowband speech signal respectively to obtain a first amplitude feature corresponding to the rectified signal and a second amplitude feature corresponding to the target narrowband speech signal. The first amplitude feature and the second amplitude feature are superimposed to obtain the corresponding superimposed amplitude feature. Based on the superimposed amplitude feature and the pre-trained target neural network model, the reconstructed amplitude spectrum corresponding to the target narrowband speech signal and the consonant probability corresponding to the target narrowband speech signal are obtained. The consonant probability represents the probability that each frame in the target narrowband speech signal includes a consonant. The target fusion ratio of the preset white noise is determined based on the consonant probability. The preset white noise is then fused with the reconstructed amplitude spectrum based on the target fusion ratio to obtain the corresponding target amplitude spectrum. Finally, the target narrowband speech signal is reconstructed based on the phase information corresponding to the rectified signal and the target amplitude spectrum.

[0005] Optionally, after performing half-wave rectification on the target narrowband speech signal, the process further includes: The rectified signal is pre-emphasized to obtain a corresponding processed signal; Accordingly, the step of extracting features from the rectified signal and the target narrowband speech signal respectively to obtain a first amplitude feature corresponding to the rectified signal and a second amplitude feature corresponding to the target narrowband speech signal includes: Feature extraction is performed on the processed signal and the target narrowband speech signal respectively to obtain the first amplitude feature corresponding to the processed signal and the second amplitude feature corresponding to the target narrowband speech signal.

[0006] Optionally, the step of extracting features from the rectified signal and the target narrowband speech signal respectively to obtain a first amplitude feature corresponding to the rectified signal and a second amplitude feature corresponding to the target narrowband speech signal includes: The rectified signal and the target narrowband speech are subjected to frame-by-frame windowing processing to obtain corresponding first and second signals. The first and second signals are then subjected to Fast Fourier Transform to obtain the first amplitude feature corresponding to the rectified signal and the second amplitude feature corresponding to the target narrowband speech signal.

[0007] Optionally, before obtaining the reconstructed amplitude spectrum corresponding to the target narrowband speech signal and the consonant probability corresponding to the target narrowband speech signal based on the superimposed amplitude features and the pre-trained target neural network model, the method further includes: Acquire a clean human voice signal at a target sampling rate and a background sound signal at the target sampling rate; wherein, the clean human voice signal represents a human voice signal without background sound. The pure human voice signal and the background sound signal are superimposed based on a preset signal-to-noise ratio range, and the superimposed signal is low-pass filtered to obtain target training data. The superimposed signal is determined as target label data, and the initial neural network model is trained using the target label data and the target training data to obtain the target neural network model.

[0008] Optionally, before obtaining the reconstructed amplitude spectrum corresponding to the target narrowband speech signal and the consonant probability corresponding to the target narrowband speech signal based on the superimposed amplitude features and the pre-trained target neural network model, the method further includes: The device computing power and storage space of the target edge device are obtained, and the model structure and model parameter quantity of the target neural network model are determined based on the device computing power and storage space; wherein, the target edge device is the device on which the target neural network is deployed.

[0009] Optionally, obtaining the reconstructed amplitude spectrum and consonant probability of the target narrowband speech signal based on the superimposed amplitude features and the pre-trained target neural network model includes: The superimposed amplitude features are converted into the corresponding target Mel spectrum, and the target Mel spectrum is input into the target neural network model so that the target neural network model outputs the reconstructed amplitude spectrum corresponding to the target narrowband speech signal and the consonant probability corresponding to the target narrowband speech signal.

[0010] Optionally, the step of reconstructing the target narrowband speech signal based on the phase information corresponding to the rectified signal and the target amplitude spectrum includes: Obtain the phase information corresponding to the rectified signal, and multiply the phase information with the target amplitude spectrum to obtain the target frequency domain signal; The target frequency domain signal is subjected to inverse fast Fourier transform to obtain the corresponding target time domain signal, and the target time domain signal is processed by the overlap addition method or the overlap preservation method to complete the signal reconstruction operation of the target narrowband speech signal.

[0011] Secondly, this application provides a speech signal reconstruction apparatus, comprising: The feature extraction module is used to acquire a target narrowband speech signal, perform half-wave rectification on the target narrowband speech signal to obtain a corresponding rectified signal, and extract features from the rectified signal and the target narrowband speech signal respectively to obtain a first amplitude feature corresponding to the rectified signal and a second amplitude feature corresponding to the target narrowband speech signal. The model inference module is used to superimpose the first amplitude feature and the second amplitude feature to obtain the corresponding superimposed amplitude feature, and to obtain the reconstructed amplitude spectrum corresponding to the target narrowband speech signal and the consonant probability corresponding to the target narrowband speech signal based on the superimposed amplitude feature and the pre-trained target neural network model; the consonant probability represents the probability that each frame in the target narrowband speech signal includes a consonant. The signal reconstruction module is used to determine the target fusion ratio of the preset white noise based on the consonant probability, fuse the preset white noise with the reconstructed amplitude spectrum based on the target fusion ratio to obtain the corresponding target amplitude spectrum, and reconstruct the target narrowband speech signal based on the phase information corresponding to the rectified signal and the target amplitude spectrum.

[0012] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor for executing the computer program to implement the aforementioned speech signal reconstruction method.

[0013] Fourthly, this application provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the aforementioned speech signal reconstruction method.

[0014] This application first acquires a target narrowband speech signal, performs half-wave rectification on the target narrowband speech signal to obtain a corresponding rectified signal, and extracts features from the rectified signal and the target narrowband speech signal respectively to obtain a first amplitude feature corresponding to the rectified signal and a second amplitude feature corresponding to the target narrowband speech signal. Then, the first amplitude feature and the second amplitude feature are superimposed to obtain a corresponding superimposed amplitude feature. Based on the superimposed amplitude feature and a pre-trained target neural network model, the reconstructed amplitude spectrum and the consonant probability of the target narrowband speech signal are obtained. The consonant probability represents the probability that each frame in the target narrowband speech signal includes a consonant. Finally, a target fusion ratio of preset white noise is determined according to the consonant probability. Based on the target fusion ratio, the preset white noise is fused with the reconstructed amplitude spectrum to obtain a corresponding target amplitude spectrum. Finally, the target narrowband speech signal is reconstructed based on the phase information corresponding to the rectified signal and the target amplitude spectrum. As can be seen, this application reconstructs the amplitude spectrum and determines the consonant probability simultaneously by using a neural network output. It replaces the traditional zero-crossing rate calculation with data-driven probability estimation, thus avoiding the problem of zero-crossing rate calculation deviation in noisy or multi-person speech overlap scenarios. By determining the white noise fusion ratio based on the consonant probability and fusing it, the amount of white noise injected corresponding to the consonant is precisely controlled by the neural network, which solves the problem of unstable consonant reconstruction in complex scenarios. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of a speech signal reconstruction method disclosed in this application; Figure 2 This is a schematic diagram of the structure of a speech signal reconstruction device disclosed in this application; Figure 3 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] See Figure 1 As shown in the figure, an embodiment of the present invention discloses a speech signal reconstruction method, including: Step S11: Obtain the target narrowband speech signal, perform half-wave rectification on the target narrowband speech signal to obtain the corresponding rectified signal, and extract features from the rectified signal and the target narrowband speech signal respectively to obtain the first amplitude feature corresponding to the rectified signal and the second amplitude feature corresponding to the target narrowband speech signal.

[0019] In this embodiment, the overall process for signal reconstruction is as follows: 1. Preprocessing: The real narrowband data (initial narrowband speech signal) is first resampled to 44.1kHz or 48kHz to obtain the target narrowband speech signal (determined by the model training configuration), and then subjected to half-wave rectification. Whether the processed data is pre-emphasized is determined by the model training configuration.

[0020] 2. Input Feature Extraction: The data is segmented into frames (frame length and frame shift are determined by the model training configuration), windowed, and then subjected to FFT (Fast Fourier Transform) to obtain a narrowband amplitude spectrum (i.e., the first amplitude feature). Whether this narrowband amplitude spectrum is superimposed on the original input narrowband amplitude spectrum (i.e., the second amplitude feature) is determined by the model training configuration. Then, the narrowband amplitude spectrum is multiplied by the Mel domain mapping matrix to obtain the Mel domain amplitude spectrum. Whether the Mel domain amplitude spectrum undergoes logarithmic operations is determined by the model training configuration.

[0021] 3. Model inference: Input the Mel domain amplitude spectrum into the model, and the model forward calculates to obtain the reconstructed narrowband amplitude spectrum (i.e., the reconstructed amplitude spectrum).

[0022] 4. Post-processing: Whether to add the original input amplitude spectrum to the reconstructed narrowband amplitude spectrum is determined by the model training configuration. Finally, the reconstructed narrowband amplitude spectrum is dot-multiplied by the half-wave rectified phase spectrum to obtain the reconstructed complex spectrum, which is then processed by IFFT (Inverse Fast Fourier Transform) and OLA (Over Lap Add) / OLS (Over Lap Save) to obtain the reconstructed speech time-domain waveform.

[0023] In this embodiment, the target narrowband speech signal is a speech signal with a sampling rate lower than the target sampling rate; the reconstructed amplitude spectrum represents the frequency domain amplitude spectrum containing high-frequency component information, which is predicted and generated by the target neural network model, and the high-frequency component is a signal component with a frequency greater than a preset frequency threshold.

[0024] Half-wave rectification involves setting the sample points with amplitudes less than 0 within each frame of the input signal to 0, output = max(input, 0), and vice versa, setting the sample points with amplitudes greater than 0 to 0, output = min(input, 0). The purpose of half-wave rectification is to introduce abrupt amplitude changes, increasing spectral leakage and thus generating harmonic distortion-like components at high frequencies, preparing for vowel reconstruction.

[0025] In this embodiment, after performing half-wave rectification on the target narrowband speech signal, the method further includes: performing pre-emphasis processing on the rectified signal to obtain the corresponding processed signal; Accordingly, the step of extracting features from the rectified signal and the target narrowband speech signal respectively to obtain the first amplitude feature corresponding to the rectified signal and the second amplitude feature corresponding to the target narrowband speech signal includes: extracting features from the processed signal and the target narrowband speech signal respectively to obtain the first amplitude feature corresponding to the processed signal and the second amplitude feature corresponding to the target narrowband speech signal.

[0026] In other words, to increase the energy proportion of high-frequency components, the data after half-wave rectification can be pre-emphasized. Then, features (i.e., the first and second amplitude features) of the original input (i.e., the target narrowband speech signal) and the data after half-wave rectification (i.e., the rectified signal) are extracted, such as the amplitude spectrum / complex spectrum of the FFT narrowband / Mel domain FBANK / Bark domain subbands, MFCC (Mel Frequency Cepstrum Coefficient), etc., or a combination of these features, as input to the model. Typically, the frame length for time-frequency conversion is 10-50ms, with a frame shift of 5-25ms. Because of the vocal cord's sound production characteristics, human voice / singing has temporal continuity, so performing frame stitching on the model input can better model the speech. However, backward frame stitching introduces latency issues, which is usually controlled within 30ms. Forward frame stitching does not present a latency issue, but it should not be excessive. This is because, firstly, the temporal correlation of vocal cord output is not particularly long, and secondly, too many frames can lead to an excessively large input dimension and a large number of model parameters, thus affecting edge deployment. Typically, the total duration after frame stitching should not exceed 100ms, with the specific number of frames determined by the data sampling rate and the frame shift of the time-frequency conversion. Furthermore, the model can choose whether to perform CMVN (Consecutive Mean and Variance Normalization) on the input features. Enabling this makes the model less sensitive to the absolute amplitude of the input data, which is beneficial for model convergence and also advantageous for signals with low volume and small amplitude.

[0027] Here, this embodiment selects an FFT time-frequency transformation with a frame length of 50ms and a frame shift of 25ms to obtain the narrowband amplitude spectrum of the original input and the data after half-wave rectification. Optionally, the narrowband amplitude spectrum of the original input can be superimposed on the narrowband amplitude spectrum of the half-wave rectified data. This is because the low-frequency components of the original input are real and distortion-free, while half-wave rectification introduces some distortion, resulting in a more accurate low-frequency amplitude spectrum after superposition. Then, the narrowband amplitude spectrum is converted into a 128 / 64 / 48-dimensional Mel domain amplitude spectrum to reduce the model input dimension, thereby significantly reducing the number of model parameters and the learning difficulty. Furthermore, the Mel domain conforms to the characteristics of human hearing and has a clear physical meaning. Experimental comparisons show that when the number of model parameters is small, the stability of the Mel domain is better than that of a model that models all narrowband frequency points.

[0028] In addition, feature extraction is performed on the rectified signal and the target narrowband speech signal respectively to obtain the first amplitude feature corresponding to the rectified signal and the second amplitude feature corresponding to the target narrowband speech signal. This includes: performing frame-segmentation and windowing processing on the rectified signal and the target narrowband speech signal respectively to obtain the corresponding first signal and second signal, and performing Fast Fourier Transform on the first signal and the second signal respectively to obtain the first amplitude feature corresponding to the rectified signal and the second amplitude feature corresponding to the target narrowband speech signal.

[0029] In other words, the data is segmented into frames (frame length and frame shift are determined by the model training configuration), windowed, and then subjected to FFT to obtain a narrowband amplitude spectrum (i.e., the first amplitude feature). Whether this narrowband amplitude spectrum is superimposed on the original input narrowband amplitude spectrum (i.e., the second amplitude feature) is determined by the model training configuration.

[0030] Step S12: Superimpose the first amplitude feature and the second amplitude feature to obtain the corresponding superimposed amplitude feature, and obtain the reconstructed amplitude spectrum corresponding to the target narrowband speech signal and the consonant probability corresponding to the target narrowband speech signal based on the superimposed amplitude feature and the pre-trained target neural network model; the consonant probability represents the probability that each frame in the target narrowband speech signal includes a consonant.

[0031] The reconstructed amplitude spectrum represents the full-band amplitude spectrum, including the high-frequency reconstructed component, generated by the target neural network model based on the superimposed amplitude features. The high-frequency reconstructed component represents the amplitude information of the high-frequency bands that were originally missing in the target narrowband speech signal but were generated by the model prediction.

[0032] In this embodiment, before obtaining the reconstructed amplitude spectrum and consonant probability of the target narrowband speech signal based on the superimposed amplitude features and the pre-trained target neural network model, the method further includes: obtaining a clean human voice signal at the target sampling rate and a background sound signal at the target sampling rate; wherein, the clean human voice signal represents a human voice signal without background sound; superimposing the clean human voice signal and the background sound signal based on a preset signal-to-noise ratio range, and performing low-pass filtering on the corresponding superimposed signal to obtain target training data; determining the superimposed signal as target label data, and using the target label data and the target training data to train the initial neural network model to obtain the target neural network model.

[0033] Specifically, the model training process is as follows: First, pure human voice signals (i.e., pure human voice signals) with a sampling rate of 44.1kHz / 48kHz are collected, such as human voices / singing voices recorded in professional recording studios, high-quality / high signal-to-noise ratio TTS synthesized audio, etc.; various music / background sounds / ambient sounds with a sampling rate of 44.1kHz / 48kHz are also collected (i.e., background sound signals). The pure human voice signals can be augmented by pitch shifting. Then, according to the actual application scenario requirements, different human voices and various music / background sounds / ambient noises are superimposed according to a certain type distribution and SNR range. Then, the high-frequency components are removed by low-pass filtering at random cutoff frequencies within a specific range to obtain the original input data for model training. The human voice before low-pass filtering is used as the label for model training.

[0034] For example, in an application scenario where a song with a sampling rate of 16kHz is reconstructed to 44.1kHz, the 44.1kHz vocals / singing are superimposed with various background music according to their type distribution and SNR. Then, a low-pass filter with a cutoff frequency of 8kHz is used. The filtered data serves as the raw input for model training, while the unfiltered data serves as the training labels. In another scenario, where the high-frequency components of everyday conversations with a sampling rate of 4-8kHz are reconstructed to 48kHz, the 48kHz vocals are superimposed with various environmental noises according to their type distribution and SNR to simulate a noisy scene. Additionally, multiple (e.g., 2-3) different vocals are superimposed according to their SNR range to simulate a multi-person conversation scene. Then, for each training data point, a low-pass filter with a randomly selected cutoff frequency from 2-4kHz is used. The filtered data serves as the raw input for model training, while the unfiltered data serves as the training labels.

[0035] In addition, before obtaining the reconstructed amplitude spectrum and consonant probability of the target narrowband speech signal based on the superimposed amplitude features and the pre-trained target neural network model, the method further includes: obtaining the device computing power and device storage space of the target end device, and determining the model structure and model parameter quantity of the target neural network model according to the device computing power and the device storage space; wherein, the target end device is the device that deploys the target neural network.

[0036] In other words, this embodiment offers a wide variety of layer functions to choose from, such as DNN (Dense Neural Network), CNN (Convolutional Neural Network), LSTM (Long Short Term Memory), FSMN (Feedforward Sequential Memory Network), GRU (Gate Recurrent Unit), DCNN (Deconvolutional Neural Network), GMM (Gaussian Mixture Model), and WGMM (Weighted Gaussian Mixture Model), among others, in combination. Note that some layer functions require careful configuration to achieve real-time processing; for example, LSTM / GRU should be configured unidirectionally, and the time-dimensional parameters of 2D CNN / DCNN, such as kernel_size, stride, and padding, need to be considered. Finally, the model structure type and parameter count are determined based on the computing power and storage space available on the edge device.

[0037] Furthermore, obtaining the reconstructed amplitude spectrum and consonant probability of the target narrowband speech signal based on the superimposed amplitude features and the pre-trained target neural network model includes: converting the superimposed amplitude features into a corresponding target Mel spectrum, and inputting the target Mel spectrum into the target neural network model so that the target neural network model outputs the reconstructed amplitude spectrum and consonant probability of the target narrowband speech signal.

[0038] Furthermore, this implementation can also perform logarithmic calculations on the Mel-domain amplitude spectrum before inputting it into the model, which can also reduce the difference in energy levels between low and high frequencies and increase the weight of high-frequency components on the loss function. In this embodiment, the Mel-domain amplitude spectrum (i.e., the target Mel-domain spectrum) is obtained by multiplying the narrowband amplitude spectrum by the Mel-domain mapping matrix. Whether the Mel-domain amplitude spectrum is logarithmically calculated is determined by the model training configuration.

[0039] It should be noted that this solution uses an encoder + enhancer + decoder fusion model. The encoder is typically a one-dimensional or two-dimensional convolutional layer, with 3-6 layers. The two-dimensional convolutional layer has a temporal dimension of 2, including data from one historical frame and the current frame, without involving future data and thus not introducing latency. The encoder is mainly used to extract high-dimensional features in time and frequency, and these features are fed into the enhancer. The enhancer is typically an LSTM or GRU in unidirectional mode (without introducing latency), with 1-2 layers. Using GRU can reduce the number of model parameters, making it easier to deploy on the edge, while maintaining enhancement effects close to LSTM. The enhancer is used to model the temporal changes of features to establish the correlation between the current time and historical time, and its output is fed into the decoder fusion. The decoder fusion typically consists of deconvolutional layers, WGMM, and linear layers. The deconvolutional layers typically have 3-6 layers, the WGMM typically has 16-64 components, and the linear layer typically has 1 layer. The decoder-fusion unit "decodes" the low-dimensional output (e.g., 32 / 48 dimensions) of the enhancer and backtracks it to all narrowband frequency points (thousands of dimensions, such as 1025 dimensions). Furthermore, it estimates the consonant probabilities using a linear layer with an activation function (e.g., sigmoid) and then determines the amount of preset white noise to be fused into the final output. This avoids the problem of poor consonant reconstruction quality caused by zero-crossing rate calculation errors in noisy / multi-person scenarios, a problem encountered in previous solutions.

[0040] The final model has 60,000 to 100,000 parameters, far fewer than large models with millions of parameters. Optionally, the model's output, the reconstructed narrowband amplitude spectrum, can be supplemented with the original input (without half-wave rectification), allowing the model to focus on modeling high-frequency components, since the original input's low frequencies are realistic and distortion-free. Finally, the reconstructed narrowband amplitude spectrum is dot-multiplied by the half-wave rectified phase spectrum to obtain the reconstructed complex spectrum, which is then processed by IFFT and OLA / OLS to obtain the reconstructed speech waveform. At this point, the low frequencies of the speech remain consistent with the original input, while the mid-to-high frequencies represent the reconstructed vowels and consonants.

[0041] Loss function: The time-domain waveform, frequency-domain amplitude spectrum / complex spectrum / masking value, etc., of the unfiltered speech are selected as labels for the model output. Then, L1, smooth L1, and L2 norms are calculated for the labels and the time-domain waveform, frequency-domain amplitude spectrum / complex spectrum / masking value of the model output. Multi-scale frequency-domain resolution (SNR / SISNR / amplitude spectrum / complex spectrum), speech signal distortion ratio (SDR), and objective speech quality assessment (PESQ) metrics can also be combined, and each metric is weighted and summed (weighting is used because different metrics may have orders of magnitude differences). Finally, the model is trained on a large amount of data (usually thousands of hours or more) using deep learning tools and optimizers, and converges after multiple iterations. The converged model has accurate reconstruction capabilities for high-frequency components of speech at various low sampling rates and in various scenarios.

[0042] This scheme uses time-domain waveform L1 loss and relative average least squares GAN loss, multi-scale frequency domain resolution SNR, compressed amplitude spectrum and logarithmic amplitude spectrum L1 loss, and corresponding weight scaling coefficients as loss functions.

[0043] Step S13: Determine the target fusion ratio of the preset white noise based on the consonant probability, fuse the preset white noise with the reconstructed amplitude spectrum based on the target fusion ratio to obtain the corresponding target amplitude spectrum, and reconstruct the target narrowband speech signal based on the phase information corresponding to the rectified signal and the target amplitude spectrum.

[0044] In this embodiment, signal reconstruction of the target narrowband speech signal is performed based on the phase information corresponding to the rectified signal and the target amplitude spectrum. This includes: obtaining the phase information corresponding to the rectified signal, and performing a dot product between the phase information and the target amplitude spectrum to obtain the target frequency domain signal; performing an inverse fast Fourier transform on the target frequency domain signal to obtain the corresponding target time domain signal, and processing the target time domain signal using an overlap addition method or an overlap preservation method to complete the signal reconstruction operation of the target narrowband speech signal.

[0045] That is, the reconstructed narrowband amplitude spectrum is multiplied by the phase spectrum (i.e., phase information) after half-wave rectification to obtain the reconstructed complex spectrum (i.e., the target frequency domain signal), and then the reconstructed speech waveform is obtained by IFFT and OLA / OLS.

[0046] As can be seen, this application reconstructs the amplitude spectrum and determines the consonant probability simultaneously by using a neural network output. It replaces the traditional zero-crossing rate calculation with data-driven probability estimation, thus avoiding the problem of zero-crossing rate calculation deviation in noisy or multi-person speech overlap scenarios. By determining the white noise fusion ratio based on the consonant probability and fusing it, the amount of white noise injected corresponding to the consonant is precisely controlled by the neural network, which solves the problem of unstable consonant reconstruction in complex scenarios.

[0047] See Figure 2 As shown, an embodiment of the present invention discloses a speech signal reconstruction apparatus, comprising: The feature extraction module 11 is used to acquire a target narrowband speech signal, perform half-wave rectification on the target narrowband speech signal to obtain a corresponding rectified signal, and extract features from the rectified signal and the target narrowband speech signal respectively to obtain a first amplitude feature corresponding to the rectified signal and a second amplitude feature corresponding to the target narrowband speech signal. The model inference module 12 is used to superimpose the first amplitude feature and the second amplitude feature to obtain the corresponding superimposed amplitude feature, and to obtain the reconstructed amplitude spectrum corresponding to the target narrowband speech signal and the consonant probability corresponding to the target narrowband speech signal based on the superimposed amplitude feature and the pre-trained target neural network model; the consonant probability represents the probability that each frame in the target narrowband speech signal includes a consonant. The signal reconstruction module 13 is used to determine the target fusion ratio of the preset white noise according to the consonant probability, fuse the preset white noise with the reconstructed amplitude spectrum based on the target fusion ratio to obtain the corresponding target amplitude spectrum, and reconstruct the target narrowband speech signal based on the phase information corresponding to the rectified signal and the target amplitude spectrum.

[0048] In some specific embodiments, the feature extraction module 11 further includes: A signal pre-emphasis unit is used to pre-emphasize the rectified signal to obtain a corresponding processed signal. Accordingly, the feature extraction module 11 may specifically include: The feature extraction unit is used to extract features from the processed signal and the target narrowband speech signal respectively, so as to obtain the first amplitude feature corresponding to the processed signal and the second amplitude feature corresponding to the target narrowband speech signal.

[0049] In some specific embodiments, the feature extraction module 11 may specifically include: The signal transformation unit is used to perform frame-segmentation and windowing processing on the rectified signal and the target narrowband speech signal respectively to obtain the corresponding first signal and second signal, and to perform fast Fourier transform on the first signal and the second signal respectively to obtain the first amplitude feature corresponding to the rectified signal and the second amplitude feature corresponding to the target narrowband speech signal.

[0050] In some specific embodiments, the model inference module 12 further includes: A signal acquisition unit is used to acquire a clean human voice signal at a target sampling rate and a background sound signal at a target sampling rate; wherein, the clean human voice signal represents a human voice signal without background sound. The signal superposition unit is used to superimpose the pure human voice signal and the background sound signal based on a preset signal-to-noise ratio range, and to perform low-pass filtering on the corresponding superimposed signal to obtain target training data. The model training unit is used to determine the superimposed signal as target label data, and to train the initial neural network model using the target label data and the target training data to obtain the target neural network model.

[0051] In some specific embodiments, the model inference module 12 further includes: The model structure determination unit is used to obtain the device computing power and device storage space of the target edge device, and determine the model structure and model parameter quantity of the target neural network model based on the device computing power and device storage space; wherein, the target edge device is the device that deploys the target neural network.

[0052] In some specific embodiments, the model inference module 12 may specifically include: The spectrum input unit is used to convert the superimposed amplitude features into the corresponding target Mel spectrum, and input the target Mel spectrum into the target neural network model so that the target neural network model outputs the reconstructed amplitude spectrum corresponding to the target narrowband speech signal and the consonant probability corresponding to the target narrowband speech signal.

[0053] In some specific embodiments, the signal reconstruction module 13 may specifically include: A phase information acquisition unit is used to acquire the phase information corresponding to the rectified signal and multiply the phase information by the target amplitude spectrum to obtain the target frequency domain signal. The signal reconstruction unit is used to perform a fast inverse Fourier transform on the target frequency domain signal to obtain the corresponding target time domain signal, and to process the target time domain signal using an overlap addition method or an overlap preservation method to complete the signal reconstruction operation of the target narrowband speech signal.

[0054] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0055] Figure 3 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the speech signal reconstruction method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0056] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0057] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0058] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the speech signal reconstruction method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0059] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed speech signal reconstruction method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0060] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0061] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0062] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0063] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0064] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for reconstructing speech signals, characterized in that, include: A target narrowband speech signal is acquired, and the target narrowband speech signal is subjected to half-wave rectification to obtain a corresponding rectified signal. Feature extraction is performed on the rectified signal and the target narrowband speech signal respectively to obtain a first amplitude feature corresponding to the rectified signal and a second amplitude feature corresponding to the target narrowband speech signal. The first amplitude feature and the second amplitude feature are superimposed to obtain the corresponding superimposed amplitude feature. Based on the superimposed amplitude feature and the pre-trained target neural network model, the reconstructed amplitude spectrum corresponding to the target narrowband speech signal and the consonant probability corresponding to the target narrowband speech signal are obtained. The consonant probability represents the probability that each frame in the target narrowband speech signal includes a consonant. The target fusion ratio of the preset white noise is determined based on the consonant probability. The preset white noise is then fused with the reconstructed amplitude spectrum based on the target fusion ratio to obtain the corresponding target amplitude spectrum. Finally, the target narrowband speech signal is reconstructed based on the phase information corresponding to the rectified signal and the target amplitude spectrum.

2. The speech signal reconstruction method according to claim 1, characterized in that, After performing half-wave rectification on the target narrowband speech signal, the process further includes: The rectified signal is pre-emphasized to obtain a corresponding processed signal; Accordingly, the step of extracting features from the rectified signal and the target narrowband speech signal respectively to obtain a first amplitude feature corresponding to the rectified signal and a second amplitude feature corresponding to the target narrowband speech signal includes: Feature extraction is performed on the processed signal and the target narrowband speech signal respectively to obtain the first amplitude feature corresponding to the processed signal and the second amplitude feature corresponding to the target narrowband speech signal.

3. The speech signal reconstruction method according to claim 1, characterized in that, The step of extracting features from the rectified signal and the target narrowband speech signal respectively to obtain a first amplitude feature corresponding to the rectified signal and a second amplitude feature corresponding to the target narrowband speech signal includes: The rectified signal and the target narrowband speech are subjected to frame-by-frame windowing processing to obtain corresponding first and second signals. The first and second signals are then subjected to Fast Fourier Transform to obtain the first amplitude feature corresponding to the rectified signal and the second amplitude feature corresponding to the target narrowband speech signal.

4. The speech signal reconstruction method according to claim 1, characterized in that, Before obtaining the reconstructed amplitude spectrum and consonant probability of the target narrowband speech signal based on the superimposed amplitude features and the pre-trained target neural network model, the method further includes: Acquire a clean human voice signal at a target sampling rate and a background sound signal at the target sampling rate; wherein, the clean human voice signal represents a human voice signal without background sound. The pure human voice signal and the background sound signal are superimposed based on a preset signal-to-noise ratio range, and the superimposed signal is low-pass filtered to obtain target training data. The superimposed signal is determined as target label data, and the initial neural network model is trained using the target label data and the target training data to obtain the target neural network model.

5. The speech signal reconstruction method according to claim 1, characterized in that, Before obtaining the reconstructed amplitude spectrum and consonant probability of the target narrowband speech signal based on the superimposed amplitude features and the pre-trained target neural network model, the method further includes: The device computing power and storage space of the target edge device are obtained, and the model structure and model parameter quantity of the target neural network model are determined based on the device computing power and storage space; wherein, the target edge device is the device on which the target neural network is deployed.

6. The speech signal reconstruction method according to claim 1, characterized in that, The step of obtaining the reconstructed amplitude spectrum and consonant probabilities of the target narrowband speech signal based on the superimposed amplitude features and the pre-trained target neural network model includes: The superimposed amplitude features are converted into the corresponding target Mel spectrum, and the target Mel spectrum is input into the target neural network model so that the target neural network model outputs the reconstructed amplitude spectrum corresponding to the target narrowband speech signal and the consonant probability corresponding to the target narrowband speech signal.

7. The speech signal reconstruction method according to claim 1, characterized in that, The process of reconstructing the target narrowband speech signal based on the phase information corresponding to the rectified signal and the target amplitude spectrum includes: Obtain the phase information corresponding to the rectified signal, and multiply the phase information with the target amplitude spectrum to obtain the target frequency domain signal; The target frequency domain signal is subjected to inverse fast Fourier transform to obtain the corresponding target time domain signal, and the target time domain signal is processed by the overlap addition method or the overlap preservation method to complete the signal reconstruction operation of the target narrowband speech signal.

8. A speech signal reconstruction device, characterized in that, include: The feature extraction module is used to acquire a target narrowband speech signal, perform half-wave rectification on the target narrowband speech signal to obtain a corresponding rectified signal, and extract features from the rectified signal and the target narrowband speech signal respectively to obtain a first amplitude feature corresponding to the rectified signal and a second amplitude feature corresponding to the target narrowband speech signal. The model inference module is used to superimpose the first amplitude feature and the second amplitude feature to obtain the corresponding superimposed amplitude feature, and to obtain the reconstructed amplitude spectrum corresponding to the target narrowband speech signal and the consonant probability corresponding to the target narrowband speech signal based on the superimposed amplitude feature and the pre-trained target neural network model; the consonant probability represents the probability that each frame in the target narrowband speech signal includes a consonant. The signal reconstruction module is used to determine the target fusion ratio of the preset white noise based on the consonant probability, fuse the preset white noise with the reconstructed amplitude spectrum based on the target fusion ratio to obtain the corresponding target amplitude spectrum, and reconstruct the target narrowband speech signal based on the phase information corresponding to the rectified signal and the target amplitude spectrum.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the speech signal reconstruction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the speech signal reconstruction method as described in any one of claims 1 to 7.

Images