Voice processing method and device, storage medium and computer equipment

Abstract

The embodiment of the invention discloses a voice processing method and device, a storage medium and computer equipment. The method comprises the following steps: acquiring an initial voice signal, obtaining a first frequency domain signal of the initial voice signal through short-time Fourier transform, performing Mel spectrum transform on the first frequency domain signal to obtain a Mel spectrum, and inputting the Mel spectrum into a preset neural network model to output the Mel spectrum of a voice mask signal, the voice mask is used for representing weight values of different frequencies, performing Mel spectrum inverse transformation on the Mel spectrum of the voice mask signal to obtain a second frequency domain signal, calculating an output signal according to the first frequency domain signal and the second frequency domain signal, and performing short-time Fourier inverse transformation processing on the output signal. According to the method, the voice mask is generated through the neural network model capable of processing howling suppression and voice noise reduction at the same time to accurately distinguish voice components and noise components in the signals, so that noise is recognized and suppressed more accurately, and voice information is reserved at the same time.

Description

Technical Field

[0001] The present application relates to the field of audio signal technology, and in particular to a speech processing method, apparatus, storage medium and computer equipment. Background Art

[0002] In various acoustic scenarios where sound amplification is required, a common problem is that after the sound signals collected by the microphone are amplified by the speaker, these sounds may be picked up again by the microphone, thus forming a closed acoustic feedback loop. In the acoustic feedback loop, the sound signals are constantly superimposed and amplified, and the resulting positive feedback will eventually lead to an oscillation effect, resulting in a howling phenomenon. Howling not only limits the maximum gain level of hearing aids such as hearing aids and broadcasting systems, but also seriously affects the quality of the sound, greatly reducing the user's auditory experience. In addition, various noises in the real environment, such as background noise, environmental reverberation, etc., will also be captured and amplified during the process of sound collection and amplification.

[0003] In view of the above situation, the prior art generally processes the problem by combining a howling suppression algorithm with a speech noise reduction algorithm. Specifically, the howling is detected in real time in the sound signal. After the howling is detected, the suppression mechanism is immediately activated, such as adjusting the speaker gain, changing the filter parameters or applying phase adjustment technology to quickly weaken or eliminate the howling. Then, the pure speech component is extracted from the noisy signal by spectral subtraction, Wiener filtering, minimum mean square error estimation, etc. The applicant found that running the two algorithms consumes a lot of computing resources and memory space. In addition, the howling suppression algorithm and the speech noise reduction algorithm may interfere with each other when processing the sound signal, resulting in poor results. Summary of the invention

[0004] The embodiments of the present application provide a speech processing method, apparatus, storage medium and computer device, which generate a speech mask by a neural network model that can simultaneously process howling suppression tasks and speech noise reduction tasks, and can accurately distinguish between speech components and noise components in a signal, thereby more accurately identifying and suppressing noise while retaining speech information.

[0005] The present application provides a speech processing method, including:

[0006] Acquire an initial speech signal, obtain a first frequency domain signal of the initial speech signal through short-time Fourier transform, and perform Mel spectrum transform on the first frequency domain signal to obtain a Mel spectrum;

[0007] Inputting the Mel spectrum into a preset neural network model to output a Mel spectrum of a speech mask signal, wherein the speech mask is used to characterize weight values ​​of different frequencies;

[0008] Performing an inverse Mel spectrum transform on the Mel spectrum of the speech mask signal to obtain a second frequency domain signal;

[0009] An output signal is calculated according to the first frequency domain signal and the second frequency domain signal, and short-time inverse Fourier transform processing is performed on the output signal.

[0010] In one embodiment, the Mel spectrum is input into a preset neural network model to output the Mel spectrum of the speech mask signal, including:

[0011] Extract local features of different frequencies and times from the Mel spectrum through the convolution layer;

[0012] Extracting temporal features on the Mel-spectrogram through a gated recurrent unit layer;

[0013] The local features and the temporal features are integrated through a fully connected layer, and a weight vector for each frequency is generated, and the Mel spectrum of the speech mask signal is output based on the weight vector.

[0014] In one embodiment, calculating an output signal according to the first frequency domain signal and the second frequency domain signal includes:

[0015] The frequency spectrum of the first frequency domain signal is multiplied element by element with the frequency spectrum of the second frequency domain signal to obtain the frequency spectrum of the output signal.

[0016] In one embodiment, the method further comprises:

[0017] Collecting sample sets of speech signals in various scenarios, wherein the sample sets include positive sample data and negative sample data;

[0018] Performing data augmentation processing on the positive sample data and the negative sample data respectively;

[0019] The preset neural network model is trained according to the augmented sample set and the corresponding labels to correct the parameters in the preset neural network model.

[0020] In one embodiment, data augmentation processing is performed on the positive sample data and the negative sample data respectively, including:

[0021] Setting a parameter range of the acoustic model, wherein the parameter range includes a coordinate range of a microphone position, an intensity range of ambient noise, and a range of a howling frequency;

[0022] Extracting a source signal from the positive sample data, and inputting the source speech signal into the acoustic model to generate a contaminated speech sample as negative sample data;

[0023] The negative sample data is subjected to superposition or intensity adjustment processing.

[0024] In one embodiment, the method further comprises:

[0025] Using a mean square error function as a loss function, the loss function is used to measure the difference between the output of the preset neural network model and the true label;

[0026] The gradient of the loss function is calculated by a back-propagation algorithm and a gradient descent method, and the parameters in the preset neural network model are updated according to the gradient.

[0027] In one embodiment, after performing short-time inverse Fourier transform processing on the output signal, the method further includes:

[0028] Amplifying the target speech signal after the inverse short-time Fourier transform process;

[0029] The amplified signal is input to a loudspeaker for playback.

[0030] The present application also provides a speech processing device, including:

[0031] A first transform unit is used to obtain an initial speech signal, obtain a first frequency domain signal of the initial speech signal by short-time Fourier transform, and perform Mel spectrum transform on the first frequency domain signal to obtain a Mel spectrum;

[0032] A processing unit, configured to input the Mel spectrum into a preset neural network model to output a Mel spectrum of a speech mask signal, wherein the speech mask is used to characterize weight values ​​of different frequencies;

[0033] A second transform unit, configured to perform an inverse Mel spectrum transform on the Mel spectrum of the speech mask signal to obtain a second frequency domain signal;

[0034] A calculation unit is used to calculate an output signal according to the first frequency domain signal and the second frequency domain signal, and perform short-time inverse Fourier transform processing on the output signal.

[0035] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed on a computer, the computer executes the speech processing method as described above.

[0036] An embodiment of the present application further provides a computer device, comprising a memory and a processor, wherein the processor executes the above-mentioned speech processing method by calling a computer program stored in the memory.

[0037] The speech processing method, apparatus, storage medium and computer equipment provided in the embodiments of the present application can obtain an initial speech signal, obtain a first frequency domain signal of the initial speech signal through a short-time Fourier transform, perform a Mel spectrum transform on the first frequency domain signal to obtain a Mel spectrum, input the Mel spectrum into a preset neural network model to output a Mel spectrum of a speech mask signal, wherein the speech mask is used to characterize the weight values ​​of different frequencies, perform an inverse Mel spectrum transform on the Mel spectrum of the speech mask signal to obtain a second frequency domain signal, calculate an output signal based on the first frequency domain signal and the second frequency domain signal, and perform an inverse short-time Fourier transform on the output signal. The solution provided in the embodiments of the present application generates a speech mask through a neural network model that can simultaneously process howling suppression tasks and speech noise reduction tasks, which can accurately distinguish between speech components and noise components in the signal, thereby more accurately identifying and suppressing noise while retaining speech information. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 A flowchart of a speech processing method provided in an embodiment of the present application.

[0040] Figure 2 A schematic diagram of a flow chart of a preset neural network model training provided in an embodiment of the present application.

[0041] Figure 3 A second flow chart of the speech processing method provided in the embodiment of the present application.

[0042] Figure 4 A third flow chart of the speech processing method provided in an embodiment of the present application.

[0043] Figure 5 A schematic diagram of the structure of a speech processing device provided in an embodiment of the present application.

[0044] Figure 6 A schematic diagram of the computer device structure provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0046] It should be noted that, in this article, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the existence of other identical elements in the process, method, article or device that includes the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0047] The embodiments of the present application provide a speech processing method, apparatus, storage medium and computer equipment. Specifically, the speech processing method of the embodiments of the present application can be executed by a computer device or a server, wherein the computer device can be a terminal. The terminal can be a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, a personal computer (PC, Personal Computer), a personal digital assistant (Personal Digital Assistant, PDA), a smart home and other devices, and the terminal can also include a client, which can be a media playback client or an instant messaging client, etc.

[0048] See also Figure 1 , the specific process of the speech processing method can be as follows:

[0049] Step 101: Acquire an initial speech signal, obtain a first frequency domain signal of the initial speech signal by short-time Fourier transform, and perform Mel spectrum transform on the first frequency domain signal to obtain a Mel spectrum.

[0050] In one embodiment, the initial voice signal may be a sound signal captured from the environment by a microphone or other audio recording device. These signals are usually in analog form and can be converted into digital signals by an analog-to-digital converter (ADC) for subsequent processing in a computer. Digital voice signals are usually represented as a series of discrete sample values ​​that vary over time and carry the waveform information of the voice.

[0051] Furthermore, the Short Time Fourier Transform (STFT) is a tool for analyzing non-stationary signals (such as speech). It divides a long signal into shorter time segments and performs a Fourier transform on each time segment to provide time and frequency information. Specifically, the STFT first divides the continuous signal into shorter time segments. For example, it can be achieved by multiplying a sliding window function, which is non-zero in a specific time interval and slides over time. The window function can include a rectangular window, a Hanning window, a Hamming window, etc. The width of the window function (or the window length) determines the balance between time resolution and frequency resolution: the wider the window, the higher the frequency resolution and the lower the time resolution; the narrower the window, the higher the time resolution and the lower the frequency resolution. Then, the Fourier transform is applied to each time segment to calculate the frequency component of the signal in the time segment, so that each time segment corresponds to a spectrum. Finally, the Fourier transform results of all time segments can be combined to obtain the time-frequency representation of the signal, that is, the result of the STFT. This result is usually represented as a two-dimensional array, in which one dimension represents time and the other dimension represents frequency.

[0052] Mel spectrum transform is a method for mapping a linear spectrum to a Mel nonlinear spectrum based on auditory perception. It converts the actual linear frequency scale into a Mel scale, so that the low frequency part has a higher frequency resolution, while the high frequency part has a relatively low resolution, which matches the perceptual sensitivity of the human ear to sounds of different frequencies. Specifically, mapping the linear spectrum to the Mel scale can be achieved by applying a Mel filter bank, which is composed of a series of triangular filters evenly distributed on the Mel scale. Each filter weights the spectrum to extract the frequency components on the corresponding Mel scale. In other embodiments, since the human ear is also nonlinear in its perception of volume, the output of the Mel filter bank can also be logarithmically transformed to obtain a Mel spectrum (Mel-spectrogram, MEL) that is more in line with the perceptual characteristics of the human ear.

[0053] In the neural network processing of the embodiment of the present application, the input signal is converted into a Mel spectrum because it can better highlight the speech features that are important to human ear perception, thereby helping the network model to more effectively learn and distinguish components such as speech, howling and noise.

[0054] Step 102: input the Mel spectrum into a preset neural network model to output the Mel spectrum of the speech mask signal, wherein the speech mask is used to characterize the weight values ​​of different frequencies.

[0055] In one embodiment, the preset neural network model includes a convolution layer and a GRU (gated recurrent unit) layer, wherein the convolution layer is mainly used for feature extraction of the Mel spectrum. The convolution layer performs convolution operations by sliding the convolution kernel (filter) on the spectrum data, and can automatically capture the local characteristic patterns in the spectrum, such as the change law of different frequency components over time, the energy distribution of a specific frequency band, etc. The GRU layer is used to process the timing information in the sequence data. Since the speech signal is a sequence that changes over time, there is an important correlation between the previous and next frames. The GRU layer can selectively remember and forget previous information through the internal gating mechanism (update gate and reset gate), thereby modeling the long-term dependency in the sequence. In this scenario, the GRU layer can learn the dynamic change pattern of the speech signal in the time dimension, such as the phoneme transition, rhythmic features, and the persistence characteristics of howling and noise in time.

[0056] Furthermore, after being processed by the convolutional layer and the GRU layer, the preset neural network model finally outputs a Mel spectrum of a speech mask signal. This speech mask is actually a weight matrix, which is the same size as the input Mel spectrum, and each element corresponds to the weight value of the corresponding frequency component in the input spectrum. In this weight matrix, elements with values ​​close to 1 indicate that the corresponding frequency components should be retained, which mainly correspond to the speech signal part, while elements with values ​​close to 0 indicate that the corresponding frequency components should be suppressed or weakened, which are usually related to howling and noise.

[0057] During the learning process, the preset neural network model gradually learns to identify which frequency features are unique to the speech signal through a large amount of training data, and then gives these frequencies a higher weight in the mask to ensure that the speech information is retained in subsequent processing. Based on the training data, the network also identifies the characteristic patterns of howling and noise on the spectrum, and gives these frequencies a lower weight in the mask, thereby reducing or eliminating their impact on the speech signal in subsequent processing. In this way, the speech mask can effectively separate the speech part of the input signal from the howling and noise part, providing a key decision-making basis for achieving howling suppression and speech noise reduction. Among them, the training process of the above preset neural network model can be found in Figure 2 , Figure 2 A schematic diagram of a process flow of a preset neural network model training provided in an embodiment of the present application may include the following steps:

[0058] Step 1021, collecting sample sets of speech signals in various scenarios, where the sample sets include positive sample data and negative sample data;

[0059] Step 1022, performing data augmentation processing on the positive sample data and the negative sample data respectively;

[0060] Step 1023, training the preset neural network model according to the augmented sample set and the corresponding labels to correct the parameters in the preset neural network model.

[0061] The step of performing data augmentation processing on the positive sample data and the negative sample data respectively may include: setting the parameter range of the acoustic model, the parameter range includes the coordinate range of the microphone position, the intensity range of the ambient noise, and the range of the howling frequency, extracting the source signal from the positive sample data, and inputting the source voice signal into the acoustic model to generate a contaminated voice sample as the negative sample data, and superimposing or intensity adjusting the negative sample data. Specifically, the various parameter ranges to be simulated in the acoustic model may be determined first, including the coordinate range of different microphone positions (such as x, y, z coordinates in three-dimensional space), the intensity range of the ambient noise level (for example, from 30dB to 100dB), and the range of the howling frequency (set according to the actual possible howling frequency range, such as 500Hz-5000Hz), etc. At the same time, the environmental parameters required for the acoustic model are initialized, such as room size, wall reflection coefficient, etc. These parameters determine the propagation characteristics of the sound in the simulated environment. Then, the source signal is extracted from the positive sample data, and the source speech signal is input into the acoustic model to calculate the process of sound propagation from the sound source to the microphone according to the set microphone position, and obtain the speech signal received by the microphone at this position. And according to the set environmental noise level, a noise signal of corresponding intensity is generated, and it is superimposed with the propagated speech signal to simulate the interference of environmental noise on the speech signal. At the same time, according to the method of howling frequency simulation, a simulated howling signal is introduced at an appropriate time, and it is superimposed again with the speech signal to which noise has been added to generate a polluted speech signal containing howling and noise, so that the polluted speech signal is used as negative sample data. This embodiment enables the neural network to learn a wider combination of speech, noise and howling features through a rich sample data set, thereby improving the adaptability and generalization ability of the model in different practical scenarios.

[0062] In one embodiment, the preset neural network model can also use the mean square error function as a loss function, which is used to measure the difference between the output of the preset neural network model and the true label, and then calculate the gradient of the loss function through the back propagation algorithm and the gradient descent method, and update the parameters in the preset neural network model according to the gradient. Among them, the back propagation algorithm and the gradient descent optimization method are used in a specific process. First, the gradient is calculated according to the loss function, and then the calculated gradient is used to update the network parameters. By repeating this process continuously, the model gradually reaches a better state. For example, in the training of a neural network, the back propagation algorithm can efficiently calculate the gradients of the parameters of each layer, and the gradient descent optimization method adjusts the parameter values ​​according to these gradients, so that the loss of the model on the training data gradually decreases, thereby improving the performance and accuracy of the model.

[0063] Step 103: Perform an inverse Mel spectrum transform on the Mel spectrum of the speech mask signal to obtain a second frequency domain signal.

[0064] In one embodiment, after the preset neural network model outputs the Mel spectrum of the speech mask signal, it needs to be converted back to a conventional frequency domain representation through a Mel inverse transform. This process is the inverse operation of the Mel spectrum generation process, which remaps the spectrum information under the Mel scale back to the linear frequency scale, restoring it to the same frequency representation as the original input signal spectrum, so as to accurately process it with the input signal spectrum later.

[0065] Step 104: Calculate an output signal according to the first frequency domain signal and the second frequency domain signal, and perform inverse short-time Fourier transform processing on the output signal.

[0066] In one embodiment, the speech mask spectrum converted back to the frequency domain can be element-by-element multiplied with the spectrum of the input signal, and its core purpose is to adjust the corresponding frequency components in the input signal spectrum according to the weight value of each frequency component in the speech mask. The speech mask is a weight matrix, in which a weight value close to 1 indicates that the corresponding frequency component should be retained (considered to be the speech part), and a weight value close to 0 indicates that it should be suppressed or weakened (considered to be the howling or noise part). By element-by-element multiplication, the frequency components related to speech in the input signal spectrum are retained or enhanced, while the frequency components related to howling and noise are weakened or eliminated.

[0067] Furthermore, after obtaining the output signal spectrum through element-by-element multiplication, the last step is to convert the spectrum back to the time domain signal through the inverse Fourier transform (ISTFT) to obtain the final output signal. The inverse Fourier transform is the inverse process of the Fourier transform. It converts the signal representation in the frequency domain back to the time domain signal and restores the waveform representation of the sound signal in time. This process combines the results of all the previous processing steps, so that the final output signal achieves the effect of howling suppression and speech noise reduction in the time domain. The output speech signal is clearer and purer, reducing the interference of howling and noise, improving the speech quality, and meeting the needs of high-quality speech processing in practical applications.

[0068] As can be seen from the above, the speech processing method provided in the embodiment of the present application can obtain an initial speech signal, obtain a first frequency domain signal of the initial speech signal through a short-time Fourier transform, and perform a Mel spectrum transform on the first frequency domain signal to obtain a Mel spectrum, and input the Mel spectrum into a preset neural network model to output a Mel spectrum of a speech mask signal, wherein the speech mask is used to characterize the weight values ​​of different frequencies, and perform a Mel spectrum inverse transform on the Mel spectrum of the speech mask signal to obtain a second frequency domain signal, calculate an output signal based on the first frequency domain signal and the second frequency domain signal, and perform a short-time Fourier inverse transform on the output signal. The solution provided in the embodiment of the present application generates a speech mask through a neural network model that can simultaneously handle howling suppression tasks and speech noise reduction tasks, which can accurately distinguish between speech components and noise components in the signal, thereby more accurately identifying and suppressing noise while retaining speech information.

[0069] See also Figure 3 , is another flow chart of the speech processing method provided in the embodiment of the present application. The specific flow of the method may be as follows:

[0070] Step 201: Acquire an initial speech signal, obtain a first frequency domain signal of the initial speech signal by short-time Fourier transform, and perform Mel spectrum transform on the first frequency domain signal to obtain a Mel spectrum.

[0071] In one embodiment, the initial speech signal may be a sound signal captured from the environment by a microphone or other audio recording device. After the initial speech signal is obtained, it may be subjected to a short-time Fourier transform and a Mel spectrum transform. Among them, the Mel spectrum can better reflect the difference in sensitivity of the human ear to sounds of different frequencies, and compared with ordinary spectra, it can often provide more meaningful feature representation in speech processing tasks.

[0072] Step 202: extract local features of different frequencies and times from the Mel spectrum through a convolutional layer.

[0073] Specifically, the convolution layer performs convolution operations by sliding a series of convolution kernels (filters) on the input Mel spectrum. Each convolution kernel is like a detector of a specific pattern, and its parameters are constantly adjusted during the training process to learn to capture different feature patterns. For example, one convolution kernel may focus on detecting the change of energy over time in a specific frequency range in a speech signal, while another convolution kernel may be sensitive to the periodic changes of speech signals on different time scales.

[0074] The essence of the convolution operation is to perform a local weighted summation on the input data, thereby highlighting the local features in the data. In the Mel spectrum, this means that the convolution layer can detect local feature changes at different frequencies and time positions. For example, for speech signals, there may be a relatively stable energy distribution pattern in certain frequency bands, and the convolution kernel can extract these features through the convolution operation; while for howling signals, there are often sudden energy enhancements at specific frequencies, and the convolution layer can also keenly capture this local change.

[0075] Step 203: extracting temporal features from the Mel-spectrogram through a gated recurrent unit layer.

[0076] Among them, the GRU layer is mainly used to process the timing information in the sequence data. The speech signal is essentially a time series, and there is a close connection between the previous and next frames. The GRU layer controls the flow and update of information through its internal gating mechanism (update gate and reset gate). The update gate determines how much of the hidden state information of the previous moment is retained at the current moment, and the reset gate controls how to combine the new input information with the hidden state of the previous moment.

[0077] When processing speech signals, the GRU layer can learn the dynamic change patterns of speech in the time dimension, such as the transition between phonemes, the rhythmic features of speech, and the persistence of howling and noise in time. This ability to model long-term dependencies enables the network to better understand the contextual information of the entire speech sequence, rather than just being limited to local features. For example, a continuous howling signal may show similar frequency characteristics and energy change trends in multiple frames of data. The GRU layer can more accurately determine that the signal is a howling signal by learning this timing information, and process it accordingly in the subsequently generated speech mask. At the same time, for situations such as connected reading and intonation changes in speech signals, the GRU layer can also use the contextual information it has learned to reasonably process them to ensure the integrity and coherence of the speech information.

[0078] Step 204: integrate the local features and the temporal features through a fully connected layer, generate a weight vector for each frequency, and output the Mel spectrum of the speech mask signal based on the weight vector.

[0079] After being processed by the convolutional layer and the GRU layer, the neural network has extracted rich feature information from the input Mel spectrum, including local features and temporal features. At this point, the network integrates these feature information through a fully connected layer (or a similar decision layer) to generate a weight vector representing the importance of each frequency component. This weight vector constitutes the Mel spectrum of the speech mask signal.

[0080] Furthermore, in the Mel spectrum of this speech mask, the weight value corresponding to each frequency component reflects the possibility of whether the frequency component belongs to the speech part or the howling / noise part in the current speech signal. A weight value close to 1 indicates that the frequency component is more likely to be a speech signal and should be retained; a weight value close to 0 indicates that the frequency component is more likely to be a howling or noise and needs to be suppressed or weakened. For example, if in a certain frequency band, the features extracted by the convolution layer are highly matched with the typical features of the speech signal, and the GRU layer also determines that the role of the frequency band in the entire speech sequence is consistent with the speech characteristics based on the context information, then in the generated speech mask, the weight value corresponding to the frequency band will be higher, so that when it is subsequently multiplied with the input spectrum, the information of the frequency band can be retained; conversely, if the characteristics of a certain frequency band are similar to the howling or noise characteristics, its weight value will be lower, so that it will be weakened during the multiplication process.

[0081] Step 205: Perform an inverse Mel spectrum transform on the Mel spectrum of the speech mask signal to obtain a second frequency domain signal.

[0082] Step 206 : multiply the spectrum of the first frequency domain signal and the spectrum of the second frequency domain signal element by element to obtain the spectrum of the output signal.

[0083] By element-by-element multiplication, the frequency components related to speech in the input signal spectrum are retained or enhanced, while the frequency components related to howling and noise are weakened or eliminated. For example, assuming that the input signal spectrum is S(f), where f represents frequency, and the speech mask spectrum is M(f), the output signal spectrum O(f) is calculated by the following element-by-element multiplication: O(f) = S(f)*M(f), for each frequency f, the value of the output signal spectrum at that frequency is equal to the product of the input signal spectrum value and the speech mask spectrum value at that frequency. For example, if at a certain frequency f1, the input signal spectrum value S(f1) = 0.8, the speech mask spectrum value M(f1) = 0.9 (indicating that the frequency component is likely to be the speech part), then the value of the output signal spectrum at this frequency is O(f1) = 0.8*0.9 = 0.72, that is, the frequency component is retained to a certain extent in the output signal spectrum; conversely, if at another frequency f2, S(f2) = 0.6, M(f2) = 0.1 (indicating that the frequency component may be the noise or howling part), then O(f2) = 0.6*0.1 = 0.06, and the frequency component is greatly weakened in the output signal spectrum.

[0084] Step 207, performing short-time inverse Fourier transform processing on the output signal, and amplifying the target speech signal after the short-time inverse Fourier transform processing.

[0085] Step 208: input the amplified signal into the speaker for playback.

[0086] After obtaining the output signal spectrum through element-by-element multiplication, the last step is to convert the spectrum back to the time domain signal through inverse Fourier transform to obtain the final output signal, and then amplify the output signal through the amplifier and input it to the speaker for playback, such as Figure 4 shown.

[0087] This embodiment can significantly reduce or even eliminate the interference of howling and background noise on the voice signal by combining the howling suppression algorithm with the voice noise reduction algorithm, making the output voice clearer and purer. This is crucial to improving the accuracy and fluency of voice communication, especially in noisy environments or long-distance communications. In addition, compared with the traditional method of using two independent algorithms to deal with howling and noise problems, this solution achieves efficient resource utilization by integrating them into a neural network model. This not only reduces the consumption of computing resources, but also reduces the demand for storage space, making the entire system more lightweight and efficient, and achieving low-latency processing while maintaining high performance, significantly improving user experience.

[0088] All of the above technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described in detail here.

[0089] As can be seen from the above, the speech processing method provided in the embodiment of the present application can obtain an initial speech signal, obtain a first frequency domain signal of the initial speech signal through a short-time Fourier transform, and perform a Mel spectrum transform on the first frequency domain signal to obtain a Mel spectrum, extract local features at different frequencies and times on the Mel spectrum through a convolutional layer, extract temporal features on the Mel spectrum through a gated recurrent unit layer, integrate local features and temporal features through a fully connected layer, and generate a weight vector for each frequency, output the Mel spectrum of the speech mask signal based on the weight vector, perform a Mel spectrum inverse transform on the Mel spectrum of the speech mask signal to obtain a second frequency domain signal, multiply the spectrum of the first frequency domain signal by the spectrum of the second frequency domain signal element by element to obtain the spectrum of the output signal, perform a short-time Fourier inverse transform on the output signal, amplify the target speech signal after the short-time Fourier inverse transform, and input the amplified signal into a speaker for playback. The solution provided in the embodiment of the present application generates a speech mask through a neural network model that can simultaneously process the howling suppression task and the speech noise reduction task, which can accurately distinguish the speech component and the noise component in the signal, thereby more accurately identifying and suppressing the noise while retaining the speech information.

[0090] In order to implement the above method, an embodiment of the present invention further provides a speech processing device, which can be integrated into a computer device such as a mobile phone, a personal computer, a tablet computer, etc.

[0091] For example, Figure 5 FIG. 1 is a schematic diagram of a first structure of a speech processing device provided by an embodiment of the present invention. The speech processing device may include:

[0092] The first transform unit 301 is used to obtain an initial speech signal, obtain a first frequency domain signal of the initial speech signal by short-time Fourier transform, and perform Mel spectrum transform on the first frequency domain signal to obtain a Mel spectrum;

[0093] The processing unit 302 is used to input the Mel spectrum into a preset neural network model to output a Mel spectrum of a speech mask signal, wherein the speech mask is used to represent weight values ​​of different frequencies;

[0094] A second transform unit 303 is used to perform an inverse Mel spectrum transform on the Mel spectrum of the speech mask signal to obtain a second frequency domain signal;

[0095] The calculation unit 304 is used to calculate an output signal according to the first frequency domain signal and the second frequency domain signal, and perform short-time inverse Fourier transform processing on the output signal.

[0096] The speech processing device proposed in the embodiment of the present invention can obtain an initial speech signal, obtain a first frequency domain signal of the initial speech signal through a short-time Fourier transform, perform a Mel spectrum transform on the first frequency domain signal to obtain a Mel spectrum, input the Mel spectrum into a preset neural network model to output a Mel spectrum of a speech mask signal, wherein the speech mask is used to characterize the weight values ​​of different frequencies, perform an inverse Mel spectrum transform on the Mel spectrum of the speech mask signal to obtain a second frequency domain signal, calculate an output signal based on the first frequency domain signal and the second frequency domain signal, and perform an inverse short-time Fourier transform on the output signal. The present application generates a speech mask through a neural network model that can simultaneously process howling suppression and speech denoising to accurately distinguish between speech components and noise components in the signal, thereby more accurately identifying and suppressing noise while retaining speech information.

[0097] All of the above technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described in detail here.

[0098] An embodiment of the present application also provides a computer device, including a memory and a processor, wherein the processor is used to execute the process in the speech processing method provided in this embodiment by calling a computer program stored in the memory.

[0099] For example, the above-mentioned computer device can be a terminal device with corresponding functions such as a mobile phone, tablet computer, personal computer, cloud computer, etc. Figure 6 .

[0100] The computer device 400 may include components such as a memory 401 and a processor 402. Those skilled in the art will appreciate that Figure 6 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0101] In this embodiment, the processor 402 in the computer device loads the executable code corresponding to the process of one or more applications into the memory 401, and the processor 402 runs the application stored in the memory 401, thereby executing the steps in the above-mentioned speech processing method in this embodiment.

[0102] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0103] To this end, an embodiment of the present application provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any data calculation method provided in the embodiment of the present application. For example, the instructions can execute the steps in the above-mentioned speech processing method in this embodiment.

[0104] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0105] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0106] Since the instructions stored in the storage medium can execute the steps in any data calculation method provided in the embodiments of the present application, the beneficial effects that can be achieved by any data calculation method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0107] It should be noted that, for the speech processing method described in the embodiment of the present application, a person of ordinary skill in the art can understand that all or part of the process of implementing the speech processing method described in the embodiment of the present application can be completed by controlling the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, such as a memory, and executed by at least one processor, and the execution process may include the process of the embodiment of the speech processing method. Among them, the storage medium can be a disk, an optical disk, a read-only memory, a random access memory (Random Access Memory, RAM), etc.

[0108] The above is a detailed introduction to a speech processing method, device, storage medium and computer equipment provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A speech processing method, characterized in that: include: Acquire an initial speech signal, obtain a first frequency domain signal of the initial speech signal through short-time Fourier transform, and perform Mel spectrum transform on the first frequency domain signal to obtain a Mel spectrum; Inputting the Mel spectrum into a preset neural network model to output a Mel spectrum of a speech mask signal, wherein the speech mask is used to characterize weight values ​​of different frequencies; Performing an inverse Mel spectrum transform on the Mel spectrum of the speech mask signal to obtain a second frequency domain signal; An output signal is calculated according to the first frequency domain signal and the second frequency domain signal, and a short-time inverse Fourier transform process is performed on the output signal.

2. The speech processing method according to claim 1, characterized in that: Inputting the Mel spectrum into a preset neural network model to output the Mel spectrum of the speech mask signal includes: Extract local features of different frequencies and times from the Mel spectrum through the convolution layer; Extracting temporal features on the Mel-spectrogram through a gated recurrent unit layer; The local features and the temporal features are integrated through a fully connected layer, and a weight vector for each frequency is generated, and the Mel spectrum of the speech mask signal is output based on the weight vector.

3. The speech processing method according to claim 1, characterized in that: Calculating an output signal according to the first frequency domain signal and the second frequency domain signal includes: The frequency spectrum of the first frequency domain signal is multiplied element by element with the frequency spectrum of the second frequency domain signal to obtain the frequency spectrum of the output signal.

4. The speech processing method according to claim 1, characterized in that: The method further comprises: Collecting sample sets of speech signals in various scenarios, wherein the sample sets include positive sample data and negative sample data; Performing data augmentation processing on the positive sample data and the negative sample data respectively; The preset neural network model is trained according to the augmented sample set and the corresponding labels to correct the parameters in the preset neural network model.

5. The speech processing method according to claim 4, characterized in that: Performing data augmentation processing on the positive sample data and the negative sample data respectively includes: Setting a parameter range of the acoustic model, wherein the parameter range includes a coordinate range of a microphone position, an intensity range of ambient noise, and a range of a howling frequency; Extracting a source signal from the positive sample data, and inputting the source speech signal into the acoustic model to generate a contaminated speech sample as negative sample data; The negative sample data is subjected to superposition or intensity adjustment processing.

6. The speech processing method according to claim 4, characterized in that: The method further comprises: Using a mean square error function as a loss function, the loss function is used to measure the difference between the output of the preset neural network model and the true label; The gradient of the loss function is calculated by a back-propagation algorithm and a gradient descent method, and the parameters in the preset neural network model are updated according to the gradient.

7. The speech processing method according to claim 1, characterized in that: After performing short-time inverse Fourier transform processing on the output signal, the method further includes: Amplifying the target speech signal after the inverse short-time Fourier transform process; The amplified signal is input to a loudspeaker for playback.

8. A speech processing device, characterized in that: include: A first transform unit is used to obtain an initial speech signal, obtain a first frequency domain signal of the initial speech signal by short-time Fourier transform, and perform Mel spectrum transform on the first frequency domain signal to obtain a Mel spectrum; A processing unit, configured to input the Mel spectrum into a preset neural network model to output a Mel spectrum of a speech mask signal, wherein the speech mask is used to characterize weight values ​​of different frequencies; A second transform unit, configured to perform an inverse Mel spectrum transform on the Mel spectrum of the speech mask signal to obtain a second frequency domain signal; A calculation unit is used to calculate an output signal according to the first frequency domain signal and the second frequency domain signal, and perform short-time inverse Fourier transform processing on the output signal.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

10. A computer device comprising a memory and a processor, characterized in that: The processor executes the method according to any one of claims 1 to 7 by calling the computer program stored in the memory.

Images