Real-time speech noise reduction method, device and electronic device based on target person

Abstract

The embodiments of the present disclosure disclose a real-time speech noise reduction method, device and electronic device based on a target person. A specific implementation of the method includes: obtaining a target speech signal and a target speaker speech signal; performing feature extraction on the target speaker speech signal to obtain the target speaker's voiceprint features; preprocessing the target speech signal to obtain the frequency spectrum features of the target speech signal; inputting the frequency spectrum features of the target speech signal and the target speaker's voiceprint features into a pre-trained noise reduction neural network to obtain the noise reduction parameter features of the target speech signal; post-processing the target speech signal according to the noise reduction parameter features to obtain a noise-reduced speech signal. This implementation achieves a more lightweight noise reduction process and a better denoising effect.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method, device, and electronic device for real-time speech noise reduction based on a target person. Background Art

[0002] As communication technology matures, the volume of voice / video calls is increasing, and users are using more diverse terminals, such as personal computers, mobile phones, and edge devices such as microphones. However, due to the variety of environments in which users live, the presence of various background noises can have a significant impact on the quality of voice calls, such as the sound of indoor air conditioner fans, keyboard tapping, and outdoor car sounds, bird calls, and so on.

[0003] In addition, when there are other people in the user's communication environment, the voices of other people will also interfere with the voice signal call. In order to hear the voice signal more clearly, it is necessary to perform noise reduction processing on the collected noisy voice signal to filter out background noise and other people's voices. Summary of the invention

[0004] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0005] Some embodiments of the present disclosure propose a real-time speech noise reduction method, apparatus, storage medium and device based on a target person to solve the technical problems mentioned in the above background technology section.

[0006] In a first aspect, some embodiments of the present disclosure provide a real-time speech denoising method based on a target person, the method comprising: acquiring a target speech signal and a target speaker speech signal; performing feature extraction on the target speaker speech signal to obtain a voiceprint feature of the target speaker; preprocessing the target speech signal to obtain a frequency spectrum feature of the target speech signal; inputting the frequency spectrum feature of the target speech signal and the voiceprint feature of the target speaker into a pre-trained denoising neural network to obtain a denoising parameter feature of the target speech signal; post-processing the target speech signal according to the parameter feature to obtain a denoised speech signal.

[0007] In a second aspect, some embodiments of the present disclosure provide a real-time speech denoising device based on a target person, the device comprising: an acquisition unit, configured to acquire a target speech signal and a target speaker speech signal; an extraction unit, configured to perform feature extraction on the target speaker speech signal to obtain the voiceprint feature of the target speaker; a preprocessing unit, configured to preprocess the target speech signal to obtain the frequency spectrum feature of the target speech signal; a generation unit, configured to input the frequency spectrum feature of the target speech signal and the voiceprint feature of the target speaker into a pre-trained denoising neural network to obtain the denoising parameter feature of the target speech signal; and a post-processing unit, configured to post-process the target speech signal according to the parameter feature to obtain a denoised speech signal.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which network device includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation method in the first aspect.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0010] One of the above embodiments of the present disclosure has the following beneficial effects: firstly, a target speech signal and a target speaker speech signal are obtained, then the target speaker speech signal is feature extracted to obtain the target speaker's voiceprint feature, then the target speech signal is preprocessed to obtain the frequency spectrum feature of the target speech signal, and then the frequency spectrum feature and the voiceprint feature are input into a pre-trained denoising neural network to obtain a denoising parameter feature, and finally the target speech signal is post-processed according to the parameter feature to obtain a denoised speech signal. Thus, by processing the target speech signal, the interference of background noise and other human voices is eliminated in the obtained denoised speech signal, achieving a lighter denoising process and a better denoising effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0012] Figure 1 is a schematic diagram of an application scenario of a real-time speech noise reduction method based on a target person according to some embodiments of the present disclosure;

[0013] Figure 2 is a flow chart of some embodiments of the target person-based real-time speech noise reduction method according to the present disclosure;

[0014] Figure 3 is a schematic diagram of a denoising neural network structure according to some embodiments of the present disclosure;

[0015] Figure 4 is a schematic diagram of a self-attention module structure according to some embodiments of the present disclosure;

[0016] Figure 5 is a schematic diagram of a convolution module structure according to some embodiments of the present disclosure;

[0017] Figure 6 It is a structural schematic diagram of some embodiments of the real-time speech noise reduction device based on the target person according to the present disclosure;

[0018] Figure 7 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0021] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0022] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0024] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0025] Figure 1 It is a schematic diagram of an application scenario of a real-time speech noise reduction method based on a target person according to some embodiments of the present disclosure.

[0026] like Figure 1 As shown, the execution subject server 101 can obtain the target speech signal 103 and the target speaker speech signal 102, and then perform feature extraction on the target speaker speech signal 102 to obtain the target speaker's voiceprint feature 104, and then pre-process the target speech signal 103 to obtain feature 105 and frequency spectrum feature 106, and then input the frequency spectrum feature 105 and the voiceprint feature 106 into a pre-trained denoising neural network to obtain denoising parameter feature 107, and finally post-process the frequency spectrum feature 106 according to the parameter feature 107 to obtain the denoised speech signal 108.

[0027] It is understandable that the real-time speech noise reduction method based on the target person can be executed by a terminal device, or it can also be executed by a server 101. The execution subject of the above method can also include a device formed by integrating the above terminal device and the above server 101 through a network, or it can also be executed by various software programs. Among them, the terminal device can be various electronic devices with information processing capabilities, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers and desktop computers, etc. The execution subject can also be embodied as a server 101, software, etc. When the execution subject is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0028] It should be understood that Figure 1 The number of servers in the example is only for illustration purposes. Any number of servers may be provided as required.

[0029] Continue to refer Figure 2 , shows a process 200 of some embodiments of the real-time speech noise reduction method based on a target person according to the present disclosure. The real-time speech noise reduction method based on a target person comprises the following steps:

[0030] Step 201: Acquire a target speech signal and a target speaker speech signal.

[0031] In some embodiments, the execution subject (eg, Figure 1The target speech signal and the target speaker speech signal can be obtained by a wired connection or a wireless connection. It should be noted that the location of the acquisition can be a local storage device. In addition, the above-mentioned wireless connection method can include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0032] Here, the target speech signal generally refers to a speech signal with noise and the target speaker's voice. The target speaker speech signal generally refers to a speech signal with the target speaker's voice. The target speaker is generally selected in advance by the user.

[0033] Step 202: extract features from the target speaker's speech signal to obtain the target speaker's voiceprint features.

[0034] In some embodiments, based on the target speaker's speech signal obtained in step 201, the execution subject (eg Figure 1 The server shown in the figure can perform feature extraction on the target speaker's voice signal to obtain the target speaker's voiceprint feature. Here, there are many ways to extract features, which will not be repeated here.

[0035] In some optional implementations of some embodiments, the execution subject may input the target speaker's voice signal into a pre-trained voiceprint feature extraction network to obtain the voiceprint features of the target speaker, wherein the frequency spectrum feature voiceprint feature extraction network includes a deep residual layer and a local aggregation vector layer. Here, the deep residual layer (ResNet) is used to extract the features of the segmented voice signal, and the local aggregation vector layer (NetVLAD) is used to aggregate the segmented features.

[0036] The voiceprint feature extraction network uses the ResNet network to extract segmented speech features, and uses NetVLAD with automatic training parameters to aggregate segmented features. The voiceprint features extracted in the end have strong anti-interference characteristics. Even if the target person is in a slightly noisy environment, the voiceprint information can still be accurately extracted.

[0037] Step 203: preprocess the target speech signal to obtain frequency spectrum features of the target speech signal.

[0038] In some embodiments, the execution subject may preprocess the target speech signal to obtain the frequency spectrum characteristics of the target speech signal. Here, the preprocessing operation generally refers to an operation including extracting features of the target speech signal and determining the frequency spectrum characteristics of the target speech signal. There are many ways to extract the frequency spectrum characteristics and determine the frequency spectrum characteristics of the speech signal, which will not be repeated here.

[0039] In some optional implementations of some embodiments, the execution subject performs framing, windowing and short-time Fourier transform on the target speech signal to obtain the real part and imaginary part of the frequency spectrum of the target speech signal. Here, the windowing process usually uses a Hanning window function.

[0040] Next, the energy spectrum and phase spectrum are obtained according to the real part and the imaginary part. As an example, the energy spectrum can be determined according to the following formula:

[0041] PCEN(t,f)=(E(t,f) / (M(t,f)+ε) α +δ) r -δ r , where PCEN(t, f) represents energy normalization for each frequency band channel, t represents the time domain sequence, f represents the frequency domain sequence, and ε represents a small constant (for example, ε can be 10 -8 ), α represents the energy normalization coefficient, δ and the exponent r represent the dynamic range coefficient, E(t,f) represents the energy spectrum, and M(t,f) represents the smoothed energy spectrum, which is determined according to the following formula: M(t,f) = (1-s)·M(t-1,f)+s·E(t,f), where s represents the smoothing coefficient.

[0042] Afterwards, the execution subject may determine the phase spectrum. As an example, the execution subject may determine the correction coefficient as -2πthv, where h represents the frame shift and v represents the normalized frequency, and the compensated phase is moved to the range of (-π, π) to obtain a corrected phase spectrum. Thus, by correcting and compensating the phase spectrum, the problem of eliminating the phase of the sine wave changing with the frequency interval and frame shift is solved.

[0043] Finally, the real part, imaginary part, energy spectrum and phase spectrum are determined as the features of the target speech signal.

[0044] In this way, in the preprocessing stage, not only the speech spectrum is used, but also trainable per-channel energy normalization and phase compensation are added as network input. Per-channel energy normalization performs automatic gain control and dynamic range control on the Mel spectrum, supplementing more past time axis information. At the same time, its trainable parameters also better adapt to the denoising neural network, which is beneficial to improving the overall speech denoising effect. Phase compensation eliminates the problem that the phase of the sine wave changes with frequency interval and frame shift, providing more accurate phase information for the neural network.

[0045] Step 204: input the frequency spectrum features and the voiceprint features into a pre-trained denoising neural network to obtain denoising parameter features.

[0046] In some embodiments, the execution entity may input the frequency spectrum features and voiceprint features into a pre-trained denoising neural network to obtain denoising parameter features.

[0047] Here, the denoising neural network is usually used to characterize the correspondence between the features and voiceprint features and parameter features. Specifically, the parameter features may be amplitude mask parameters, noise mask parameters, gain parameters and / or angle coefficients.

[0048] As an example, the above-mentioned denoising neural network can be a correspondence table determined by researchers based on a large amount of data. The correspondence table contains a large number of sample features and sample voiceprint features as well as sample parameter features corresponding to the sample features and sample voiceprint features. The above-mentioned execution entity can find sample features and sample voiceprint features that are similar or identical to the above-mentioned frequency spectrum features and voiceprint features, and determine the corresponding sample parameter features as parameter features.

[0049] As another example, the denoising neural network can be trained according to the following steps: obtaining a training sample set, wherein the training sample set includes sample features and sample voiceprint features and parameter features corresponding to the sample features and the sample voiceprint features; inputting the sample features and the sample voiceprint features into the model to be trained to obtain sample parameter features; comparing the parameter features and the sample parameter features according to a preset loss function to determine a loss value; in response to the loss value satisfying a preset condition, determining that the training of the model to be trained is completed, and determining the model to be trained as a denoising neural network. In response to the loss value not satisfying the preset condition, adjusting the relevant parameters in the model to be trained.

[0050] In some optional implementations of some embodiments, the denoising neural network includes a sequentially connected convolution module, a self-attention module, a gated recurrent unit module and a deconvolution module.

[0051] like Figure 3As shown, the above-mentioned denoising neural network can be a neural network composed of a convolution module 301, a self-attention module 302, a gated recurrent unit module 303 and a deconvolution module 304.

[0052] Among them, the convolution module 301 is as follows Figure 4 As shown. The convolution module 301 is composed of 6 convolution units with the same structure and different parameters. Each convolution unit is composed of a point convolution layer 401 (PConv), a batch normalization layer 402 (BatchNorm), a linear rectification function layer 403 (ReLU), a depth convolution layer 404 (DSConv), a batch normalization layer 405 (BatchNorm), and a linear rectification function layer 406 (ReLU) in sequence. The output of the last linear rectification function layer (Linear rectification function, ReLU) of each convolution unit is sent to the next layer of the network and to the corresponding deconvolution unit of the deconvolution module 407 as partial input. The convolution kernel of the point convolution layer of each convolution unit is 1, and the step size is 1. The convolution kernels of the depth convolution layer of each convolution unit are 5, 3, 5, 3, 5, 3, and the step sizes are 2, 1, 2, 1, 2, 2, respectively.

[0053] The self-attention module 302 is as follows Figure 5 As shown. The self-attention module is composed of a multi-head self-attention layer 502MultiheadAttention), a normalization layer 503LayerNorm), a point convolution layer 504PConv), a batch normalization layer 505BatchNorm), and a linear rectifier function layer 506ReLU). First, the output of the convolution module is concatenated with the voiceprint feature 501 of the target speaker's speech signal, and sent to the multi-head self-attention layer 502 (MultiheadAttention), and then after the layer normalization layer 503 (LayerNorm), it enters the point convolution layer 504 (PConv), and finally processed by the batch normalization layer 505 (BatchNorm) and the linear rectifier function layer 506 (ReLU) and sent to the next module.

[0054] The gated recurrent unit module 303 is composed of a gated recurrent unit layer (GRU), a point convolution layer (PConv), a batch normalization layer (BatchNorm) and a linear rectifier layer (ReLU). First, the output of the self-attention module is concatenated with the voiceprint features of the target speaker's speech signal and sent to the gated recurrent unit layer (GRU), and then passes through the point convolution layer (PConv), batch normalization layer (BatchNorm) and linear rectifier layer (ReLU) in sequence.

[0055] The deconvolution module 304 is composed of 6 deconvolution units with the same structure but different parameters. Each deconvolution unit is composed of a point convolution layer (PConv), a batch normalization layer (BatchNorm), a linear rectifier function layer (ReLU), a deconvolution layer (ConvT), a batch normalization layer (BatchNorm), and a linear rectifier function layer (ReLU) in sequence. The input of the first PConV of each deconvolution unit is received from both the upper layer of the network and the last RuLU layer of the corresponding convolution unit of the convolution module, and the two parts are spliced ​​as the overall input. The convolution kernel of the point convolution layer of each deconvolution unit is 1, and the step size is 1. The convolution kernels of the deconvolution layers of each deconvolution unit are 3, 5, 3, 5, 3, 5, and the step sizes are 2, 2, 1, 2, 1, 2, respectively.

[0056] In some optional implementations of some embodiments, the self-attention module includes a multi-head self-attention layer, a normalization layer, a point convolution layer, a batch normalization layer and / or a linear rectification function layer.

[0057] In some optional implementations of some embodiments, the gated recurrent unit module includes a gated recurrent unit layer, a point convolution layer, a batch normalization layer and / or a linear rectification function layer.

[0058] Here, the convolution module of the denoising neural network plays the role of compressing the spectral dimension. Point convolution and depth convolution are used, which reduces the number of parameters and computational complexity of convolution compared with the standard one-step convolution. The self-attention module used integrates the frequency domain features extracted by the convolution module. The self-attention network is used instead of the recurrent neural network because the self-attention network is more capable of capturing global information. The gated recurrent unit is used to process the relationship between each previous and next speech frame and establish the information flow of the time axis. The gated recurrent unit is used instead of the long short-term memory network to further reduce the computational complexity of the network. The voiceprint feature vector of the target speaker is added to the self-attention module and the gated recurrent unit module respectively, so that the network can learn to retain the voice of the target speaker in a targeted manner and eliminate other voices and noise. The deconvolution module restores the compressed frequency domain features to the original spectral dimension to output the frequency domain mask. The deconvolution module adopts a symmetrical structure with the convolution module. The input not only receives the output of the upper layer of the network, but also directly receives the output of the corresponding unit of the convolution module. It can make the gradient change more stable during network training and make it easier to train a good network model. At the same time, the overall neural network is lightweight. Except for the voiceprint extractor, the network parameters that need to run in real time are only about 700k. It has a small number of parameters, small amount of calculation, low system resource usage, and good noise reduction effect.

[0059] Step 205: post-process the target speech signal according to the parameter characteristics to obtain a noise-reduced speech signal.

[0060] In some embodiments, the execution subject may perform post-processing on the target speech signal to obtain a noise-reduced speech signal based on the parameter features obtained in step 204. Here, the post-processing generally refers to the operation of optimizing the frequency spectrum features using the parameter features and restoring them to a speech signal.

[0061] In some optional implementations of some embodiments, the execution subject may determine the phase-aware mask gain coefficient according to the parameter characteristics. Here, the phase-aware mask gain coefficient generally refers to the speech signal amplitude mask gain coefficient and the speech signal phase mask gain coefficient.

[0062] As an example, the above parameter features may be the amplitude mask parameter z, the noise mask parameter z_n, the gain parameter φ β , angle coefficient φ1 and angle coefficient φ2. The above execution body can be composed of gain coefficient φ β Calculate the lower limit of the amplitude mask gain multiplier: Then set an upper limit for the amplitude mask gain multiple to obtain the actual amplitude mask gain multiple β = min (β s ,1 / |sigmoid(z-z_n)-sigmoid(z_n-z)|), where the sigmoid function is represented by sigmoid(x)=1 / (1+e -x ). Then we can calculate the speech signal amplitude mask M = β·sigmoid(z-z_n) and the noise amplitude mask M_n = β·sigmoid(z_n-z). According to the cosine theorem, we can calculate the cosine expression of the absolute phase difference cos(Δθ) = (1+|M| 2 -|M_n| 2 ) / (2|M|). Then the directional coefficient is calculated from the angle coefficient φ1 and the angle coefficient φ2. Directivity coefficient Then calculate the angle direction So the speech signal phase mask e is calculated jθ =cos(Δθ)+jξsin(Δθ).

[0063] Afterwards, a noise reduction energy spectrum and a noise reduction phase spectrum are generated according to the phase-aware mask gain coefficient, the energy spectrum and the phase spectrum. As an example, the execution subject can use the speech signal amplitude mask gain coefficient and the speech signal phase mask gain coefficient to act on the amplitude spectrum and the phase spectrum respectively to obtain the amplitude spectrum and the phase spectrum of the noise reduction speech signal.

[0064] Next, the denoised real part and the denoised imaginary part of the denoised frequency spectrum feature are determined according to the denoised energy spectrum and the denoised phase spectrum, and then the denoised real part and the denoised imaginary part are subjected to windowing processing and inverse short-time Fourier transform to obtain the denoised speech signal.

[0065] The phase-aware mask is a complex mask that acts not only on the amplitude spectrum but also on the phase spectrum. The phase information provided can help better restore pure speech. In addition, the amplitude mask in the phase-aware mask has no range restrictions and can cover more scenarios with different signal-to-noise ratios.

[0066] One of the above embodiments of the present disclosure has the following beneficial effects: firstly, a target speech signal and a target speaker speech signal are obtained, then the target speaker speech signal is feature extracted to obtain the target speaker's voiceprint feature, then the target speech signal is preprocessed to obtain the frequency spectrum feature of the target speech signal, and then the frequency spectrum feature and the voiceprint feature are input into a pre-trained denoising neural network to obtain a denoising parameter feature, and finally the target speech signal is post-processed according to the parameter feature to obtain a denoised speech signal. Thus, by processing the target speech signal, the interference of background noise and other human voices is eliminated in the obtained denoised speech signal, achieving a lighter denoising process and a better denoising effect.

[0067] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a real-time speech noise reduction device based on a target person, and these device embodiments are similar to Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0068] like Figure 6 As shown, some embodiments of the target person-based real-time speech denoising device 600 include: an acquisition unit 601, an extraction unit 602, a pre-processing unit 603, a generation unit 604 and a post-processing unit 605. The acquisition unit 601 is configured to acquire a target speech signal and a target speaker speech signal; the extraction unit 602 is configured to perform feature extraction on the target speaker speech signal to obtain the target speaker's voiceprint feature; the pre-processing unit 603 is configured to pre-process the target speech signal to obtain the frequency spectrum feature of the target speech signal; the generation unit 604 is configured to input the frequency spectrum feature and the voiceprint feature into a pre-trained denoising neural network to obtain a denoising parameter feature, wherein the denoising neural network includes a sequentially connected convolution module, a self-attention module, a gated recurrent unit module and / or a deconvolution module; the post-processing unit 605 is configured to post-process the target speech signal according to the parameter feature to obtain a denoised speech signal.

[0069] In some optional implementations of some embodiments, the denoising neural network includes a sequentially connected convolution module, a self-attention module, a gated recurrent unit module and a deconvolution module.

[0070] In an optional implementation of some embodiments, the self-attention module includes a multi-head self-attention layer, a normalization layer, a point convolution layer, a batch normalization layer and / or a linear rectification function layer.

[0071] In an optional implementation of some embodiments, the gated recurrent unit module includes a gated recurrent unit layer, a point convolution layer, a batch normalization layer and / or a linear rectification function layer.

[0072] In some optional implementations of the embodiments, the preprocessing unit is further configured to: perform framing, windowing, and short-time Fourier transform on the target speech signal to obtain the real and imaginary parts of the frequency spectrum of the target speech signal; obtain an energy spectrum and a phase spectrum based on the real and imaginary parts of the frequency spectrum of the target speech signal; and use the energy spectrum, phase spectrum, and real and imaginary parts of the frequency spectrum of the target speech signal as frequency spectrum features of the target speech signal.

[0073] In optional implementations of some embodiments, the post-processing unit is further configured to: determine the phase-aware mask gain coefficient based on the above-mentioned parameter characteristics; generate a noise reduction energy spectrum and a noise reduction phase spectrum based on the above-mentioned phase-aware mask gain coefficient, the above-mentioned energy spectrum and the above-mentioned phase spectrum; determine the noise reduction real part and the noise reduction imaginary part of the noise reduction frequency spectrum characteristics based on the above-mentioned noise reduction energy spectrum and the noise reduction phase spectrum; perform windowing and inverse short-time Fourier transform on the above-mentioned noise reduction real part and noise reduction imaginary part to obtain a noise reduction speech signal.

[0074] In an optional implementation of some embodiments, the extraction unit is further configured to: input the target speaker's speech signal into a pre-trained voiceprint feature extraction network to obtain the voiceprint features of the target speaker, wherein the voiceprint feature extraction network includes a deep residual layer and a local aggregation vector layer, the deep residual layer is used to extract segmented speech signal features, and the local aggregation vector layer is used to aggregate segmented speech signal features.

[0075] It is understood that the units described in the device 600 are similar to those described in the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 600 and the units included therein, and will not be described in detail here.

[0076] One of the above embodiments of the present disclosure has the following beneficial effects: firstly, a target speech signal and a target speaker speech signal are obtained, then the target speaker speech signal is feature extracted to obtain the target speaker's voiceprint feature, then the target speech signal is preprocessed to obtain the frequency spectrum feature of the target speech signal, and then the frequency spectrum feature and the voiceprint feature are input into a pre-trained denoising neural network to obtain a denoising parameter feature, and finally the target speech signal is post-processed according to the parameter feature to obtain a denoised speech signal. Thus, by processing the target speech signal, the interference of background noise and other human voices is eliminated in the obtained denoised speech signal, achieving a lighter denoising process and a better denoising effect.

[0077] Reference below Figure 7 , which shows an electronic device (eg, Figure 1 A schematic diagram of the structure of the server 700. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0078] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0079] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0080] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0081] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0082] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0083] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a target speech signal and a target speaker speech signal; performs feature extraction on the target speaker speech signal to obtain the voiceprint feature of the target speaker; pre-processes the target speech signal to obtain the frequency spectrum feature of the target speech signal; inputs the frequency spectrum feature and the voiceprint feature into a pre-trained denoising neural network to obtain denoising parameter features, wherein the denoising neural network includes a sequentially connected convolution module, a self-attention module, a gated recurrent unit module and / or a deconvolution module; and post-processes the target speech signal according to the parameter features to obtain a denoised speech signal.

[0084] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0085] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0086] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes an acquisition unit, an extraction unit, a preprocessing unit, a generation unit, and a post-processing unit. The names of these units do not constitute a limitation on the units themselves in some cases, for example, the receiving unit may also be described as a "unit for acquiring a target speech signal and a target speaker speech signal".

[0087] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0088] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above frequency spectrum features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) by each other.

Claims

1. A real-time speech noise reduction method based on a target person, comprising: Acquire a target speech signal and a target speaker speech signal; Extracting features from the speech signal of the target speaker to obtain the voiceprint features of the target speaker; Preprocessing the target speech signal to obtain frequency spectrum characteristics of the target speech signal; Inputting the frequency spectrum characteristics of the target speech signal and the voiceprint characteristics of the target speaker into a pre-trained denoising neural network to obtain denoising parameter characteristics of the target speech signal; Post-processing the target speech signal according to the noise reduction parameter characteristics to obtain a noise-reduced speech signal; The frequency spectrum characteristics of the target speech signal include the energy spectrum, phase spectrum, real part and imaginary part of the frequency spectrum of the target speech signal; The denoising neural network is trained according to the following steps: obtaining a training sample set, wherein the training sample set includes sample features and sample voiceprint features and parameter features corresponding to the sample features and the sample voiceprint features; inputting the sample features and the sample voiceprint features into the model to be trained to obtain sample parameter features; comparing the parameter features and the sample parameter features according to a preset loss function to determine a loss value; in response to the loss value satisfying a preset condition, determining that the training of the model to be trained is completed, and determining the model to be trained as a denoising neural network; The post-processing of the target speech signal according to the noise reduction parameter feature comprises: Determining a phase-aware mask gain coefficient according to the noise reduction parameter characteristics; Generate a noise reduction energy spectrum and a noise reduction phase spectrum according to the phase-aware mask gain coefficient, the energy spectrum and the phase spectrum; Determine the real part and the imaginary part of the noise reduction frequency spectrum according to the noise reduction energy spectrum and the noise reduction phase spectrum; The real part and the imaginary part of the noise reduction frequency spectrum are subjected to windowing processing and inverse short-time Fourier transform to obtain a noise reduction speech signal.

2. The real-time speech noise reduction method according to claim 1, wherein: The denoising neural network includes a sequentially connected convolution module, a self-attention module, a gated recurrent unit module and a deconvolution module.

3. The real-time speech noise reduction method according to claim 2, wherein: The self-attention module includes a multi-head self-attention layer, a normalization layer, a point convolution layer, a batch normalization layer and / or a linear rectification function layer.

4. The real-time speech noise reduction method according to claim 2, wherein: The gated recurrent unit module includes a gated recurrent unit layer, a point convolution layer, a batch normalization layer and / or a linear rectification function layer.

5. The real-time speech noise reduction method according to claim 1, wherein: The preprocessing of the target speech signal comprises: The target speech signal is framed, windowed, and short-time Fourier transformed to obtain a real part and an imaginary part of a frequency spectrum of the target speech signal; Obtaining an energy spectrum and a phase spectrum according to the real part and the imaginary part of the frequency spectrum of the target speech signal; The energy spectrum, phase spectrum, real part and imaginary part of the frequency spectrum of the target speech signal are used as the frequency spectrum features of the target speech signal.

6. The real-time speech noise reduction method according to claim 1, wherein: The feature extraction of the target speaker's speech signal comprises: The target speaker's speech signal is input into a pre-trained voiceprint feature extraction network to obtain the target speaker's voiceprint features, wherein the voiceprint feature extraction network includes a deep residual layer and a local aggregation vector layer, the deep residual layer is used to extract segmented speech signal features, and the local aggregation vector layer is used to aggregate segmented speech signal features.

7. A real-time speech noise reduction device based on a target person, comprising: An acquisition unit, configured to acquire a target speech signal and a target speaker speech signal; an extraction unit, configured to perform feature extraction on the speech signal of the target speaker to obtain a voiceprint feature of the target speaker; A preprocessing unit is configured to preprocess the target speech signal to obtain a frequency spectrum feature of the target speech signal; wherein the frequency spectrum feature of the target speech signal includes an energy spectrum, a phase spectrum, and a real part and an imaginary part of a frequency spectrum of the target speech signal; The denoising unit is configured to input the frequency spectrum features of the target speech signal and the voiceprint features of the target speaker into a pre-trained denoising neural network to obtain denoising parameter features of the target speech signal; the denoising neural network is trained according to the following steps: obtaining a training sample set, wherein the training sample set includes sample features and sample voiceprint features and parameter features corresponding to the sample features and the sample voiceprint features; inputting the sample features and the sample voiceprint features into a model to be trained to obtain sample parameter features; comparing the parameter features and the sample parameter features according to a preset loss function to determine a loss value; in response to the loss value satisfying a preset condition, determining that the training of the model to be trained is completed, and determining the model to be trained as a denoising neural network; A post-processing unit is configured to post-process the target speech signal according to the noise reduction parameter feature to obtain a noise-reduced speech signal, including: Determining a phase-aware mask gain coefficient according to the noise reduction parameter characteristics; Generate a noise reduction energy spectrum and a noise reduction phase spectrum according to the phase-aware mask gain coefficient, the energy spectrum and the phase spectrum; Determine the real part and the imaginary part of the noise reduction frequency spectrum according to the noise reduction energy spectrum and the noise reduction phase spectrum; The real part and the imaginary part of the noise reduction frequency spectrum are subjected to windowing processing and inverse short-time Fourier transform to obtain a noise reduction speech signal.

8. The real-time speech noise reduction device according to claim 7, wherein: The denoising neural network includes a sequentially connected convolution module, a self-attention module, a gated recurrent unit module and a deconvolution module.

9. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

Images