Speech processing method, device, storage medium, and program product

Abstract

The application provides a speech processing method and device, a storage medium and a program product. The method comprises the following steps: determining feature information corresponding to a to-be-processed speech in multiple frequency bands; obtaining a processing result corresponding to each feature information based on a deep learning model for processing sequence data, for a sequence formed by the feature information in the multiple frequency bands; and obtaining a processed speech based on the processing result corresponding to each feature information. The application can simultaneously realize denoising and dereverberation based on the deep learning model, avoid error propagation caused by the connection of different algorithm modules, improve the accuracy of the model, and effectively reduce the size of the network model and the calculation amount during processing, thereby improving the processing efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech processing, and particularly relates to a speech processing method and device, a storage medium and a program product. BACKGROUND

[0002] Speech enhancement technology can extract useful speech signals from noisy speech signals, and restore pure original speech as much as possible, and plays a very important role in real-time speech communication.

[0003] In the process of real-time speech communication, not only are there different types of environmental noise, but also there is the interference of room reverberation. Therefore, the speech processing model needs to be able to simultaneously realize the functions of denoising and dereverberation. At present, the existing speech processing model has the problems of low efficiency and poor accuracy, and the problems of separating denoising and dereverberation as two problems and not considering joint processing. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a speech processing method, device, storage medium and program product to improve the efficiency and accuracy of speech processing.

[0005] In a first aspect, the embodiments of the present application provide a speech processing method, comprising:

[0006] determining feature information corresponding to a to-be-processed speech on a plurality of frequency bands;

[0007] forming a sequence for the feature information on the plurality of frequency bands, and obtaining a processing result corresponding to each feature information based on a deep learning model for processing sequence data;

[0008] obtaining a processed speech based on the processing result corresponding to each feature information.

[0009] Optionally, the method further comprises:

[0010] obtaining the to-be-processed speech according to the obtained speech stream and a preset time window;

[0011] performing short-time Fourier transform on the to-be-processed speech to obtain frequency domain information corresponding to the to-be-processed speech on a plurality of frequency bands;

[0012] The feature information is the frequency domain information, or the feature information is feature information corresponding to the frequency domain information extracted by a feature extractor.

[0013] Optionally, determining the feature information corresponding to the to-be-processed speech on the plurality of frequency bands comprises:

[0014] inputting the frequency domain information corresponding to the to-be-processed speech on the plurality of frequency bands into a feature extractor to extract feature information corresponding to each frequency band;

[0015] The feature extractor comprises a feature extractor in a time domain and / or a feature extractor in a frequency domain.

[0016] Optionally, the corresponding frequency domain information of the to-be-processed speech in multiple frequency bands is input to the feature extractor to extract the corresponding feature information of each frequency band, comprising:

[0017] The corresponding frequency domain information of the to-be-processed speech in the speech stream is input to the first feature extraction network in a time sequence to obtain the first feature information of each frequency band corresponding to the to-be-processed speech;

[0018] The first feature information of each frequency band is input to the second feature extraction network in a frequency sequence to obtain the second feature information of each frequency band; the second feature information is used to input to a deep learning model to obtain a processing result;

[0019] The first feature extraction network comprises a two-dimensional convolutional neural network; and the second feature extraction network comprises a bidirectional LSTM network.

[0020] Optionally, the first feature extraction network further comprises a batch normalization layer.

[0021] The input end of the batch normalization layer is connected with the output end of the two-dimensional convolutional neural network, and the output end of the batch normalization layer is connected with the input end of the bidirectional LSTM network.

[0022] The second feature extraction network further comprises a full connection layer and a linear rectifier unit.

[0023] The input end of the full connection layer is connected with the output end of the bidirectional LSTM network, and the output end of the full connection layer is connected with the linear rectifier unit.

[0024] Optionally, the deep learning model comprises a unidirectional LSTM network and a full connection layer; the input end of the unidirectional LSTM network is connected with the output end of the second feature extraction network, and the input end of the full connection layer is connected with the output end of the unidirectional LSTM network; and the full connection layer is used to obtain the complex ideal ratio masking information corresponding to each frequency band.

[0025] Correspondingly, the processed speech is obtained based on the processing result corresponding to each feature information, comprising:

[0026] The de-noising and de-reverberation information of the to-be-processed speech in the frequency domain is calculated based on the frequency domain signals of the to-be-processed speech in multiple frequency bands and the corresponding complex ideal ratio masking information.

[0027] The de-noised and de-reverberated information in the frequency domain is subjected to inverse short-time Fourier transform to obtain de-noised and de-reverberated speech.

[0028] Optionally, the method further comprises:

[0029] Obtaining at least one of the following information input by a user: length of a time window for intercepting speech to be processed from a speech stream, information of a plurality of frequency bands, and output dimension of each module in a feature extractor and a deep learning model;

[0030] Model training is performed according to the at least one information.

[0031] Optionally, a loss function used in the model training process is determined by L2 norm of a difference between a speech signal obtained based on the deep learning model and an actual clean speech signal and L2 norm of the actual clean speech signal.

[0032] In a second aspect, an embodiment of the present application further provides a speech processing method applied to a first device, and the method comprises:

[0033] Obtaining single-channel speech collected by a second device during a conference;

[0034] Determining corresponding feature information of the single-channel speech on a plurality of frequency bands;

[0035] Forming a sequence based on the feature information on the plurality of frequency bands, and obtaining a processing result corresponding to each frequency domain information based on a deep learning model for processing sequence data;

[0036] Obtaining processed speech based on the processing result corresponding to each frequency domain information and playing the processed speech.

[0037] In a third aspect, an embodiment of the present application provides an electronic device, which comprises:

[0038] At least one processor; and

[0039] A memory in communication connection with the at least one processor;

[0040] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to perform the method of any one of the above aspects.

[0041] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer execution instructions, and when a processor executes the computer execution instructions, the method of any one of the above aspects is implemented.

[0042] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method of any of the preceding aspects.

[0043] The voice processing method, device, storage medium and program product provided in the present application can determine the feature information corresponding to the to-be-processed voice in multiple frequency bands, form a sequence based on the feature information in the multiple frequency bands, obtain a processing result corresponding to each feature information based on a deep learning model for processing sequence data, and obtain the voice after denoising and dereverberation based on the processing result corresponding to each feature information. The deep learning model can be used to simultaneously realize denoising and dereverberation, avoid error propagation caused by the connection of different algorithm modules, and reference the information of other frequency bands when processing the information of each frequency band based on the sequence in the frequency domain. Since the reverberation is frequency-dependent, the processing is performed according to the frequency sequence, which can effectively enhance the dereverberation effect and improve the accuracy of the model. In addition, the network model coefficients of each frequency band are shared, which can effectively reduce the size of the network model and the amount of calculation during processing, and improve the processing efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the present application.

[0045] Figure 1 An application scenario diagram of an embodiment of the present application;

[0046] Figure 2 A flowchart of a voice processing method provided by an embodiment of the present application;

[0047] Figure 3 A principle diagram of a feature extractor in the time domain provided by an embodiment of the present application;

[0048] Figure 4 A principle diagram of a feature extractor in the frequency domain provided by an embodiment of the present application;

[0049] Figure 5 A principle diagram of dereverberation and denoising by a model provided by an embodiment of the present application;

[0050] Figure 6 A principle diagram of a feature extractor and a subnetwork provided by an embodiment of the present application;

[0051] Figure 7 A flowchart of extracting feature information provided by an embodiment of the present application;

[0052] Figure 8 A flowchart of another voice processing method provided by an embodiment of the present application;

[0053] Figure 9 A structural schematic diagram of an electronic device is provided for an embodiment of the present application.

[0054] The specific embodiments of the present application have been shown through the above-described drawings, and will be described in more detail hereinafter. These drawings and the written description are not intended to restrict the scope of the present application concept in any way, but to illustrate the present application concept to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0055] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The same or similar components are denoted by the same or similar reference numerals throughout the drawings and the written description, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application.

[0056] First, the terms involved in the present application are explained:

[0057] Speech Denoising: Also known as noise elimination, the speech containing noise received by the microphone is removed by the algorithm module to preserve the fidelity of the original speech signal as much as possible.

[0058] Speech Dereverberation: The speech containing reverberation received by the microphone is removed by the algorithm module to achieve the effect that the original speech does not contain reverberation.

[0059] Long short-term memory (LSTM) network: A kind of recurrent neural network that can learn long-term dependent information, uni-directional (single direction) can be used in real-time operation, bi-directional (double direction) can better learn the information before and after the dependence, if it is applied to the time axis direction then not suitable for real-time streaming operation.

[0060] Short-time Fourier transform (STFT): A time-frequency conversion method used to determine the frequency domain information of the local region of the time-varying signal.

[0061] The embodiments of the present application can be applied to any scene that needs to process speech, and can be particularly applied to processing speech in an audio-video communication system.

[0062] In audio and video communication systems, various complex spatial acoustic environments will inevitably be encountered, especially different reverberations in different spaces (rooms, conference rooms), different types of environmental noise (steady noise, non-steady noise), therefore, real-time speech enhancement technology has become a key technology of audio and video communication systems, especially in the case of single channel and limited hardware resources.

[0063] Single-channel real-time speech de-noising aims to remove environmental noise in speech, especially non-steady noise, and the commonly used noise suppression algorithm is one of the audio 3A algorithms; similarly, single-channel real-time speech de-reverberation aims to remove the reverberation effect in space and improve speech recognition, and the commonly used solution is generally a spectral enhancement to remove the late reverberation effect.

[0064] The modules of the traditional solution often focus on solving only one of the de-noising and de-reverberation problems, and when de-noising and de-reverberation are required at the same time, the two modules are connected in series, which may result in worse effects. To solve this problem, an algorithm is urgently needed to remove noise and reverberation in real time and at the same time, so as to solve the above two problems in the same module, simplify the link and reduce the amount of calculation.

[0065] Figure 1 An application scenario of an embodiment of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, in an online audio and video conference, when a first user speaks, the original human voice is superimposed with noise and reverberation effects, and a speech signal containing noise and reverberation is obtained, which can be transmitted by the device of the first user to the device of a second user. After the device of the second user obtains the speech signal, the speech signal can be de-noised and de-reverberated by a model, and a speech signal without noise and reverberation is obtained and played to the second user, so that online real-time communication between the first user and the second user can be realized, and the effect of speech communication can be improved.

[0066] A clean speech signal can be obtained through the model, which can achieve de-noising and de-reverberation to a certain extent, but the efficiency and accuracy need to be improved.

[0067] Therefore, an embodiment of the present application provides a speech processing method based on deep narrowband filtering, which can simultaneously achieve de-noising and de-reverberation in the same model, and can optimize the model size and algorithm complexity for real-time calculation, aiming to solve the pain point problem of single-channel speech enhancement in real-time speech communication.

[0068] The deep narrowband filtering can refer to that a network model coefficient is trained by a deep learning method, the coefficient is equivalent to a filter coefficient to filter each frequency band signal in a frequency domain, and the network model coefficients of each frequency band are shared, that is, the same set of coefficients is used for filtering different frequency bands, the size of the network coefficient is reduced, and the calculation amount is reduced.

[0069] Specifically, the feature information corresponding to the to-be-processed voice in multiple frequency bands can be extracted, and the feature information in each frequency band is sequentially input to a deep learning model capable of processing sequence data, such as LSTM, so as to refer to the information of adjacent frequency bands when processing the information of each frequency band. Since the reverberation is frequency-dependent, the processing is performed according to the frequency band sequence, which can effectively enhance the effect of reverberation removal and improve the accuracy of the model. In addition, the deep learning model can make the output result approximate to the real clean signal, so it also has the ability to remove noise.

[0070] In addition, compared with the scheme of directly inputting the entire spectrum information (full-band information) into the model for processing, the present application can effectively reduce the model size and the calculation amount, and improve the processing efficiency.

[0071] Specifically, the full-band scheme needs to directly process the full-band information, and the input dimension is large, so the model needs more parameters, resulting in a large model. In addition, due to the complexity of the deep learning model, a large number of nodes in the model may have a dependency relationship with each input data. Therefore, when the input dimension increases, the calculation amount of the model will increase much more than the increase of the input dimension, resulting in excessive calculation amount. In the embodiment of the present application, the parameter amount of the model only needs to be able to process the feature information of a single frequency band. For each frequency band, the network model coefficients are shared, which can effectively reduce the data amount of the network model coefficients, reduce the calculation amount, and improve the processing efficiency. In addition, the model can be trained by sequentially inputting the single frequency band feature information during training, so that the model has good performance in different frequency bands, and efficiency and accuracy are considered.

[0072] Therefore, compared with the deep learning scheme of directly processing the entire spectrum information, the embodiment of the present application can effectively improve the efficiency and accuracy, and is very suitable for real-time voice communication, especially real-time voice communication on devices with limited hardware resources.

[0073] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and the features in the embodiments can be combined with each other without conflict. In addition, the sequence of steps in each method embodiment is only an example, not a strict limitation.

[0074] Figure 2A flowchart of a voice processing method provided in an embodiment of the present application is shown. The execution subject of the method in the embodiment can be any device with a data processing function, such as a server, a terminal device, etc. The terminal device can be a special-purpose device such as an audio-video integrated machine, or a general-purpose device such as a mobile phone or a personal computer. As shown in Figure 2 the method can include:

[0075] Step 201: Determine the feature information corresponding to the to-be-processed voice in multiple frequency bands.

[0076] Optionally, the to-be-processed voice can be a voice that needs to be de-noised and de-reverberated. Exemplarily, it can be a segment of voice stream collected in a conference process.

[0077] The to-be-processed voice can be extracted to have feature information corresponding to multiple frequency bands in the frequency domain. The frequency band can be represented by a frequency range, for example, a range of 90 Hz-150 Hz represents a frequency band, or can be represented by a corresponding center frequency, for example, a frequency band with a center frequency of 120 Hz.

[0078] Each frequency band can have corresponding feature information, which can be any information that can reflect the frequency domain feature of the to-be-processed voice in the frequency band, for example, frequency domain information obtained by short-time Fourier transform, or feature information obtained by further extracting the frequency domain information.

[0079] Step 202: Based on a deep learning model for processing sequence data, obtain a processing result corresponding to each feature information for the sequence formed by the feature information in the multiple frequency bands.

[0080] The deep learning model is a deep learning model capable of processing sequence data, and when the model calculates each data in the sequence, the information of other data in the sequence can be referred to.

[0081] Optionally, the deep learning model can be an LSTM model or an RNN (Recurrent Neural Network) model, etc.

[0082] When processing the feature information corresponding to multiple frequency bands, the feature information corresponding to the first frequency band, the feature information corresponding to the second frequency band, and so on can be input in sequence, and the feature information of each frequency band forms a sequence and is processed by the deep learning model. When processing the feature information of each frequency band, the feature information of one or more other frequency bands is referred to.

[0083] Since the deep learning model is used to process information on each frequency band, it can be referred to as a subband neural network (Subband neural network), simply referred to as a subnet. Hereinafter, the deep learning model used to process feature information corresponding to multiple frequency bands is referred to as a subnet unless otherwise specified.

[0084] In step 203, the processed speech is obtained based on the processing result corresponding to each feature information.

[0085] Optionally, for each frequency band, the processing result obtained after the subnet processing can be frequency domain information corresponding to the clean speech in the frequency band, or other intermediate results used to determine the clean speech, such as certain intermediate parameters. The final clean speech is obtained by calculating the intermediate parameters and the frequency domain information.

[0086] In actual applications, the device can obtain a speech stream containing noise and reverberation. A preset length of speech to be processed can be intercepted from the speech stream. For example, after receiving 20 ms of speech, the feature information of the speech in multiple frequency bands can be sequentially input to the subnet according to the above method, and the clean speech is obtained based on the processing result and played to the user.

[0087] In summary, the speech processing method provided in this embodiment can determine the feature information corresponding to the speech to be processed in multiple frequency bands, form a sequence based on the feature information in the multiple frequency bands, obtain the processing result corresponding to each feature information based on the deep learning model for processing sequence data, and obtain the speech after denoising and dereverberation based on the processing result corresponding to each feature information. The deep learning model can be used to simultaneously realize denoising and dereverberation, avoid error propagation caused by the concatenation of different algorithm modules, and reference the information of other frequency bands when processing the information of each frequency band based on the sequence in the frequency. Since the reverberation is frequency-dependent, processing according to the frequency sequence can effectively enhance the dereverberation effect and improve the accuracy of the model. In addition, the network model coefficients of each frequency band are shared, which can effectively reduce the size of the network model and the calculation amount during processing, and improve the processing efficiency.

[0088] In one or more embodiments of the present application, the deep learning model for processing sequence data can include: sequentially inputting the feature information in the multiple frequency bands to the deep learning model to obtain the complex ideal ratio mask (cIRM) corresponding to each frequency band.

[0089] Correspondingly, based on the processing result corresponding to each feature information, the processed speech can be obtained, which can include: according to the frequency domain signal of the to-be-processed speech in multiple frequency bands and the corresponding complex ideal ratio mask information, the de-noising and de-reverberation information of the to-be-processed speech in the frequency domain is calculated; the de-noising and de-reverberation information in the frequency domain is subjected to inverse short-time Fourier transform to obtain the de-noising and de-reverberation speech.

[0090] Optionally, the to-be-processed speech collected by the device can be a time domain speech signal:

[0091] y(t) = x(t) + n(t) = s(t) * h(t) + n(t) (1)

[0092] Wherein, x(t) is a reverberation speech signal, n(t) is an environmental noise, s(t) is a clean speech signal, and h(t) is a spatial impulse response (used to represent the reverberation effect).

[0093] After short-time Fourier transform, the frequency spectrum of the to-be-processed speech, i.e. the frequency domain information, can be obtained:

[0094] Y(t, f) = X(t, f) + N(t, f) (2)

[0095] Further, the complex ideal ratio mask M(t, f) in the frequency domain can be estimated based on the sub-network, and the processed frequency spectrum can be obtained according to M(t, f):

[0096]

[0097] The processed frequency spectrum after iSTFT (inverse STFT) obtains the processed time domain speech signal That is, the de-noising and de-reverberation clean speech signal.

[0098] In summary, the frequency domain information of the to-be-processed speech can be obtained by short-time Fourier transform, and the complex ideal ratio mask information can be obtained by the deep learning model, and the de-noising and de-reverberation clean speech signal can be calculated by the complex ideal ratio mask information, which can locate the output of the model in the complex ideal ratio mask information, simplify the output of the model, effectively improve the processing speed of the model, and improve the efficiency of speech processing.

[0099] In one or more embodiments of the present application, optionally, the to-be-processed speech can be obtained according to the obtained speech stream and the preset time window; the short-time Fourier transform is performed on the to-be-processed speech to obtain the corresponding frequency domain information of the to-be-processed speech in multiple frequency bands.

[0100] The preset time window can be set according to actual needs. For example, the length of the preset time window can be between 20 ms and 150 ms.

[0101] Optionally, the input of the subnetwork can be feature information, and the output can be complex ideal ratio masking information. The feature information can be the frequency domain information, or the feature information can be feature information corresponding to the frequency domain information extracted by a feature extractor.

[0102] In an example, the frequency domain information obtained after short-time Fourier transform can be directly input into the subnetwork to obtain the corresponding result. This way is simple in principle and easy to implement.

[0103] In another example, the frequency domain information obtained after short-time Fourier transform can be input into a feature extractor to extract feature information corresponding to the frequency domain information, and the feature information can be input into the subnetwork to obtain the corresponding result. This way can extract deeper features of the frequency domain information and improve the voice processing effect.

[0104] In summary, by extracting the voice to be processed from the voice stream according to the preset time window and performing short-time Fourier transform, the corresponding frequency domain information on multiple frequency bands is obtained, and then the clean signal in the time domain is restored according to the frequency domain information and the deep learning model. The voice stream can be decomposed and processed in the time and frequency domain dimensions, fine-grained processing of the voice stream is realized, and the processing efficiency and accuracy of the voice stream are improved.

[0105] In one or more embodiments of the present application, optionally, the feature information corresponding to the voice to be processed on multiple frequency bands is determined, including: inputting the frequency domain information corresponding to the voice to be processed on multiple frequency bands into a feature extractor to extract the feature information corresponding to each frequency band; wherein the feature extractor includes a time-domain feature extractor and / or a frequency-domain feature extractor.

[0106] The input into the time-domain feature extractor can mean that the frequency domain information is sequentially input into the feature extractor in time sequence; and the input into the frequency-domain feature extractor can mean that the frequency domain information is sequentially input into the feature extractor in frequency sequence.

[0107] Figure 3 A principle schematic diagram of a time-domain feature extractor provided by an embodiment of the present application is shown in FIG. 3. Figure 3As shown, the speech stream can be divided into multiple time periods according to a time window, denoted as a t1 period, a t2 period, a t3 period, and so on, each time period corresponding to a to-be-processed speech. When input to the feature extractor in the time domain, the information can be sequentially input to the feature extractor in the sequence in the time direction, that is, the frequency domain information corresponding to the t1 period is input first, and then the frequency domain information corresponding to the t2 period is input after the frequency domain information corresponding to the t1 period is input. Therefore, the information is sequentially input to the feature extractor in the time direction, and therefore, the feature extractor can be referred to as a feature extractor in the time domain.

[0108] Figure 4 A principle schematic diagram of a feature extractor in the frequency domain provided by an embodiment of the present application is shown in FIG. 3. Figure 4 As shown, each to-be-processed speech in the speech stream can be divided into multiple frequency bands according to a frequency window, denoted as a frequency band f1, a frequency band f2, a frequency band f3, and so on. When input to the feature extractor in the frequency domain, the information can be sequentially input to the feature extractor in the sequence in the frequency direction, that is, the frequency domain information corresponding to the frequency band f1 is input first, and then the frequency domain information corresponding to the frequency band f2 is input after the frequency domain information corresponding to the frequency band f1 is input. Therefore, the information is sequentially input to the feature extractor in the time direction, and therefore, the feature extractor can be referred to as a feature extractor in the frequency domain.

[0109] Optionally, the feature information of the to-be-processed speech can also be extracted by the feature extractor in the time domain and the feature extractor in the frequency domain. The feature extractor in the time domain and the feature extractor in the frequency domain can extract the features of the to-be-processed speech in the Spectro-temporal respective fields (STRF).

[0110] Figure 5 A principle schematic diagram of denoising and dereverberation by a model provided by an embodiment of the present application is shown in FIG. 4. Figure 5 As shown, the frequency domain information Y(t, f) obtained after the time domain speech signal y(t) is subjected to the STFT can have two dimensions of time and frequency. Exemplarily, it is assumed that the speech stream is two-dimensionally divided in the time and frequency directions according to a time window and a frequency window. In the time dimension, it can be divided into a period t, a period t+1, and so on. In the frequency dimension, it can be divided into a frequency band f-1, a frequency band f, a frequency band f+1, and so on until a frequency band F. Then, the information in the two dimensions of time and frequency can be processed based on a model. The output result of the model is calculated with Y(t, f), and the spectrum of the clean speech signal can be obtained. Then, the iSTFT is performed, and the clean speech signal in the time domain can be obtained. The clean speech signal obtained by the model The actual clean speech signal s(t) can be used to construct a loss function to optimize the model. The model can include the subnetwork described above, and can also include a feature extractor.

[0111] In summary, by inputting the frequency domain information corresponding to the to-be-processed speech in multiple frequency bands into the feature extractor, extracting the feature information corresponding to each frequency band, and obtaining the processed speech signal based on the subnetwork according to the feature information, the deep features of the frequency domain information can be extracted, the model effect can be improved, and the accuracy of speech processing can be further improved.

[0112] Figure 6 A principle diagram of a feature extractor and a subnetwork provided by an embodiment of the present application is shown in FIG. 1. Figure 6 As shown in FIG. 1, the frequency domain information sequentially passes through a first feature extraction network, a second feature extraction network, and a subnetwork to obtain a processing result.

[0113] The first feature extraction network can be a feature extractor in the time domain, and the second feature extraction network can be a feature extractor in the frequency domain. The first feature extraction network and the second feature extraction network are connected in series, which can extract the features of the frequency domain information in the STRF domain and send them to the subnetwork for processing. The structure and principle of each network will be described in detail below.

[0114] Figure 7 A flowchart of extracting feature information provided by an embodiment of the present application is shown in FIG. 2. Figure 7 As shown in FIG. 2, the frequency domain information corresponding to the to-be-processed speech in multiple frequency bands is input into the feature extractor to extract the feature information corresponding to each frequency band, which can include the following steps.

[0115] Step 701: The frequency domain information corresponding to the to-be-processed speech in the speech stream is input into the first feature extraction network in a time sequence to obtain the first feature information corresponding to each frequency band of the to-be-processed speech.

[0116] The first feature extraction network can include a two-dimensional convolutional neural network (2D Conv).

[0117] For example, the speech stream containing noise and reverberation can be processed in a time window of 20 ms, and the frequency domain information of the 20 ms to-be-processed speech can be divided into 129 frequency bands. Assuming that the full frequency band is 0-8000 Hz, each frequency band differs by 8000 / 129≈62 Hz. For example, the center frequency of the first frequency band is 62 Hz, and the center frequency of the second frequency band is 124 Hz.

[0118] It should be noted that the numerical values listed in the present embodiment are only examples for convenience of explanation, and can be adjusted in actual applications.

[0119] In this step, the frequency domain information can be input into the first feature extraction network according to the sequence in time. Since there can be multiple frequency bands of frequency domain information in each time period, the frequency domain information of multiple frequency bands can be spliced and input into the first feature extraction network.

[0120] For example, referring to Figure 6 , the frequency domain information of 129 frequency bands in time period t is input first, and then the frequency domain information of 129 frequency bands in time period t+1 is input, and so on. Therefore, the first feature extraction network can be regarded as feature extraction on the time axis.

[0121] Optionally, the first feature extraction network further comprises a batch normalization layer (Batch Normalization). The input end of the batch normalization layer is connected with the output end of the two-dimensional convolutional neural network, and the output end of the batch normalization layer is connected with the input end of the second feature extraction network.

[0122] After the frequency domain information is processed by the two-dimensional convolutional neural network and the batch normalization layer, the first feature information with a dimension of D1 can be obtained.

[0123] For example, for each voice to be processed, the first feature extraction network can output D1-dimensional features corresponding to each frequency band, and 129 D1-dimensional features can be obtained for 129 frequency bands.

[0124] The batch normalization layer is added after the two-dimensional convolutional neural network, so that the distribution of the features extracted from the frequency domain information input in the time direction can be adjusted, so that the features input into the next layer network are more stable, and the overall effect of the model is improved.

[0125] In step 702, the first feature information corresponding to each frequency band is input into the second feature extraction network in the frequency sequence mode to obtain second feature information corresponding to each frequency band.

[0126] The second feature extraction network can comprise a bi-directional LSTM (bi-directional LSTM) network. The input end of the bi-directional LSTM network can be connected with the output end of the batch normalization layer in the first feature extraction network.

[0127] For example, for each voice to be processed, the first feature extraction network extracts 129 D1-dimensional features of frequency bands, and the D1-dimensional features of each frequency band can be input into the second feature extraction network in sequence. Therefore, the second feature extraction network can be regarded as a feature extraction on the frequency axis, and the number of bi-directional LSTMs therein can be one or more.

[0128] Optionally, the second feature extraction network can further include a full connection layer (Full) and a rectified linear unit (ReLu); an input end of the full connection layer is connected with an output end of the bidirectional LSTM network, and an output end of the full connection layer is connected with an input end of the rectified linear unit. An output end of the rectified linear unit can be connected with a subnetwork.

[0129] Exemplarily, the number of bidirectional LSTMs can be 2. For the first feature information of each frequency band, after passing through the two bidirectional LSTMs, the feature corresponding to the frequency band with a dimension of D2 is obtained, and then input to the full connection layer and the rectified linear unit to obtain the feature corresponding to the frequency band with a dimension of D3. Therefore, for each to-be-processed voice, the second feature network extraction unit can finally obtain the D3-dimensional features of 129 frequency bands, denoted as second feature information, which can be used for input to the subnetwork for processing.

[0130] Adding the full connection layer and the rectified linear unit after the bidirectional LSTM network can integrate the extracted features for each frequency band, reduce the problems of gradient explosion and gradient disappearance, reduce the overall network calculation cost, realize feature extraction on each frequency band, and further improve the overall performance of the model.

[0131] It should be noted that if the information is input to the bidirectional LSTM in time sequence, it is difficult to process real-time signals. In the embodiment of the present application, the information is input to the bidirectional LSTM in frequency sequence, and the adjacent two data only have the sequence relationship in frequency band, but not necessarily in time. Therefore, real-time processing of voice can be realized, real-time performance is improved, and actual application requirements are met.

[0132] Optionally, the deep learning model, i.e., the subnetwork, can include a unidirectional LSTM network and a full connection layer; an input end of the unidirectional LSTM network is connected with an output end of the second feature extraction network, and an input end of the full connection layer is connected with an output end of the unidirectional LSTM network; and the full connection layer is used to obtain the complex ideal ratio masking information corresponding to each frequency band.

[0133] Exemplarily, the second feature extraction network outputs the D3-dimensional features of each frequency band, and the D3-dimensional features of each frequency band are sequentially input to the subnetwork. Different frequency bands share the same set of subnetwork parameters, and each frequency band obtains two numbers as a result, which are the real part and the imaginary part of the complex ideal ratio masking, respectively. After sequentially inputting the feature information of 129 frequency bands in a time period, the feature information of 129 frequency bands in the next time period is sequentially input. After obtaining the complex ideal ratio masking information of each frequency band, the clean voice signal can be restored according to the complex ideal ratio masking information.

[0134] The one-way LSTM can guide the information processing of the subsequent frequency band according to the information of the previous frequency band, improve the accuracy of the processing result, and the full connection layer can obtain the real part and the imaginary part of the complex ideal ratio masking information, simplify the model structure, and further improve the efficiency of the speech processing.

[0135] In the embodiment, in order to better obtain the interdependent information of each frequency band, a two-dimensional convolutional neural network and a bidirectional LSTM network in the frequency direction are added to the frequency domain information of each frequency band, and the combination of the two is equivalent to an extractor in the spectrum-time corresponding domain, and the whole model can be called STSubNet.

[0136] In summary, by inputting the frequency domain information corresponding to the to-be-processed speech in the speech stream into the first feature extraction network in a time sequence manner, the first feature information corresponding to each frequency band of the to-be-processed speech is obtained, and the first feature information corresponding to each frequency band is input into the second feature extraction network in a frequency sequence manner, the second feature information corresponding to each frequency band is obtained, the deep features of the frequency domain information can be extracted based on the time axis, and the feature information of each frequency band can be extracted based on the frequency axis, the spectrum-time domain feature extraction is realized, the bidirectional LSTM can refer to the information of the previous and subsequent frequency bands, and therefore the interdependent information of each frequency band of the to-be-processed speech containing reverberation can be better extracted, the extracted features are input into the subnetwork, more accurate results can be obtained, and the accuracy of the model is further improved.

[0137] In other optional implementation manners, the model can be adjusted according to actual needs. For example, the first feature extraction network can be omitted, and the frequency domain information can be directly input into the second feature extraction network and then into the subnetwork for processing. Alternatively, the second feature extraction network can be omitted, and the frequency domain information is directly input into the subnetwork for processing after passing through the first feature extraction network. The number of modules in each network can also be adjusted according to actual needs, for example, the number of one-way LSTM networks and bidirectional LSTM networks can be adjusted, or the one-way LSTM network and the bidirectional LSTM network can be replaced.

[0138] In one or more embodiments of the application, optionally, the loss function used in the model training process is determined by the L2 norm of the difference between the speech signal obtained based on the deep learning model and the actual clean speech signal and the L2 norm of the actual clean speech signal.

[0139] Optionally, the model can be trained using a training sample set, where the model can include the preceding subnetwork and the feature extractor, and these networks can be trained simultaneously. The training sample set can include a plurality of training samples, each sample including an actual clean speech signal and a speech signal added with noise and reverberation, where the actual clean speech signal can be a speech signal without noise and reverberation, and the corresponding speech signal added with noise and reverberation can be obtained by adding noise and reverberation to the actual clean speech signal.

[0140] Optionally, a loss function can be constructed by the following formula:

[0141]

[0142] wherein || || denotes an L2 norm, s(t) is an actual clean speech signal, 2 may be used to represent an L2 norm, s(t) is an actual clean speech signal, is a clean speech signal obtained by the model.

[0143] In summary, by constructing a loss function based on the L2 norm of the difference between the speech signal obtained by the deep learning model and the actual clean speech signal and the L2 norm of the actual clean speech signal, the clean speech signal output by the model can be made to be as close as possible to the actual clean speech signal, and the model has strong overfitting ability, so that the generalization ability of the model is stronger, and the application requirements of the model in different scenarios are met.

[0144] In one or more embodiments of the present application, optionally, the parameter size of the model can be adjustable. By adjusting the neural network parameters (such as S, T, D1, D2, D3, D4), the network parameters of the model can be made small without losing the effect of noise and reverberation removal.

[0145] Generally, the larger the parameters are, the larger the network size is, and the larger the calculation amount is. In the present embodiment, the real-time performance of the network model can be reflected by a real-time factor (RTF), which can be specifically the calculation time for processing one frame divided by the duration of one frame, and less than 1 can realize real-time operation.

[0146] Table 1 RTF corresponding to different parameter sizes

[0147]

[0148] As shown in Table 1, D1, D2, D3, D4 can be different size values, which can enable each module of the model to output features of different dimensions. STSubNet, STSubNet2, and STSubNet3 represent three different size models, and #Para represents the number of corresponding network parameters, in units of M (million). For example, STSubNet only needs 360,000 parameters, and the corresponding RTF is 0.537, which has good real-time performance. Although the number of parameters of STSubNet2 and STSubNet3 increases, the RTF is still less than 1, which can meet the requirements of real-time communication.

[0149] Optionally, at least one of the following information input by the user can be obtained: the length of the time window for intercepting the to-be-processed voice from the voice stream, the information of the plurality of frequency bands, the output dimension of each module in the feature extractor and the deep learning model; and the model is trained according to the at least one information.

[0150] The time window can be used to determine the length corresponding to a frame of to-be-processed voice, and the plurality of frequency bands can refer to the number of divided frequency bands, for example, whether the frequency bands are divided into 129 or more or fewer frequency bands. Each module in the feature extractor and the deep learning model can specifically include a batch normalization layer, a bidirectional LSTM, a linear rectifier unit, a fully connected layer in a subnetwork, and the like. By modifying the at least one information, the size, the amount of calculation, and the accuracy of the network model can be adjusted.

[0151] In summary, by obtaining the length of the time window, the information of the plurality of frequency bands, the output dimension of each module in the feature extractor and the deep learning model, and the like input by the user, the model can be constructed and trained based on the user's selection, the individual needs of the user can be met, the voice processing in different scenarios can be realized, and strong flexibility is achieved.

[0152] In one or more embodiments of the present application, optionally, a normalization operation can be performed before the frequency domain information is input to the model. Optionally, the average value corresponding to the time period t is calculated by the following formula:

[0153]

[0154] wherein α is a smoothing factor, and optionally, L is the length of the time window; μ(t) is the average value corresponding to the time period t, μ(t-1) is the average value of the previous time period t-1, f is a positive integer between 1 and F, F is the number of frequency bands, and Y(t, f) is the frequency domain information corresponding to the fth frequency band in the time period t.

[0155] After obtaining the average value corresponding to each time period, the frequency domain information corresponding to the voice to be processed in the time period can be normalized according to the average value. For example, the frequency domain information corresponding to the time period can be divided by the average value to obtain normalized frequency domain information. The frequency domain information input to the model can be normalized frequency domain information.

[0156] The average value used in the above normalization operation can be constructed by the average value of the previous time period and the information of the current time period, and can meet the real-time requirement.

[0157] Optionally, the evaluation index of speech denoising can include a wide-band perceptual evaluation of speech quality score (PESQ), a short-time objective intelligibility score (STOI), and a scale-invariant signal-to-distortion ratio (SISDR); and the index of dereverberation can include a PESQ, an STOI, and a normalized speech-to-reverberation modulation energy ratio (SRMR). The processing effects of the model provided in the embodiments of the present application and other models are compared on a public test data set.

[0158] Table 2 Comparison of denoising indexes

[0159]

[0160] Table 3 Comparison of dereverberation indexes

[0161]

[0162] In Table 3, near and far represent two different scenes, in which the voice collection device and the playback device have different distances, which are 0.5 meters and 2 meters, respectively.

[0163] The experimental results show that the performance of STSubnet is better than that of other models on the public test data set, and STSubnet can be designed in a lightweight manner using fewer network parameters (only 360,000), while maintaining competitive performance.

[0164] In one or more embodiments of the present application, optionally, preference information input by a user can be acquired, the preference information being used to represent a desire for de-noising and de-reverberation; and a corresponding model is selected for speech processing according to the preference information. Exemplarily, a user can select: preferential de-noising, or preferential de-reverberation.

[0165] Since noise and reverberation are different principles of interference, when both noise interference and reverberation interference are relatively large, a model can be difficult to achieve particularly good results in both aspects. A plurality of models can be pre-trained, and different models perform differently in different scenarios, some models can better remove noise, and some models can better remove reverberation.

[0166] For example, increasing the LSTM network can have better performance in removing reverberation, and thus the degree of removing reverberation by the model can be adjusted by adjusting the number and position of unidirectional LSTM networks and bidirectional LSTM networks in the model.

[0167] In actual application, a user can select preference information for de-noising and de-reverberation according to his own needs, and a device can select a corresponding model for speech processing according to the preference information. Optionally, different models can be stored in a server, and a terminal device can acquire the models from the server for speech processing, or the server can perform speech processing through the models and deliver the processed speech to a terminal device.

[0168] By pre-training models that can remove reverberation and noise to different degrees and selecting a corresponding model according to preference information input by a user, individualized needs of the user can be met, and user experience can be improved.

[0169] In addition, a corresponding model can also be selected according to an actual scenario. Optionally, a room image can be acquired through a shooting device before speech processing, room size, wall material, and indoor obstacle information are determined according to the room image, and a degree of reverberation is determined according to the room size, the wall material, and the indoor obstacle information; environmental noise when a user does not speak is acquired through a voice acquisition device, and a degree of noise is determined according to the environmental noise; or, scenario information, room size, wall material, and indoor obstacle information input by a user can be acquired to determine a degree of noise and / or a degree of reverberation according to the information.

[0170] Optionally, the room image can be input to an image recognition model to determine room size, wall material, and indoor obstacle information. The room image can be a 3D image, which can improve recognition accuracy. The degree of reverberation can be determined according to the room size, the wall material, and the indoor obstacle information, and the degree of reverberation can distinguish different reverberation effects.

[0171] Optionally, the noise level can be determined according to the environmental noise in a period of time when the user does not speak, which is used to distinguish different noise effects. Whether the user speaks can be realized through voice recognition, voiceprint recognition, etc.

[0172] According to the determined reverberation level and / or noise level, a corresponding model can be selected for speech processing. Specifically, different models can be pre-trained, and different models have different structures, or different models have the same structure but are trained using different training sample sets.

[0173] Exemplarily, the scene information can include a bedroom, a coffee shop or a conference room, and the reverberation level and noise level of different scenes are different. The training samples corresponding to different scenes can be obtained and the models can be trained respectively to obtain the trained models corresponding to different scenes.

[0174] In actual application, before the speech is processed by the model, the noise level and the reverberation level of the current environment can be determined, and the corresponding model can be selected for speech processing, so as to improve the speech processing effect in different environments, meet the needs of various scenes, and have strong adaptability.

[0175] Figure 8 Another flowchart of a speech processing method provided by an embodiment of the present application is provided. The embodiment is based on the technical solutions provided in the foregoing embodiments, and provides a speech processing method applied to an online conference scene. The execution subject of the method can be a first device. As shown in Figure 8 The method comprises the following steps:

[0176] Step 801: acquiring single-channel speech collected by a second device in a conference process.

[0177] Optionally, the first device and the second device are used to transmit speech in real time, and the speech can be single-channel speech.

[0178] Step 802: determining corresponding feature information of the single-channel speech on multiple frequency bands.

[0179] Step 803: based on a deep learning model for processing sequence data, a processing result corresponding to each frequency domain information is obtained for a sequence formed by the feature information on the multiple frequency bands.

[0180] In the embodiment, the single-channel speech can be speech to be processed, and the specific implementation process and principles of steps 802 to 803 can be referred to the foregoing embodiments, which will not be described here.

[0181] Step 804: based on the processing result corresponding to each frequency domain information, a processed speech is obtained and played.

[0182] After obtaining the processed speech, the first device can play the speech.

[0183] In actual applications, the first user and the second user can participate in an online meeting using the first device and the second device respectively. The second user can input speech through the second device, and the first user can play the speech through the first device. The speech collected by the second device can contain noise and reverberation. After the first device receives the speech from the second device, the first device can process the speech collected by the second device according to the method provided in this embodiment, obtain the speech after denoising and dereverberation, and play the speech to the first user.

[0184] Similarly, the second device can also obtain the speech collected by the first device, process the speech according to a similar method, obtain the speech after denoising and dereverberation, and play the speech to the second user.

[0185] In summary, the speech processing method provided in this embodiment can obtain single-channel speech collected by the second device during a meeting, determine feature information corresponding to the single-channel speech in multiple frequency bands, form a sequence based on the feature information in the multiple frequency bands, obtain a processing result corresponding to each frequency domain information based on a deep learning model for processing sequence data, obtain processed speech based on the processing result corresponding to each frequency domain information, and play the processed speech. The method can realize real-time speech denoising and dereverberation processing during an online meeting, has high efficiency and accuracy, provides smooth and clear speech experience for users, and improves user experience.

[0186] In addition to being applicable to online meetings, the method in this embodiment can also be applied to other online speech communication scenarios, including but not limited to online teaching, online court trials, and the like. After a device obtains speech input by a user during online teaching or an online court trial, the device can process the collected speech according to the method in this embodiment to obtain speech after denoising and dereverberation.

[0187] This embodiment of the present application also provides a speech processing method applied to the field of speech recognition, including: determining feature information corresponding to speech to be recognized in multiple frequency bands; forming a sequence based on the feature information in the multiple frequency bands, obtaining a processing result corresponding to each frequency domain information based on a deep learning model for processing sequence data; obtaining processed speech based on the processing result corresponding to each frequency domain information; recognizing the processed speech to obtain corresponding speaker information and / or text information corresponding to the speech.

[0188] The to-be-recognized speech can be taken as the to-be-processed speech, and the to-be-recognized speech can be processed by any of the foregoing embodiments. Since the to-be-recognized speech can contain noise and reverberation, which can affect the recognition effect, the method provided in the embodiment is used to perform noise reduction and reverberation reduction on the to-be-recognized speech, and then perform speech recognition, so that the accuracy of speech recognition can be effectively improved.

[0189] In the embodiments of the present application, the execution subject of the method can be set according to actual needs. For example, the method can be executed by a terminal device, or the method can be executed by a server, or part of the steps are executed by a terminal device and part of the steps are executed by a server.

[0190] Corresponding to the speech processing method, the embodiment of the present application further provides a speech processing apparatus, which can include:

[0191] A first determination module is configured to determine feature information corresponding to a to-be-processed speech in a plurality of frequency bands;

[0192] A first processing module is configured to obtain a processing result corresponding to each feature information based on a deep learning model for processing sequence data for a sequence formed by the feature information in the plurality of frequency bands;

[0193] A second processing module is configured to obtain a processed speech based on the processing result corresponding to each feature information.

[0194] In one or more embodiments of the present application, optionally, the first determination module is further configured to:

[0195] obtain the to-be-processed speech based on the obtained speech stream and a preset time window;

[0196] perform short-time Fourier transform on the to-be-processed speech to obtain frequency domain information corresponding to the to-be-processed speech in the plurality of frequency bands;

[0197] The feature information is the frequency domain information, or the feature information is feature information corresponding to the frequency domain information extracted by a feature extractor.

[0198] In one or more embodiments of the present application, optionally, the first determination module is specifically configured to:

[0199] input the frequency domain information corresponding to the to-be-processed speech in the plurality of frequency bands to a feature extractor to extract feature information corresponding to each frequency band;

[0200] The feature extractor includes a time-domain feature extractor and / or a frequency-domain feature extractor.

[0201] In one or more embodiments of the present application, optionally, the first processing module is specifically configured to:

[0202] inputting frequency domain information corresponding to the to-be-processed voice in the voice stream into the first feature extraction network in a time sequence manner to obtain first feature information corresponding to each frequency band of the to-be-processed voice;

[0203] inputting the first feature information corresponding to each frequency band into the second feature extraction network in a frequency sequence manner to obtain second feature information corresponding to each frequency band; the second feature information is used for inputting into a deep learning model to obtain a processing result;

[0204] The first feature extraction network comprises a two-dimensional convolutional neural network, and the second feature extraction network comprises a bidirectional LSTM network.

[0205] In one or more embodiments of the present application, optionally, the first feature extraction network further comprises a batch normalization layer.

[0206] The input end of the batch normalization layer is connected with the output end of the two-dimensional convolutional neural network, and the output end of the batch normalization layer is connected with the input end of the bidirectional LSTM network.

[0207] The second feature extraction network further comprises a full connection layer and a linear rectifier unit.

[0208] The input end of the full connection layer is connected with the output end of the bidirectional LSTM network, and the output end of the full connection layer is connected with the linear rectifier unit.

[0209] In one or more embodiments of the present application, optionally, the deep learning model comprises a unidirectional LSTM network and a full connection layer; the input end of the unidirectional LSTM network is connected with the output end of the second feature extraction network, and the input end of the full connection layer is connected with the output end of the unidirectional LSTM network; the full connection layer is used for obtaining complex ideal ratio masking information corresponding to each frequency band.

[0210] Correspondingly, the second processing module is specifically configured to:

[0211] According to the frequency domain signals of the to-be-processed voice on the plurality of frequency bands and the corresponding complex ideal ratio masking information, information after denoising and dereverberation of the to-be-processed voice in the frequency domain is calculated;

[0212] The information after denoising and dereverberation in the frequency domain is subjected to inverse short-time Fourier transform to obtain voice after denoising and dereverberation.

[0213] In one or more embodiments of the present application, optionally, the first determining module is further configured to:

[0214] acquire at least one of the following information input by a user: length of a time window for intercepting speech to be processed from a speech stream, information of a plurality of frequency bands, and output dimension of each module in a feature extractor and a deep learning model;

[0215] Train the model according to the at least one information.

[0216] In one or more embodiments of the present application, optionally, a loss function used in the model training process is determined by an L2 norm of a difference between a speech signal obtained based on the deep learning model and an actual clean speech signal and an L2 norm of the actual clean speech signal.

[0217] The speech processing apparatus provided by the embodiments of the present application can be used to execute the above Figures 1 to 7 The technical solutions in the embodiments shown in the above and the implementation principles and technical effects are similar, and thus will not be repeated here.

[0218] The embodiments of the present application further provide a speech processing apparatus, which can be applied to a first device, and the apparatus can include:

[0219] An acquisition module, configured to acquire single-channel speech collected by a second device in a conference process;

[0220] A second determination module, configured to determine feature information corresponding to the single-channel speech on a plurality of frequency bands;

[0221] A third processing module, configured to obtain a processing result corresponding to each frequency domain information based on a deep learning model for processing sequence data, for a sequence formed by the feature information on the plurality of frequency bands;

[0222] A fourth processing module, configured to obtain processed speech based on the processing result corresponding to each frequency domain information and play the processed speech.

[0223] The information processing apparatus provided by the embodiments of the present application can be used to execute the above Figure 8 The technical solutions in the embodiments shown in the above and the implementation principles and technical effects are similar, and thus will not be repeated here.

[0224] Figure 9 A structural schematic diagram of an electronic device provided by the embodiments of the present application is shown in FIG. 1. Figure 9 The electronic device of the embodiments of the present application can include:

[0225] at least one processor 901; and

[0226] a memory 902 in communication connection with the at least one processor;

[0227] The memory 902 stores instructions executable by the at least one processor 901, and the instructions are executed by the at least one processor 901 to enable the electronic device to perform the method of any of the preceding embodiments.

[0228] Optionally, the memory 902 can be independent or integrated with the processor 901.

[0229] The implementation principle and technical effects of the electronic device provided in the embodiments can be referred to the foregoing embodiments, and will not be described here.

[0230] The embodiments of the present application further provide a computer readable storage medium, and the computer readable storage medium stores computer execution instructions. When the processor executes the computer execution instructions, the method of any of the foregoing embodiments is implemented.

[0231] The embodiments of the present application further provide a computer program product, which comprises a computer program. When the computer program is executed by the processor, the method of any of the foregoing embodiments is implemented.

[0232] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, another division manner can be used. For example, a plurality of modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0233] The integrated modules implemented in the form of software function modules can be stored in a computer readable storage medium. The software function modules are stored in a storage medium, and include a plurality of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in the embodiments of the present application.

[0234] It should be appreciated that referenced processors above can be a central processing unit (CPU), can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor. The memory can include a high-speed RAM memory, and can also include a non-volatile storage NVM, such as at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0235] The storage medium described above can be realized by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0236] An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a part of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a host device.

[0237] It should be noted that in this paper, the term "include", "contain" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0238] The above-mentioned sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0239] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, can also be through hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art contribution can be embodied in the form of software products, the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), including a number of instructions to make a terminal device (may be a mobile phone, computer, server, air conditioner, or network equipment, etc.) executes the method described in various embodiments of the present application.

[0240] The above is only the preferred embodiment of the present application, not therefore limit the patent scope of the present application, all use the contents of the present application specification and drawings of equivalent structure or equivalent process transformation, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A voice processing method, characterized by, The method comprises: determining feature information corresponding to the to-be-processed speech on a plurality of frequency bands; for a frequency band sequence formed according to the feature information on the plurality of frequency bands, inputting the feature information on the plurality of frequency bands in the frequency band sequence into a same deep learning model for processing sequence data in sequence to obtain a processing result corresponding to each feature information; wherein the deep learning model comprises a unidirectional LSTM network, and the unidirectional LSTM network is used to guide information processing of a subsequent frequency band according to information of a previous frequency band; the deep learning model shares network parameters among all frequency bands; based on the processing result corresponding to each feature information, obtaining processed speech.

2. The method of claim 1, wherein, The method further comprises: obtaining the to-be-processed speech according to the obtained speech stream and a preset time window; performing short-time Fourier transform on the to-be-processed speech to obtain frequency domain information corresponding to the to-be-processed speech on a plurality of frequency bands; wherein the feature information is the frequency domain information, or the feature information is feature information corresponding to the frequency domain information extracted by a feature extractor.

3. The method of claim 1, wherein, determining feature information corresponding to the to-be-processed speech on a plurality of frequency bands comprises: inputting frequency domain information corresponding to the to-be-processed speech on a plurality of frequency bands into a feature extractor to extract feature information corresponding to each frequency band; wherein the feature extractor comprises a feature extractor in the time domain and / or a feature extractor in the frequency domain.

4. The method of claim 3, wherein, inputting frequency domain information corresponding to the to-be-processed speech on a plurality of frequency bands into a feature extractor to extract feature information corresponding to each frequency band comprises: inputting frequency domain information corresponding to the to-be-processed speech in the speech stream into a first feature extraction network in a time sequence to obtain first feature information corresponding to each frequency band of the to-be-processed speech; inputting the first feature information corresponding to each frequency band into a second feature extraction network in a frequency sequence to obtain second feature information corresponding to each frequency band; the second feature information is used to input into a deep learning model to obtain a processing result; wherein the first feature extraction network comprises a two-dimensional convolutional neural network; and the second feature extraction network comprises a bidirectional LSTM network.

5. The method of claim 4, wherein, The first feature extraction network further comprises a batch normalization layer. An input end of the batch normalization layer is connected with an output end of the two-dimensional convolutional neural network, and an output end of the batch normalization layer is connected with an input end of the bidirectional LSTM network. The second feature extraction network further comprises a fully connected layer and a linear rectifier unit. An input end of the fully connected layer is connected with an output end of the bidirectional LSTM network, and an output end of the fully connected layer is connected with the linear rectifier unit.

6. The method of claim 4, wherein, The deep learning model comprises a unidirectional LSTM network and a fully connected layer; an input end of the unidirectional LSTM network is connected with an output end of the second feature extraction network, and an input end of the fully connected layer is connected with an output end of the unidirectional LSTM network; and the fully connected layer is used to obtain complex ideal ratio masking information corresponding to each frequency band. Correspondingly, based on the processing result corresponding to each feature information, obtaining processed speech comprises: According to the frequency domain signal of the to-be-processed voice on multiple frequency bands and the corresponding complex ideal ratio masking information, information after denoising and dereverberation of the to-be-processed voice in the frequency domain is calculated; The information after denoising and dereverberation in the frequency domain is subjected to inverse short-time Fourier transform to obtain the voice after denoising and dereverberation.

7. The method according to any one of claims 3-6, characterized in that, Further comprising: Obtaining at least one of the following information input by a user: the length of the time window for intercepting the to-be-processed voice from the voice stream, the information of the multiple frequency bands, the output dimension of each module in the feature extractor and the deep learning model; Model training is performed according to the at least one information.

8. The method according to any one of claims 3-6, characterized in that, The loss function used in the model training process is determined by the L2 norm of the difference between the voice signal obtained based on the deep learning model and the actual clean voice signal and the L2 norm of the actual clean voice signal.

9. A voice processing method, characterized by, Applied to a first device, the method comprises: Obtaining a single-channel voice collected by a second device during a conference; Determining the corresponding feature information of the single-channel voice on multiple frequency bands; For the frequency band sequence formed by the feature information on the multiple frequency bands, the feature information on the multiple frequency bands in the frequency band sequence is sequentially input to the same deep learning model for processing sequence data, and the feature information on the multiple frequency bands is sequentially processed according to the sequence order to obtain the processing result corresponding to each frequency domain information; wherein the deep learning model comprises a unidirectional LSTM network, the unidirectional LSTM network is used to guide the information processing of the subsequent frequency band according to the information of the previous frequency band; the deep learning model shares network parameters between all frequency bands; Based on the processing result corresponding to each frequency domain information, a processed voice is obtained and played.

10. An electronic device, comprising: Comprise: At least one processor; And The memory is connected in communication with the at least one processor; Wherein, the memory has instructions executable by the at least one processor, the instructions are executed by the at least one processor to make the electronic device execute the method of any one of claims 1-9.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, when the processor executes the computer execution instructions, the method of any one of claims 1-9 is realized.

12. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the method of any one of claims 1-9.

Images