Voice processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a voice processing method and device, electronic equipment and a storage medium. The method comprises the following steps: extracting voice features of a damaged voice signal to obtain damaged voice features; inputting the damaged voice features into a repaired voice generation network of a voice damage repair model to obtain generated repaired voice features; dividing the repaired voice features according to a plurality of preset sub-bands to obtain sub-repaired voice features corresponding to the sub-bands; inputting the sub-repaired speech features corresponding to the sub-bands into a multi-band speech enhancement network of a speech damage repair model, and performing speech enhancement processing on the corresponding sub-repaired speech features through speech enhancement sub-networks corresponding to the sub-bands in the multi-band speech enhancement network to obtain enhanced speech features of the sub-bands; and generating a deep repair voice signal corresponding to the damaged voice signal based on the enhanced voice feature of each sub-band. According to the invention, the voice quality and intelligibility can be remarkably improved in a scene in which multiple distortions exist at the same time.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a speech processing method, device, electronic device and storage medium. Background Art

[0002] With the development of Internet technology, people are increasingly relying on real-time voice communication systems, such as remote collaboration systems and smart terminals. However, during the use of these real-time voice communication systems, the voice signals collected by the microphone may be affected by various complex acoustic conditions and suffer from multiple distortions, such as noise, reverberation, equipment acquisition distortion, packet loss, coding distortion, and pre-processing damage, which seriously affect the voice quality and intelligibility.

[0003] The speech enhancement method in the related art can only remove the environmental noise and reverberation in the speech, but cannot restore the damaged speech signal in the scenario where multiple distortions exist at the same time, resulting in low speech quality and intelligibility. Summary of the invention

[0004] The present disclosure provides a speech processing method, device, electronic device and storage medium to at least solve the problem in the related art that damaged speech signals in multiple distortion scenarios cannot be restored, resulting in low speech quality and intelligibility. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a speech processing method, comprising:

[0006] Acquire a damaged speech signal, extract speech features of the damaged speech signal, and obtain damaged speech features;

[0007] The damaged speech feature is input into a repair speech generation network of a speech damage repair model to obtain a generated repair speech feature; the speech damage repair model is obtained by machine learning training based on a sample damaged speech signal and a sample speech signal corresponding to the sample damaged speech signal, and the sample damaged speech signal is obtained by performing multiple distortion processing on the sample speech signal;

[0008] Dividing the restoration speech feature according to a plurality of preset sub-frequency bands to obtain sub-restoration speech features corresponding to each of the sub-frequency bands;

[0009] Inputting the sub-repaired speech features corresponding to each of the sub-bands into the multi-band speech enhancement network of the speech damage repair model, performing speech enhancement processing on the corresponding sub-repaired speech features through the speech enhancement sub-network corresponding to each of the sub-bands in the multi-band speech enhancement network, to obtain enhanced speech features of each of the sub-bands;

[0010] Based on the enhanced speech features of each of the sub-frequency bands, a deep restoration speech signal corresponding to the damaged speech signal is generated.

[0011] In some exemplary embodiments, extracting the speech feature of the impaired speech signal to obtain the impaired speech feature includes:

[0012] A complex spectrum feature is obtained by performing short-time Fourier transform on the damaged speech signal, and the complex spectrum feature is used as the damaged speech feature.

[0013] In some exemplary embodiments, generating a deep restoration speech signal corresponding to the damaged speech signal based on the enhanced speech features of each of the sub-frequency bands includes:

[0014] splicing the enhanced speech features of each of the sub-bands to obtain a fused enhanced speech feature;

[0015] The fused enhanced speech features are subjected to an inverse short-time Fourier transform to obtain a deep repaired speech signal corresponding to the damaged speech signal.

[0016] In some exemplary embodiments, the inputting the damaged speech feature into a repair speech generation network of a speech damage repair model to obtain the generated repair speech feature comprises:

[0017] Inputting the damaged speech features into the encoder of the repaired speech generation network for encoding processing to obtain an encoding result;

[0018] Performing feature extraction on the encoding result through the temporal convolutional neural network layer of the repaired speech generation network to obtain an intermediate temporal feature vector;

[0019] Extracting features from the intermediate time series feature vector through the long short-term memory network layer of the repaired speech generation network to obtain a target time series feature vector;

[0020] The target time series feature vector is input into the decoder of the repair speech generation network to reconstruct the speech feature to obtain the generated repair speech feature.

[0021] In some exemplary embodiments, the method further comprises:

[0022] Acquire a sample damaged speech signal and a sample speech signal corresponding to the sample damaged speech signal; the sample damaged speech signal is obtained by performing multiple distortion processing on the sample speech signal;

[0023] Extracting speech features of the sample damaged speech signal and the sample speech signal respectively, to obtain a sample damaged speech feature and a corresponding sample speech feature;

[0024] Based on the sample damaged speech features and the corresponding sample speech features, a speech repair generative adversarial network is trained, wherein the speech repair generative adversarial network includes the repair speech generation network, and the repair speech generation network is used to generate sample repair speech features based on the sample damaged speech features;

[0025] Dividing the sample restoration speech features generated based on the restoration speech generation network according to the multiple sub-bands to obtain sub-sample restoration speech features corresponding to each of the sub-bands;

[0026] Inputting the sub-sample repaired speech features corresponding to each of the sub-bands into the multi-band speech enhancement network to be trained, performing speech enhancement processing on the corresponding sub-sample repaired speech features through the speech enhancement sub-network corresponding to each of the sub-bands in the multi-band speech enhancement network to be trained, and obtaining sample enhanced speech features of each sub-band;

[0027] Based on the difference between the sample enhanced speech features of each sub-band and the sub-sample restored speech features of the corresponding sub-band, the multi-band speech enhancement network to be trained is trained to obtain the trained multi-band speech enhancement network.

[0028] In some exemplary embodiments, the training of a speech restoration generative adversarial network based on the sample damaged speech feature and the corresponding sample speech feature includes:

[0029] Inputting the sample damaged speech features into a repair speech generation network to be trained to obtain a generated repair speech feature corresponding to the sample damaged speech features;

[0030] Fixing the network parameters of the repair speech generation network to be trained, taking maximizing the difference between the generated repair speech feature corresponding to the sample damaged speech feature and the sample speech feature corresponding to the sample damaged speech feature as the training goal, training the speech repair discrimination network to be trained until convergence, and obtaining a trained speech repair discrimination network;

[0031] The network parameters of the speech restoration discrimination network are fixed, and the difference between the generated restoration speech feature corresponding to the sample damaged speech feature and the sample damaged speech feature is minimized as the training target, and the restoration speech generation network to be trained is trained until convergence, so as to obtain the trained restoration speech generation network;

[0032] Among them, the speech restoration generative adversarial network includes the restoration speech generation network and the speech restoration discrimination network.

[0033] In some exemplary embodiments, the training of the multi-band speech enhancement network to be trained based on the difference between the sample enhanced speech features of each sub-band and the sub-sample repaired speech features of the corresponding sub-band to obtain the trained multi-band speech enhancement network includes:

[0034] For each sub-band, determining a loss value corresponding to the sub-band based on a difference between a sample enhanced speech feature of the sub-band and a sub-sample restored speech feature of the sub-band;

[0035] Performing weighted summation on the loss values ​​corresponding to the sub-frequency bands to obtain a total loss value;

[0036] The network parameters of each of the speech enhancement sub-networks are reversely adjusted based on the total loss value, and the iterative training is continued based on the adjusted network parameters until a preset training end condition is met, thereby obtaining the trained multi-band speech enhancement network.

[0037] In some exemplary embodiments, performing multiple distortion processing on the sample speech signal includes:

[0038] Randomly selecting a plurality of target distortion types from a plurality of preset distortion types; the plurality of preset distortion types represent distortion types existing in a real-time voice communication link;

[0039] Based on the speech impairment simulation mode corresponding to each target distortion type among the multiple target distortion types, a plurality of distortion impairments are mixed on the sample speech signal to obtain a sample impairment speech signal corresponding to the sample speech signal.

[0040] According to a second aspect of an embodiment of the present disclosure, there is provided a speech processing device, including:

[0041] The damaged speech feature determination unit is configured to obtain a damaged speech signal, extract speech features of the damaged speech signal, and obtain damaged speech features;

[0042] A repair speech feature generation unit is configured to execute inputting the damaged speech feature into a repair speech generation network of a speech damage repair model to obtain a generated repair speech feature; the speech damage repair model is obtained by machine learning training based on a sample damaged speech signal and a sample speech signal corresponding to the sample damaged speech signal, and the sample damaged speech signal is obtained by performing multiple distortion processing on the sample speech signal;

[0043] The restoration speech feature division unit is configured to divide the restoration speech feature according to a plurality of preset sub-frequency bands to obtain sub-restoration speech features corresponding to each of the sub-frequency bands;

[0044] The sub-band enhancement unit is configured to input the sub-repaired speech features corresponding to each of the sub-bands into the multi-band speech enhancement network of the speech damage repair model, and perform speech enhancement processing on the corresponding sub-repaired speech features through the speech enhancement sub-network corresponding to each of the sub-bands in the multi-band speech enhancement network to obtain enhanced speech features of each of the sub-bands;

[0045] The restoration speech signal generating unit is configured to generate a deep restoration speech signal corresponding to the damaged speech signal by performing the enhanced speech feature based on each of the sub-frequency bands.

[0046] In some exemplary embodiments, the damaged speech feature determination unit, when extracting the speech features of the damaged speech signal to obtain the damaged speech features, is specifically configured to perform short-time Fourier transform on the damaged speech signal to obtain complex spectral features; the complex spectral features are used as the damaged speech features.

[0047] In some exemplary embodiments, the repair speech signal generating unit includes:

[0048] A sub-band speech feature concatenation unit is configured to concatenate the enhanced speech features of each of the sub-bands to obtain a fused enhanced speech feature;

[0049] The inverse short-time Fourier transform unit is configured to perform an inverse short-time Fourier transform on the fused enhanced speech feature to obtain a deep repaired speech signal corresponding to the damaged speech signal.

[0050] In some exemplary embodiments, the repair speech feature generation unit includes:

[0051] An encoding unit is configured to execute encoding processing by inputting the damaged speech feature into an encoder of the repair speech generation network to obtain an encoding result;

[0052] A temporal convolution unit is configured to perform feature extraction on the encoding result through the temporal convolutional neural network layer of the repaired speech generation network to obtain an intermediate temporal feature vector;

[0053] A long short-term memory network processing unit is configured to perform feature extraction on the intermediate time series feature vector through the long short-term memory network layer of the repair speech generation network to obtain a target time series feature vector;

[0054] A decoding unit is configured to perform speech feature reconstruction by inputting the target time series feature vector into a decoder of the repair speech generation network to obtain the generated repair speech feature vector.

[0055] In some exemplary embodiments, the apparatus further comprises a model training unit, wherein the model training unit comprises:

[0056] A sample acquisition unit is configured to acquire a sample damaged speech signal and a sample speech signal corresponding to the sample damaged speech signal; the sample damaged speech signal is obtained by performing multiple distortion processing on the sample speech signal;

[0057] A speech feature extraction unit is configured to extract speech features of the sample damaged speech signal and the sample speech signal respectively, to obtain a sample damaged speech feature and a corresponding sample speech feature;

[0058] A generative adversarial network training unit is configured to perform training of a speech repair generative adversarial network based on the sample damaged speech features and the corresponding sample speech features, wherein the speech repair generative adversarial network includes the repair speech generation network, and the repair speech generation network is used to generate sample repair speech features based on the sample damaged speech features;

[0059] A sub-band division unit is configured to divide the sample restoration speech features generated based on the restoration speech generation network according to the plurality of sub-bands to obtain sub-sample restoration speech features corresponding to each of the sub-bands;

[0060] The sub-band enhancement unit is configured to input the sub-sample repaired speech features corresponding to each of the sub-bands into the multi-band speech enhancement network to be trained, and perform speech enhancement processing on the corresponding sub-sample repaired speech features through the speech enhancement sub-network corresponding to each of the sub-bands in the multi-band speech enhancement network to be trained, so as to obtain sample enhanced speech features of each sub-band;

[0061] The multi-band speech enhancement network training unit is configured to perform training based on the difference between the sample enhanced speech features of each sub-band and the sub-sample repaired speech features of the corresponding sub-band, and obtain the trained multi-band speech enhancement network.

[0062] In some exemplary embodiments, the generative adversarial network training unit includes:

[0063] A generating unit is configured to input the sample damaged speech feature into a repair speech generating network to be trained, and obtain a generated repair speech feature corresponding to the sample damaged speech feature;

[0064] The discriminant network training unit is configured to execute the fixing of the network parameters of the repair speech generation network to be trained, to maximize the difference between the generated repair speech feature corresponding to the sample damaged speech feature and the sample speech feature corresponding to the sample damaged speech feature as the training target, to train the speech repair discriminant network to be trained until convergence, and to obtain a trained speech repair discriminant network;

[0065] A generation network training unit is configured to execute the fixing of the network parameters of the speech restoration discrimination network, to minimize the difference between the generated restoration speech feature corresponding to the sample damaged speech feature and the sample damaged speech feature as a training target, to train the restoration speech generation network to be trained until convergence, and to obtain the trained restoration speech generation network;

[0066] Among them, the speech restoration generative adversarial network includes the restoration speech generation network and the speech restoration discrimination network.

[0067] In some exemplary embodiments, the multi-band speech enhancement network training unit includes:

[0068] A sub-band loss determination unit is configured to determine, for each sub-band, a loss value corresponding to the sub-band based on a difference between a sample enhanced speech feature of the sub-band and a sub-sample restored speech feature of the sub-band;

[0069] a total loss determination unit, configured to perform weighted summation of the loss values ​​corresponding to the sub-frequency bands to obtain a total loss value;

[0070] The parameter adjustment unit is configured to perform reverse adjustment of the network parameters of each of the speech enhancement sub-networks based on the total loss value, and continue iterative training based on the adjusted network parameters until a preset training end condition is met, thereby obtaining the trained multi-band speech enhancement network.

[0071] In some exemplary embodiments, the apparatus further comprises:

[0072] A target distortion type determination unit is configured to randomly select a plurality of target distortion types from a plurality of preset distortion types; the plurality of preset distortion types represent distortion types existing in a real-time voice communication link;

[0073] The sample damaged speech signal simulation unit is configured to execute a speech damage simulation method corresponding to each target distortion type among the multiple target distortion types, mix multiple distortion damages on the sample speech signal, and obtain a sample damaged speech signal corresponding to the sample speech signal.

[0074] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0075] processor;

[0076] a memory for storing instructions executable by the processor;

[0077] The processor is configured to execute the instructions to implement the speech processing method of the first aspect.

[0078] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the speech processing method of the first aspect described above.

[0079] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the speech processing method of the first aspect described above is implemented.

[0080] The disclosed embodiment inputs the damaged speech features of the damaged speech signal into the repair speech generation network of the speech damage repair model to obtain the generated repair speech features, and divides the repair speech features according to a plurality of preset sub-bands to obtain sub-repair speech features corresponding to each sub-band, and then inputs the sub-repair speech features corresponding to each sub-band into the multi-band speech enhancement network of the speech damage repair model, performs speech enhancement processing on the corresponding sub-repair speech features through the speech enhancement network corresponding to each sub-band in the multi-band speech enhancement network to obtain the enhanced speech features of each sub-band, and generates a deep repair speech signal corresponding to the damaged speech signal based on the enhanced speech features of each sub-band, so that the damaged speech that is simultaneously interfered by multiple distortions can be restored in a two-stage generative repair method, and the deep repair of multi-distortion damaged speech can be achieved, which can significantly improve the speech quality and intelligibility in the scenario where multiple distortions exist at the same time.

[0081] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0083] Figure 1 is a schematic diagram of an application environment of a speech processing method according to an exemplary embodiment;

[0084] Figure 2 is a schematic diagram showing a method of invoking a speech damage repair model to implement deep speech repair processing of a damaged speech signal interfered by various distortions according to an exemplary embodiment;

[0085] Figure 3 is a schematic diagram of a process of training a speech damage repair model according to an exemplary embodiment;

[0086] Figure 4a is a schematic diagram of the structure of a repair speech generation network according to an exemplary embodiment;

[0087] Figure 4b is a schematic structural diagram of another repair speech generation network according to an exemplary embodiment;

[0088] Figure 5 is a flowchart of a speech processing method according to an exemplary embodiment;

[0089] Figure 6 is a structural block diagram of a speech processing device according to an exemplary embodiment;

[0090] Figure 7 The present invention is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0091] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0092] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0093] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0094] See also Figure 1 , which is a schematic diagram of an application environment according to an exemplary embodiment, the application environment may include a terminal 110 and a server 120, and the terminal 110 and the server 120 may be connected via a wired network or a wireless network.

[0095] The terminal 110 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The terminal 110 may be installed with client software such as an application (Application, referred to as App) that provides a voice processing function. The application may be an independent application or a subroutine in the application. The user of the terminal 110 may log in to the application using pre-registered user information, which may include an account number and a password. Exemplarily, the application may be a dedicated voice processing application or an application with a voice processing function, such as a social application, a live broadcast application, and a video application.

[0096] The server 120 may be a server that provides background services for the application in the terminal 110. The background services may be speech processing services, and the speech processing may be a deep speech repair service. The server 120 may store a speech damage repair model, such as Figure 2 As shown, the server 120 can call the speech damage repair model to implement deep speech repair processing of damaged speech signals interfered by various distortions, so as to obtain a deeply repaired speech signal, i.e., a clean speech signal, with greatly improved intelligibility, clarity and richness.

[0097] Among them, the speech damage repair model may include a repair speech generation network and a multi-band speech enhancement network connected in sequence, wherein the repair speech generation network can be used to perform repair speech generation processing to generate repair speech features, and the generated repair speech features are further divided into sub-repair speech features corresponding to each sub-band according to a plurality of preset different sub-bands and then used as the input of the multi-band speech enhancement network; the multi-band speech enhancement network includes a plurality of parallel speech enhancement sub-networks, and the plurality of parallel speech enhancement sub-networks respectively correspond to one sub-band among the plurality of sub-bands mentioned above, so that each speech enhancement sub-network is used to predict the clean speech features of the sub-band based on the sub-repair speech features of the corresponding sub-band, and the predicted clean speech features of each sub-band can be used to generate the final deep repair speech signal, which can also be called a clean speech signal, after being re-spliced.

[0098] In a specific application scenario, the terminal 110 may send a damaged speech signal to the server 120, and the server 120 may call the above-mentioned speech damage repair model to perform deep repair processing on the damaged speech signal to obtain a deeply repaired speech signal. In another specific application scenario, the terminal 110 may download the speech damage repair model from the server 120 in advance and store it locally, and then during speech processing, the locally stored speech damage repair model may be called to process the damaged speech signal to obtain a deeply repaired speech signal.

[0099] In the disclosed embodiments, server 120 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, an edge computing node, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0100] Since the speech processing method of the embodiment of the present disclosure is implemented based on a speech damage repair model, the training process of the speech damage repair model is first introduced below.

[0101] The training process of the speech damage repair model of the embodiment of the present disclosure may include two training stages, wherein the first training stage trains a generative adversarial network, and through the mutual game between the generator and the discriminator in the generative adversarial network, the final generator can generate speech features that are closer to the real clean speech signal. The generator at the end of the first stage of training serves as the repair speech generation network of the speech damage repair model, and uses the generative adversarial network to perform speech damage repair, including denoising and dereverberation, AI-EQ, loudness equalization, audio super-resolution (BWE), packet loss concealment (PLC) and other functions. In order to solve transient noise, artifact problems (Artificats) and spectral detail damage that are difficult to solve using a single generative adversarial network, the embodiment of the present disclosure uses the repair speech generation network trained in the first training stage in the second training stage to train the multi-band speech enhancement network in the speech damage repair model for noise suppression and spectral detail repair. For details, please refer to Figure 3 , which is a schematic diagram of a process of training a speech damage repair model provided by an embodiment of the present disclosure, and may include the following steps:

[0102] In step S301, a sample impaired speech signal and a sample speech signal corresponding to the sample impaired speech signal are obtained; the sample impaired speech signal is obtained by performing multiple distortion processing on the sample speech signal.

[0103] Specifically, in order to train the model, a training data set needs to be obtained first. The training data set includes multiple sample damaged speech signals and sample speech signals corresponding to each sample damaged speech signal. In the process of training the speech damage repair model based on the training data set, for any sample damaged speech signal and its corresponding sample speech signal, Figure 2 Steps in the illustrated embodiment.

[0104] The sample speech signal is a clean speech signal, which can be understood as a non-destructive speech signal. By performing multiple distortion processing on the sample speech signal, the sample speech signal becomes a damaged speech signal interfered by multiple distortions, namely, a sample damaged speech signal. Then, the speech damage repair model trained based on the sample damaged speech signal and the corresponding sample speech signal can realize speech damage repair in multiple distortion scenarios.

[0105] Based on this, in an exemplary embodiment, before step S301, the embodiment of the present disclosure may also include performing multiple distortion processing on the sample voice signal, and the multiple distortion processing on the sample voice signal may include: randomly selecting multiple target distortion types from multiple preset distortion types; based on the voice damage simulation method corresponding to each target distortion type in the multiple target distortion types, mixing multiple distortion damages on the sample voice signal to obtain a sample damaged voice signal corresponding to the sample voice signal; wherein the multiple preset distortion types represent the distortion types existing in the real-time voice communication link.

[0106] Specifically, the various preset distortion types may include dozens of distortion impairments such as environmental noise, reverberation, spectrum coloration, popping clipping, far-field low volume, packet loss, DC bias, coding distortion, 3A pre-processing impairment, etc. By randomly selecting multiple target distortion impairments from the dozens of distortion impairments, and mixing multiple distortion impairments on the sample speech signal based on the speech impairment simulation method corresponding to each target distortion impairment, a simulated damaged speech signal interfered by multiple distortions can be obtained.

[0107] For example, for environmental noise, noise can be added at different signal-to-noise ratios; for reverberation, different types of reverberation can be added; different types of low-pass filters can be used to simulate nonlinear distortion such as spectrum leakage, truncation, half-wave rectification, etc.; audio codecs at different bit rates can be used to simulate coding distortion; packet loss rates of different proportions can be used to simulate packet loss; fixed amplitude spectrum and random phase information can be used to simulate phase distortion. For the above distortion types, sample impaired speech signals with mixed different distortion types can be randomly generated in a ratio of at least 1 and at most 3.

[0108] The above-mentioned implementation method randomly selects multiple target distortion types from multiple preset distortion types that characterize the distortion types existing in the real-time voice communication link, and then mixes the corresponding multiple distortion damages on the sample voice signal based on the voice damage simulation method corresponding to each target distortion type to obtain a damaged voice signal interfered by multiple distortions, thereby successfully realizing a set of data simulation links that simulate voice damage in real-time voice communication. The link can include dozens of distortion damages such as environmental noise, reverberation, spectral coloration, popping and clipping, far-field low volume, packet loss, DC bias, coding distortion, 3A pre-processing damage, etc., and realizes a training data set suitable for voice signal damage repair tasks in scenarios where multiple distortions exist at the same time, which helps to improve the training accuracy and training efficiency of the voice damage repair model.

[0109] In step S303, speech features of the sample impaired speech signal and the sample speech signal are extracted respectively to obtain sample impaired speech features and corresponding sample speech features.

[0110] The speech features in the embodiments of the present disclosure may be complex spectrum features, so that the trained speech damage repair model can achieve deep repair of the speech signal based on the complex domain. Usually the complex spectrum may include richer speech features, which will make the deeply repaired speech signal purer.

[0111] In a specific implementation, a short-time Fourier transform (STFT) can be performed on the sample damaged speech signal to obtain the complex spectrum features of the sample damaged speech signal, which can be referred to as the sample damaged complex spectrum features hereinafter, and then the sample damaged complex spectrum features can be used as the sample damaged speech features; similarly, a short-time Fourier transform STFT can be performed on the sample speech signal to obtain the complex spectrum features of the sample speech signal, which can be referred to as the sample complex spectrum features hereinafter, and then the sample complex spectrum features can be used as the sample speech features.

[0112] In step S305, a speech restoration generative adversarial network is trained based on the sample damaged speech features and the corresponding sample speech features.

[0113] The speech restoration generative adversarial network includes a restoration speech generation network, which is used to generate sample restoration speech features based on the sample damaged speech features.

[0114] In some exemplary implementations, the above step S305 may include:

[0115] Inputting the sample damaged speech features into a repair speech generation network to be trained to obtain a generated repair speech feature corresponding to the sample damaged speech features;

[0116] Fixing the network parameters of the repair speech generation network to be trained, taking maximizing the difference between the generated repair speech feature corresponding to the sample damaged speech feature and the sample speech feature corresponding to the sample damaged speech feature as the training goal, training the speech repair discrimination network to be trained until convergence, and obtaining a trained speech repair discrimination network;

[0117] Fixing the network parameters of the trained speech restoration discriminant network, taking minimizing the difference between the generated restoration speech feature corresponding to the sample damaged speech feature and the sample damaged speech feature as the training goal, training the restoration speech generation network to be trained until convergence, and obtaining the trained restoration speech generation network;

[0118] Among them, the speech restoration generative adversarial network includes the speech restoration generation network and the speech restoration discrimination network.

[0119] Specifically, the sample damaged speech features are input into the repair speech generation network to be trained for repair speech generation processing to obtain outputted generated repair speech features, such as generated repair complex spectrum features. When the network parameters of the repair speech generation network to be trained are fixed to train the speech repair discrimination network to be trained, the input of the speech repair discrimination network to be trained includes the generated generated repair speech features and the sample speech features (i.e., clean speech features), and the discrimination result obtained by the speech repair discrimination network to be trained for discrimination processing of the input represents the probability that the input is the sample speech feature (i.e., clean speech feature). In practical applications, the adversarial damage function shown in the following formula (1) can be used to maximize the difference between the generated repair speech features and the sample speech features when training the speech repair discrimination network, and minimize the difference between the generated repair speech features and the sample speech features when training the repair speech generation network:

[0120]

[0121] Among them, y represents the sample damaged speech feature; x represents the sample speech feature; z represents the random training noise data; G represents the repair speech generation network to be trained; D represents the speech repair discrimination network to be trained.

[0122] The above implementation method uses adversarial learning between the repair speech generation network and the speech repair discrimination network, so that the repair speech generation network can accurately learn the nonlinear mapping between damaged speech features and clean speech features, thereby improving the speech repair capability of the repair speech generation network.

[0123] In some exemplary embodiments, in order to enable the repair speech generation network to more accurately learn the nonlinear mapping between the damaged speech features and the clean speech features, such as Figure 4aAs shown, the repair speech generation network to be trained may include an encoder, a temporal convolutional neural network (such as a compressed temporal convolutional network, S-TCM) layer, and a plurality of stacked bidirectional long short-term memory networks (such as a residual time-frequency domain long short-term memory network, Residual The invention relates to a method for generating a repaired speech feature in a multi-layered speech structure, wherein the damaged speech feature of the sample is input into the repaired speech generation network to be trained for repaired speech generation processing. When the outputted generated repaired speech feature is obtained, the damaged speech feature of the sample can be input into its encoder for encoding processing to obtain a sample encoding result. Then, the sample encoding result is subjected to feature extraction through its temporal convolutional neural network layer to obtain a sample intermediate temporal feature vector. Then, the sample intermediate temporal feature vector is subjected to feature extraction through its multiple stacked bidirectional long short-term memory network layers to obtain a sample target temporal feature vector. Finally, the sample target temporal feature vector is input into its decoder for speech feature reconstruction to obtain the generated sample repaired speech feature. Thus, the advantages of the temporal convolutional network and LSTM are combined, and a longer-range temporal dependency relationship can be modeled, which is conducive to more accurately learning the nonlinear mapping relationship from damaged speech features to clean speech features to obtain a repaired speech generation network with better speech repair capability.

[0124] For example, in order to make the repair speech generation network have better feature extraction ability and speech reconstruction ability, so as to further improve its speech repair ability, Figure 4b As shown, the encoder may include a dense convolutional network layer and a 2D convolutional layer connected in sequence, wherein the dense convolutional network layer in the encoder is used to receive the sample damaged speech features input to the encoder, and the 2D convolutional layer is used to use the output of the dense convolutional network layer as input for further convolution processing to obtain the encoding result output by the encoder. In the dense convolutional network structure, the output of each layer is imported into all subsequent layers. The dense convolutional network structure uses a connection structure, which can reduce network parameters and avoid problems such as some layers being selectively discarded or information blocking that may occur in the residual neural network. In the dense convolutional network structure, each layer obtains additional input from all previous layers and passes its own feature map to all subsequent layers. Using a cascade method, each layer receives the "collective knowledge" from the previous layers, thereby improving the feature extraction ability of the encoder, so that the speech repair generation network has a larger receptive field and can better perform feature extraction. Similarly, the decoder and the encoder are symmetrically arranged, and the decoder may also include a 2D convolutional layer and a dense convolutional network layer connected in sequence, wherein the 2D convolutional layer in the decoder is used to perform convolution processing on the output of the bidirectional long short-term memory network layer as input, and the dense convolutional network layer in the decoder is used to perform dense convolution processing on the output of the 2D convolutional layer as input to obtain the sample repaired speech features output by the decoder.

[0125] In step S307, the sample restoration speech features generated based on the restoration speech generation network are divided according to a plurality of preset sub-frequency bands to obtain sub-sample restoration speech features corresponding to each sub-frequency band.

[0126] In step S309, the sub-sample repaired speech features corresponding to each sub-band are input into the multi-band speech enhancement network to be trained, and speech enhancement processing is performed on the corresponding sub-sample repaired speech features through the speech enhancement sub-network corresponding to each sub-band in the multi-band speech enhancement network to be trained to obtain the sample enhanced speech features of each sub-band.

[0127] In the disclosed embodiment, after the trained restoration speech generation network is obtained based on the generative adversarial network training, the first stage of model training is completed, and the second stage of model training can be entered at this time. In this second stage, the multi-band speech enhancement network will be trained based on the restoration speech generation network obtained by the first stage training.

[0128] The multi-band speech enhancement network of the disclosed embodiment includes a plurality of parallel speech enhancement sub-networks, each speech enhancement sub-network corresponds to a preset frequency band, and is used to optimize the characteristics of the frequency band, and each speech enhancement sub-network corresponds to a different frequency band, and the frequency band corresponding to each speech enhancement sub-network is referred to as a sub-frequency band hereinafter. Exemplarily, the multi-band speech enhancement network may include speech enhancement sub-networks corresponding to three preset different sub-frequency bands, and the three preset sub-frequency bands may be a low frequency band, a mid-frequency band, and a high frequency band, respectively, wherein the coverage range of the low frequency band, the mid-frequency band, and the high frequency band may be set based on actual experience. Exemplarily, each speech enhancement sub-network is composed of a plurality of stacked convolutional layers and fully connected layers.

[0129] Specifically, after the repair speech generation network is obtained based on the generative adversarial network training of the aforementioned step S305, the sample damaged speech features can be re-input into the trained repair speech generation network for repair speech generation processing to obtain sample repair speech features, such as generated sample complex spectrum features. Then, based on the multiple sub-bands corresponding to the multi-band speech enhancement network, the sample repair speech features at this time are divided, each sub-band corresponds to a different sound feature, so as to obtain the sub-sample repair speech features of each sub-band.

[0130] Next, the sub-sample repaired speech features of each sub-band are input into the speech enhancement sub-network corresponding to the sub-band, and the speech enhancement sub-network corresponding to the sub-band performs speech enhancement processing on the sub-sample repaired speech features of the sub-band to obtain sample enhanced speech features of each sub-band. The speech enhancement processing of each speech enhancement sub-network may be to predict clean speech features, such as predicting clean complex spectrum features, and use the predicted clean speech features as sample enhanced speech features of the corresponding sub-band.

[0131] In step S311, the multi-band speech enhancement network to be trained is trained based on the difference between the sample enhanced speech features of each sub-band and the sub-sample repaired speech features of the corresponding sub-band to obtain the trained multi-band speech enhancement network.

[0132] Specifically, a preset loss function can be used to determine a loss value based on the difference between the sample enhanced speech features of each sub-band and the sub-sample repaired speech features of the corresponding sub-band, and the network parameters of each speech enhancement sub-network in the multi-band speech enhancement network to be trained are adjusted based on the loss value, and iterative training is continued based on the adjusted network parameters until the preset training end condition is met to obtain a trained multi-band speech enhancement network. The preset training end condition can be that the loss value reaches a preset loss threshold, or the number of iterations reaches a preset iteration number threshold, or the difference between the loss values ​​of two adjacent iterations reaches a preset difference threshold. The preset loss function can be a mean square error loss function.

[0133] In some exemplary implementations, the above step S311 may include the following steps when implemented:

[0134] For each sub-band, determining a loss value corresponding to the sub-band based on a difference between a sample enhanced speech feature of the sub-band and a sub-sample restored speech feature of the sub-band;

[0135] Performing weighted summation on the loss values ​​corresponding to the sub-frequency bands to obtain a total loss value;

[0136] The network parameters of each of the speech enhancement sub-networks are reversely adjusted based on the total loss value, and the iterative training is continued based on the adjusted network parameters until a preset training end condition is met, thereby obtaining the trained multi-band speech enhancement network.

[0137] In a specific implementation, the total loss value Loss can be expressed as the following formula (2):

[0138]

[0139] Where n represents the number of sub-bands; i represents the i-th sub-band; X i-comples represents the sub-sample repaired speech feature of sub-band i, Xi ′ -comples represents the sample enhanced speech feature of subband i; ||X i-comples -X i ′ -comples || represents the loss value corresponding to sub-band i; ω i represents the weighted summation weight corresponding to sub-band i, which can be set based on actual experience, and

[0140] In the above implementation, the loss value corresponding to each sub-band is determined by the difference between the sample enhanced speech features of each sub-band and the sub-sample repaired speech features of the sub-band, and then the network parameters of each speech enhancement sub-network are reversely adjusted simultaneously based on the weighted sum of the loss values ​​corresponding to each sub-band, and iterative training is continued based on the adjusted network parameters, so as to perform more fine-grained enhancement modeling on each sub-band for the speech features repaired by the repaired speech generation network to obtain a multi-band enhanced speech network capable of noise suppression and spectral detail repair.

[0141] It should be noted that after training the multi-band enhanced speech network, the trained speech damage repair model in the embodiment of the present disclosure can be obtained by connecting the trained repair speech generation network and the trained multi-band speech enhancement network.

[0142] The disclosed embodiment obtains a repair speech generation network in a speech damage repair model through training via a generative adversarial network framework, and on this basis performs more fine-grained enhancement modeling on each sub-band of the speech features repaired by the repair speech generation network to obtain a multi-band speech enhancement network in the speech damage repair model, so that the trained speech damage repair model can perform deep speech repair processing on damaged speech signals in multiple distortion scenarios, such as denoising and dereverberation, AI-EQ, loudness equalization, audio super-resolution (BWE), packet loss concealment (PLC), noise suppression (NS) and spectral detail repair, which is beneficial to improving the quality and intelligibility of speech signals.

[0143] See also Figure 5 , which is a flow chart of performing speech processing based on the speech damage repair model trained in the above-mentioned embodiment of the present disclosure to achieve deep speech repair, such as Figure 5 As shown, the following steps are included:

[0144] In step S501, an impaired speech signal is acquired, and speech features of the impaired speech signal are extracted to obtain impaired speech features.

[0145] The damaged speech signal may be a speech signal interfered by one distortion or multiple distortions at the same time, for example, a speech signal interfered by multiple distortions including environmental noise, reverberation, spectrum coloration, popping and clipping, low volume in the far field, packet loss, DC bias, coding distortion, and 3A pre-processing damage.

[0146] Exemplarily, the speech feature may be a complex spectrum feature, and the speech feature of the damaged speech signal may be extracted by: obtaining a complex spectrum feature by performing a short-time Fourier transform on the damaged speech signal, and using the complex spectrum feature as the damaged speech feature. Since the complex spectrum may include richer speech features, deep repair processing of the lost speech signal based on the complex spectrum feature of the damaged speech signal may make the deeply repaired speech signal purer, which is beneficial to improving the quality and intelligibility of the repaired speech signal.

[0147] In step S503, the damaged speech features are input into a repair speech generation network of a speech damage repair model to obtain generated repair speech features.

[0148] The speech damage repair model is obtained by machine learning training based on a sample damaged speech signal and a sample speech signal corresponding to the sample damaged speech signal, and the sample damaged speech signal is obtained by performing multiple distortion processing on the sample speech signal.

[0149] In some exemplary implementations, step S503 may include:

[0150] Inputting the damaged speech features into the encoder of the repaired speech generation network for encoding processing to obtain an encoding result;

[0151] Performing feature extraction on the encoding result through the temporal convolutional neural network layer of the repaired speech generation network to obtain an intermediate temporal feature vector;

[0152] Extracting features from the intermediate time series feature vector through the long short-term memory network layer of the repaired speech generation network to obtain a target time series feature vector;

[0153] The target time series feature vector is input into the decoder of the repair speech generation network to reconstruct the speech feature to obtain the generated repair speech feature.

[0154] In a specific implementation, the temporal convolutional neural network layer can be a compressed temporal convolutional network (Squeezed Temporal Convolutional Module, S-TCM), and the long short-term memory network layer can be a residual time-frequency domain long short-term memory network (Residual TF-LSTM).

[0155] In practical applications, in order to further improve the degree of restoration of the generated repaired speech features, the encoding process of the above encoder may include: inputting the damaged speech features into the dense convolutional network layer of the encoder for dense convolutional processing to obtain a dense convolutional processing result, and then convolution processing the dense convolutional processing result based on the 2D convolutional layer in the encoder, and using the convolutional processing result as the encoding result. Similarly, the decoding process of the above decoder may include: inputting the target time series feature vector into the 2D convolutional layer of the decoder for decoding to obtain an intermediate decoding result, inputting the intermediate decoding result into the dense convolutional network layer of the decoder for decoding, and outputting the decoding result as the generated repaired speech features, thereby combining the feature extraction capabilities of the convolutional neural network and the dense convolutional network in the encoder and the decoder, so that the model has a larger receptive field and can better perform feature extraction.

[0156] It should be noted that when the damaged speech feature is represented by a complex spectral feature, the repaired speech feature generated by the repaired speech generation network is the complex spectral feature of the repaired speech signal.

[0157] In the above implementation, by combining the advantages of the temporal convolutional network and the long short-term memory network LSTM to generate the repaired speech features, it is possible to obtain repaired speech features that are closer to the clean speech signal, which is beneficial to improving the degree of recovery of the final repaired speech signal and improving the quality and intelligibility of the repaired speech signal.

[0158] In step S505, the restoration speech feature is divided according to a plurality of preset sub-frequency bands to obtain sub-restoration speech features corresponding to each sub-frequency band.

[0159] Specifically, the restoration speech features may be divided based on a plurality of sub-bands corresponding to the multi-band speech enhancement network, and then the sub-restoration speech features corresponding to each sub-band may be obtained.

[0160] In step S507, the sub-repaired speech features corresponding to each sub-band are input into the multi-band speech enhancement network of the speech damage repair model, and the corresponding sub-repaired speech features are subjected to speech enhancement processing by the speech enhancement sub-network corresponding to each sub-band in the multi-band speech enhancement network to obtain the enhanced speech features of each sub-band.

[0161] Specifically, each speech enhancement sub-network can predict the clean speech feature of the sub-repaired speech feature of the corresponding sub-band to obtain the predicted clean speech feature, and the predicted clean speech feature is the enhanced speech feature of the sub-band. Exemplarily, when the repaired speech feature is represented by a complex spectral feature, the enhanced speech feature of each sub-band is also represented by a complex spectral feature, that is, the complex spectral feature of the speech signal after deep repair.

[0162] In step S509, based on the enhanced speech features of each of the sub-frequency bands, a deep restoration speech signal corresponding to the damaged speech signal is generated.

[0163] Exemplarily, when generating a deep repaired speech signal corresponding to the damaged speech signal based on the enhanced speech features of each of the sub-frequency bands, the enhanced speech features of each of the sub-frequency bands can be spliced ​​to obtain a fused enhanced speech feature; then the fused enhanced speech feature is inversely short-time Fourier transformed to obtain a deep repaired speech signal corresponding to the damaged speech signal.

[0164] For example, if the multiple sub-bands are a low-frequency band, a mid-frequency band and a high-frequency band, the enhanced speech features of the low-frequency band, the enhanced speech features of the mid-frequency band and the enhanced speech features of the high-frequency band can be spliced ​​according to the frequency band, so as to obtain the fused enhanced speech features corresponding to the full-frequency band, and then the fused enhanced speech features are processed by inverse short-time Fourier transform (ISTFT), and the time domain signal after ISTFT processing is called a deep repaired speech signal.

[0165] The above implementation method obtains a fused enhanced speech feature by splicing the enhanced speech features of each sub-band, and then uses the inverse short-time Fourier transform to restore the fused enhanced speech feature into a time domain signal, thereby obtaining a deep repaired speech signal corresponding to the damaged speech signal.

[0166] The speech processing method of the embodiment of the present disclosure uses the repair speech generation network in the speech damage repair model to perform the first stage of generative repair on the damaged speech signal, which includes functions such as denoising and dereverberation, AI-EQ, loudness equalization, audio super-resolution (BWE), and packet loss concealment (PLC). It can restore clean speech features from highly damaged audio. In order to solve transient noise, artifact problems (Artificats) and spectral detail damage that are difficult to solve with a single generative network, the multi-band speech enhancement network in the speech damage repair model is further combined to perform fine-grained enhancement on multiple sub-bands of the repair speech features generated in the first stage to achieve noise suppression and spectral detail repair, so as to restore damaged speech that is simultaneously interfered by multiple distortions, and realize deep repair of multi-distortion damaged speech. In scenarios where multiple distortions exist at the same time, the speech quality and intelligibility can be significantly improved.

[0167] Figure 6 is a structural block diagram of a speech processing device according to an exemplary embodiment. Figure 6 , the speech processing device 600 comprises:

[0168] The impaired speech feature determination unit 610 is configured to obtain an impaired speech signal, extract speech features of the impaired speech signal, and obtain an impaired speech feature;

[0169] The repair speech feature generating unit 620 is configured to execute inputting the damaged speech feature into the repair speech generating network of the speech damage repair model to obtain the generated repair speech feature; the speech damage repair model is obtained by machine learning training based on the sample damaged speech signal and the sample speech signal corresponding to the sample damaged speech signal, and the sample damaged speech signal is obtained by performing multiple distortion processing on the sample speech signal;

[0170] The restoration speech feature division unit 630 is configured to divide the restoration speech feature according to a plurality of preset sub-frequency bands to obtain sub-restoration speech features corresponding to each sub-frequency band;

[0171] The sub-band enhancement unit 640 is configured to input the sub-repaired speech features of each sub-band into the multi-band speech enhancement network, and perform speech enhancement processing on the corresponding sub-repaired speech features through the speech enhancement sub-network corresponding to each sub-band in the multi-band speech enhancement network to obtain enhanced speech features of each sub-band;

[0172] The restoration speech signal generating unit 650 is configured to generate a deep restoration speech signal corresponding to the damaged speech signal by performing the enhanced speech feature based on each of the sub-frequency bands.

[0173] In some exemplary embodiments, the damaged speech feature determination unit 610, when extracting the speech features of the damaged speech signal to obtain the damaged speech features, is specifically configured to perform short-time Fourier transform on the damaged speech signal to obtain complex spectral features, and use the complex spectral features as the damaged speech features.

[0174] In some exemplary embodiments, the repaired speech signal generating unit 650 includes:

[0175] A sub-band speech feature concatenation unit is configured to concatenate the enhanced speech features of each of the sub-bands to obtain a fused enhanced speech feature;

[0176] The inverse short-time Fourier transform unit is configured to perform an inverse short-time Fourier transform on the fused enhanced speech feature to obtain a deep repaired speech signal corresponding to the damaged speech signal.

[0177] In some exemplary embodiments, the repair speech feature generation unit 620 includes:

[0178] An encoding unit is configured to execute encoding processing by inputting the damaged speech feature into an encoder of the repair speech generation network to obtain an encoding result;

[0179] A temporal convolution unit is configured to perform feature extraction on the encoding result through the temporal convolutional neural network layer of the repaired speech generation network to obtain an intermediate temporal feature vector;

[0180] A long short-term memory network processing unit is configured to perform feature extraction on the intermediate time series feature vector through the long short-term memory network layer of the repair speech generation network to obtain a target time series feature vector;

[0181] A decoding unit is configured to perform speech feature reconstruction by inputting the target time series feature vector into a decoder of the repair speech generation network to obtain the generated repair speech feature vector.

[0182] In some exemplary embodiments, the apparatus 600 further includes a model training unit, wherein the model training unit includes:

[0183] A sample acquisition unit is configured to acquire a sample damaged speech signal and a sample speech signal corresponding to the sample damaged speech signal; the sample damaged speech signal is obtained by performing multiple distortion processing on the sample speech signal;

[0184] A speech feature extraction unit is configured to extract speech features of the sample damaged speech signal and the sample speech signal respectively, to obtain a sample damaged speech feature and a corresponding sample speech feature;

[0185] A generative adversarial network training unit is configured to perform training of a speech repair generative adversarial network based on the sample damaged speech features and the corresponding sample speech features, wherein the speech repair generative adversarial network includes the repair speech generation network, and the repair speech generation network is used to generate sample repair speech features based on the sample damaged speech features;

[0186] A sub-band division unit is configured to divide the sample restoration speech features generated based on the restoration speech generation network according to the plurality of sub-bands to obtain sub-sample restoration speech features corresponding to each sub-band;

[0187] The sub-band enhancement unit is configured to input the sub-sample repaired speech features corresponding to each sub-band into the multi-band speech enhancement network to be trained, and perform speech enhancement processing on the corresponding sub-sample repaired speech features through the speech enhancement sub-network corresponding to each sub-band in the multi-band speech enhancement network to be trained, so as to obtain the sample enhanced speech features of each sub-band;

[0188] The multi-band speech enhancement network training unit is configured to perform training based on the difference between the sample enhanced speech features of each sub-band and the sub-sample repaired speech features of the corresponding sub-band to obtain the trained multi-band speech enhancement network.

[0189] In some exemplary embodiments, the generative adversarial network training unit includes:

[0190] A generating unit is configured to input the sample damaged speech feature into a repair speech generating network to be trained, and obtain a generated repair speech feature corresponding to the sample damaged speech feature;

[0191] The discriminant network training unit is configured to execute the fixing of the network parameters of the repair speech generation network to be trained, to maximize the difference between the generated repair speech feature corresponding to the sample damaged speech feature and the sample speech feature corresponding to the sample damaged speech feature as the training target, to train the speech repair discriminant network to be trained until convergence, and to obtain a trained speech repair discriminant network;

[0192] A generation network training unit is configured to execute the fixing of the network parameters of the speech restoration discrimination network, to minimize the difference between the generated restoration speech feature corresponding to the sample damaged speech feature and the sample damaged speech feature as a training target, to train the restoration speech generation network to be trained until convergence, and to obtain the trained restoration speech generation network;

[0193] Among them, the speech restoration generative adversarial network includes the restoration speech generation network and the speech restoration discrimination network.

[0194] In some exemplary embodiments, the multi-band speech enhancement network training unit includes:

[0195] A sub-band loss determination unit is configured to determine, for each sub-band, a loss value corresponding to the sub-band based on a difference between a sample enhanced speech feature of the sub-band and a sub-sample restored speech feature of the sub-band;

[0196] a total loss determination unit, configured to perform weighted summation of the loss values ​​corresponding to the sub-frequency bands to obtain a total loss value;

[0197] The parameter adjustment unit is configured to perform reverse adjustment of the network parameters of each of the speech enhancement sub-networks based on the total loss value, and continue iterative training based on the adjusted network parameters until a preset training end condition is met, thereby obtaining the trained multi-band speech enhancement network.

[0198] In some exemplary embodiments, the apparatus 600 further includes:

[0199] A target distortion type determination unit is configured to randomly select a plurality of target distortion types from a plurality of preset distortion types; the plurality of preset distortion types represent distortion types existing in a real-time voice communication link;

[0200] The sample damaged speech signal simulation unit is configured to execute a speech damage simulation method corresponding to each target distortion type among the multiple target distortion types, mix multiple distortion damages on the sample speech signal, and obtain a sample damaged speech signal corresponding to the sample speech signal.

[0201] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0202] In an exemplary embodiment, an electronic device is also provided, including a processor; a memory for storing instructions executable by the processor; wherein, when the processor is configured to execute the instructions stored in the memory, the speech processing method provided in any of the above embodiments is implemented.

[0203] The electronic device may be a terminal, a server or a similar computing device. For example, the electronic device is a server. Figure 7 is a block diagram of an electronic device running a voice processing method according to an exemplary embodiment. Figure 7 As shown, the server 700 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 710 (the processor 710 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 730 for storing data, and one or more storage media 720 (such as one or more mass storage devices) for storing application programs 723 or data 722. Among them, the memory 730 and the storage medium 720 can be short-term storage or permanent storage. The program stored in the storage medium 720 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 710 can be configured to communicate with the storage medium 720 and execute a series of instruction operations in the storage medium 720 on the server 700. The server 700 may also include one or more power supplies 760, one or more wired or wireless network interfaces 750, one or more input and output interfaces 740, and / or one or more operating systems 721, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0204] The input / output interface 740 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the server 700. In one example, the input / output interface 740 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 740 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0205] It can be understood by those skilled in the art that Figure 7 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 7 More or fewer components as shown, or with Figure 7 Different configurations are shown.

[0206] In an exemplary embodiment, a computer-readable storage medium is also provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the speech processing method provided in any of the above embodiments.

[0207] In an exemplary embodiment, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the speech processing method provided in any of the above embodiments is implemented.

[0208] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0209] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A speech processing method, characterized in that: include: Acquire a damaged speech signal, extract speech features of the damaged speech signal, and obtain damaged speech features; Inputting the damaged speech features into a repair speech generation network of a speech damage repair model to obtain generated repair speech features; The speech damage repair model is obtained by machine learning training based on a sample damaged speech signal and a sample speech signal corresponding to the sample damaged speech signal, and the sample damaged speech signal is obtained by performing multiple distortion processing on the sample speech signal; Dividing the restoration speech feature according to a plurality of preset sub-frequency bands to obtain sub-restoration speech features corresponding to each of the sub-frequency bands; Inputting the sub-repaired speech features corresponding to each of the sub-bands into the multi-band speech enhancement network of the speech damage repair model, performing speech enhancement processing on the corresponding sub-repaired speech features through the speech enhancement sub-network corresponding to each of the sub-bands in the multi-band speech enhancement network, to obtain enhanced speech features of each of the sub-bands; Based on the enhanced speech features of each of the sub-frequency bands, a deep restoration speech signal corresponding to the damaged speech signal is generated.

2. The method according to claim 1, characterized in that: The step of extracting the speech feature of the damaged speech signal to obtain the damaged speech feature comprises: A complex spectrum feature is obtained by performing short-time Fourier transform on the damaged speech signal, and the complex spectrum feature is used as the damaged speech feature.

3. The method according to claim 2, characterized in that The step of generating a deep restoration speech signal corresponding to the damaged speech signal based on the enhanced speech features of each of the sub-frequency bands comprises: splicing the enhanced speech features of each of the sub-bands to obtain a fused enhanced speech feature; The fused enhanced speech features are subjected to an inverse short-time Fourier transform to obtain a deep repaired speech signal corresponding to the damaged speech signal.

4. The method according to claim 1, characterized in that The damaged speech feature is input into the repair speech generation network of the speech damage repair model to obtain the generated repair speech feature, which includes: Inputting the damaged speech features into the encoder of the repaired speech generation network for encoding processing to obtain an encoding result; Performing feature extraction on the encoding result through the temporal convolutional neural network layer of the repaired speech generation network to obtain an intermediate temporal feature vector; Extracting features from the intermediate time series feature vector through the long short-term memory network layer of the repaired speech generation network to obtain a target time series feature vector; The target time series feature vector is input into the decoder of the repair speech generation network to reconstruct the speech feature to obtain the generated repair speech feature.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire a sample damaged speech signal and a sample speech signal corresponding to the sample damaged speech signal; the sample damaged speech signal is obtained by performing multiple distortion processing on the sample speech signal; Extracting speech features of the sample damaged speech signal and the sample speech signal respectively, to obtain a sample damaged speech feature and a corresponding sample speech feature; Based on the sample damaged speech features and the corresponding sample speech features, a speech repair generative adversarial network is trained, wherein the speech repair generative adversarial network includes the repair speech generation network, and the repair speech generation network is used to generate sample repair speech features based on the sample damaged speech features; Dividing the sample restoration speech features generated based on the restoration speech generation network according to the multiple sub-bands to obtain sub-sample restoration speech features corresponding to each of the sub-bands; Inputting the sub-sample repaired speech features corresponding to each of the sub-bands into the multi-band speech enhancement network to be trained, performing speech enhancement processing on the corresponding sub-sample repaired speech features through the speech enhancement sub-network corresponding to each of the sub-bands in the multi-band speech enhancement network to be trained, and obtaining sample enhanced speech features of each sub-band; Based on the difference between the sample enhanced speech features of each sub-band and the sub-sample restored speech features of the corresponding sub-band, the multi-band speech enhancement network to be trained is trained to obtain the trained multi-band speech enhancement network.

6. The method according to claim 5, characterized in that The training of the speech restoration generative adversarial network based on the sample damaged speech features and the corresponding sample speech features comprises: Inputting the sample damaged speech features into a repair speech generation network to be trained to obtain a generated repair speech feature corresponding to the sample damaged speech features; Fixing the network parameters of the repair speech generation network to be trained, taking maximizing the difference between the generated repair speech feature corresponding to the sample damaged speech feature and the sample speech feature corresponding to the sample damaged speech feature as the training goal, training the speech repair discrimination network to be trained until convergence, and obtaining a trained speech repair discrimination network; The network parameters of the speech restoration discrimination network are fixed, and the difference between the generated restoration speech feature corresponding to the sample damaged speech feature and the sample damaged speech feature is minimized as the training target, and the restoration speech generation network to be trained is trained until convergence, so as to obtain the trained restoration speech generation network; Among them, the speech restoration generative adversarial network includes the restoration speech generation network and the speech restoration discrimination network.

7. The method according to claim 5, characterized in that The method of training the multi-band speech enhancement network to be trained based on the difference between the sample enhanced speech features of each sub-band and the sub-sample repaired speech features of the corresponding sub-band to obtain the trained multi-band speech enhancement network comprises: For each sub-band, determining a loss value corresponding to the sub-band based on a difference between a sample enhanced speech feature of the sub-band and a sub-sample restored speech feature of the sub-band; Performing weighted summation on the loss values ​​corresponding to the sub-frequency bands to obtain a total loss value; The network parameters of each of the speech enhancement sub-networks are reversely adjusted based on the total loss value, and the iterative training is continued based on the adjusted network parameters until a preset training end condition is met, thereby obtaining the trained multi-band speech enhancement network.

8. The method according to claim 5, characterized in that The performing multiple distortion processing on the sample speech signal comprises: Randomly selecting a plurality of target distortion types from a plurality of preset distortion types; the plurality of preset distortion types represent distortion types existing in a real-time voice communication link; Based on the speech impairment simulation mode corresponding to each target distortion type among the multiple target distortion types, a plurality of distortion impairments are mixed on the sample speech signal to obtain a sample impairment speech signal corresponding to the sample speech signal.

9. A speech processing device, characterized in that: include: The damaged speech feature determination unit is configured to obtain a damaged speech signal, extract speech features of the damaged speech signal, and obtain damaged speech features; A repair speech feature generating unit is configured to execute inputting the damaged speech feature into a repair speech generating network of a speech damage repair model to obtain a generated repair speech feature; The speech damage repair model is obtained by machine learning training based on a sample damaged speech signal and a sample speech signal corresponding to the sample damaged speech signal, and the sample damaged speech signal is obtained by performing multiple distortion processing on the sample speech signal; The restoration speech feature division unit is configured to divide the restoration speech feature according to a plurality of preset sub-frequency bands to obtain sub-restoration speech features corresponding to each of the sub-frequency bands; The sub-band enhancement unit is configured to input the sub-repaired speech features corresponding to each of the sub-bands into the multi-band speech enhancement network of the speech damage repair model, and perform speech enhancement processing on the corresponding sub-repaired speech features through the speech enhancement sub-network corresponding to each of the sub-bands in the multi-band speech enhancement network to obtain enhanced speech features of each of the sub-bands; The restoration speech signal generating unit is configured to generate a deep restoration speech signal corresponding to the damaged speech signal by performing the enhanced speech feature based on each of the sub-frequency bands.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the speech processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the speech processing method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the speech processing method according to any one of claims 1 to 8 is implemented.

Images