A voice processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a speech processing method, device, electronic equipment and storage medium, the method comprising: extracting speech features of a damaged speech signal to obtain damaged speech features; inputting the damaged speech features into a repair speech generation network of a speech damage repair model to obtain generated repair speech features; dividing the repair speech features according to a plurality of preset sub-frequency bands to obtain sub-repair speech features corresponding to each sub-frequency band; inputting the sub-repair speech features corresponding to each sub-frequency band into a multi-band speech enhancement network of the speech damage repair model, performing speech enhancement processing on the corresponding sub-repair speech features through a speech enhancement sub-network corresponding to each sub-frequency band in the multi-band speech enhancement network to obtain enhanced speech features of each sub-frequency band; and generating a deep repair speech signal corresponding to the damaged speech signal based on the enhanced speech features of each sub-frequency band. The present disclosure can significantly improve speech quality and intelligibility in scenarios where multiple distortions exist simultaneously.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a speech processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the development of Internet technology, people's dependence on real-time voice communication systems, such as remote collaboration systems and intelligent terminals, is increasing. However, in the process of using these real-time voice communication systems, the voice signals collected by the microphone may be affected by various complex acoustic conditions and simultaneously exist in multiple distortion conditions, such as noise, reverberation, device collection distortion, packet loss, coding distortion and pre-processing damage, which seriously affect the voice quality and intelligibility.

[0003] The speech enhancement method in the related art can only remove environmental noise and reverberation in the speech, and cannot restore damaged speech signals in multiple distortion scenarios, resulting in low voice quality and intelligibility. SUMMARY

[0004] The present disclosure provides a speech processing method, device, electronic equipment and storage medium to at least solve the problem that the damaged speech signals in multiple distortion scenarios cannot be restored in the related art, resulting in low voice quality and intelligibility. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of the embodiments of the present disclosure, a speech processing method is provided, comprising:

[0006] obtaining a damaged speech signal, extracting a speech feature of the damaged speech signal to obtain a damaged speech feature;

[0007] inputting the damaged speech feature into a repair speech generation network of a speech damage repair model to obtain a generated repair speech feature; the speech damage repair model is obtained by machine learning training based on a sample damaged speech signal and a sample speech signal corresponding to the sample damaged speech signal, and the sample damaged speech signal is obtained by multiple distortion processing on the sample speech signal;

[0008] dividing the repair speech feature into a plurality of sub-bands according to a plurality of preset sub-bands to obtain a sub-repair speech feature corresponding to each of the sub-bands;

[0009] inputting the sub-repair speech feature corresponding to each of the sub-bands into a multi-band speech enhancement network of the speech damage repair model, performing speech enhancement processing on the corresponding sub-repair speech feature through a speech enhancement sub-network corresponding to each of the sub-bands in the multi-band speech enhancement network to obtain an enhanced speech feature of each of the sub-bands;

[0010] generate the deep repaired speech signal corresponding to the impaired speech signal based on the enhanced speech features of each of the sub-bands.

[0011] In some example embodiments, the extracting the speech features of the impaired speech signal includes:

[0012] obtaining complex spectrum features by performing short-time Fourier transform on the impaired speech signal, and taking the complex spectrum features as the impaired speech features.

[0013] In some example embodiments, the generating the deep repaired speech signal corresponding to the impaired speech signal based on the enhanced speech features of each of the sub-bands includes:

[0014] splicing the enhanced speech features of each of the sub-bands to obtain fused enhanced speech features;

[0015] performing inverse short-time Fourier transform on the fused enhanced speech features to obtain the deep repaired speech signal corresponding to the impaired speech signal.

[0016] In some example embodiments, the inputting the impaired speech features into a repaired speech generation network of a speech impairment repair model to obtain generated repaired speech features includes:

[0017] inputting the impaired speech features into an encoder of the repaired speech generation network for encoding processing to obtain an encoding result;

[0018] extracting features of the encoding result by a time sequence convolutional neural network layer of the repaired speech generation network to obtain an intermediate time sequence feature vector;

[0019] extracting features of the intermediate time sequence feature vector by a long short-term memory network layer of the repaired speech generation network to obtain a target time sequence feature vector;

[0020] inputting the target time sequence feature vector into a decoder of the repaired speech generation network for speech feature reconstruction to obtain the generated repaired speech features.

[0021] In some example embodiments, the method further includes:

[0022] obtaining a sample impaired speech signal and a sample speech signal corresponding to the sample impaired speech signal; the sample impaired speech signal being obtained by performing multiple distortion processing on the sample speech signal;

[0023] extracting speech features of the sample impaired speech signal and the sample speech signal respectively to obtain sample impaired speech features and corresponding sample speech features;

[0024] training a speech repair generation adversarial network based on the sample damaged speech feature and the corresponding sample speech feature, the speech repair generation adversarial network comprising the repair speech generation network, the repair speech generation network being configured to generate a sample repair speech feature based on the sample damaged speech feature;

[0025] dividing the sample repair speech feature generated based on the repair speech generation network into sub-sample repair speech features corresponding to the plurality of sub-bands;

[0026] inputting the sub-sample repair speech features corresponding to the plurality of sub-bands into the multi-band speech enhancement network to be trained, performing speech enhancement processing on the corresponding sub-sample repair speech features by speech enhancement sub-networks corresponding to the plurality of sub-bands in the multi-band speech enhancement network to be trained, to obtain sample enhanced speech features of the plurality of sub-bands;

[0027] training the multi-band speech enhancement network to be trained based on the difference between the sample enhanced speech features of the plurality of sub-bands and the sub-sample repair speech features of the corresponding sub-bands, to obtain the trained multi-band speech enhancement network.

[0028] In some exemplary embodiments, the training of the speech repair generation adversarial network based on the sample damaged speech feature and the corresponding sample speech feature comprises:

[0029] inputting the sample damaged speech feature into the repair speech generation network to be trained to obtain a generated repair speech feature corresponding to the sample damaged speech feature;

[0030] fixing the network parameters of the repair speech generation network to be trained, taking the difference between the generated repair speech feature corresponding to the sample damaged speech feature and the sample speech feature corresponding to the sample damaged speech feature as a training target, training the speech repair discriminant network to be trained until convergence, and obtaining the trained speech repair discriminant network;

[0031] fixing the network parameters of the speech repair discriminant network, taking the difference between the generated repair speech feature corresponding to the sample damaged speech feature and the sample damaged speech feature as a training target, training the repair speech generation network to be trained until convergence, and obtaining the trained repair speech generation network;

[0032] wherein the speech repair generation adversarial network comprises the repair speech generation network and the speech repair discriminant network.

[0033] In some example embodiments, the training of the to-be-trained multi-band speech enhancement network based on the difference between the sample enhanced speech feature of each sub-band and the sub-sample repaired speech feature of the corresponding sub-band comprises:

[0034] For each sub-band, a loss value corresponding to the sub-band is determined based on the difference between the sample enhanced speech feature of the sub-band and the sub-sample repaired speech feature of the sub-band.

[0035] The loss values corresponding to the sub-bands are weighted and summed to obtain a total loss value.

[0036] The network parameters of each speech enhancement sub-network are adjusted based on the total loss value, and the adjusted network parameters are used for continued iterative training until a preset training end condition is met, thereby obtaining the trained multi-band speech enhancement network.

[0037] In some example embodiments, the multiple distortion processing of the sample speech signal comprises:

[0038] Randomly selecting a plurality of target distortion types from a plurality of preset distortion types; the plurality of preset distortion types represent the distortion types existing in the real-time voice communication link;

[0039] Based on the speech impairment simulation mode corresponding to each target distortion type in the plurality of target distortion types, the sample speech signal is subjected to a mixture of multiple distortion impairments to obtain a sample impaired speech signal corresponding to the sample speech signal.

[0040] According to a second aspect of the embodiments of the present disclosure, a speech processing apparatus is provided, comprising:

[0041] An impaired speech feature determination unit configured to perform the following operations: acquiring an impaired speech signal, and extracting speech features of the impaired speech signal to obtain impaired speech features;

[0042] A repaired speech feature generation unit configured to perform the following operations: inputting the impaired speech features into a repaired speech generation network of a speech impairment repair model to obtain generated repaired speech features; the speech impairment repair model is obtained by machine learning training based on a sample impaired speech signal and a sample speech signal corresponding to the sample impaired speech signal, and the sample impaired speech signal is obtained by multiple distortion processing of the sample speech signal;

[0043] A repaired speech feature division unit configured to perform the following operations: dividing the repaired speech features according to a plurality of preset sub-bands to obtain sub-repaired speech features corresponding to each sub-band.

[0044] The sub-band enhancement unit is configured to input the sub-repair speech features corresponding to each of the sub-bands into a multi-band speech enhancement network of the speech damage repair model, perform speech enhancement processing on the corresponding sub-repair speech features through a speech enhancement sub-network corresponding to each of the sub-bands in the multi-band speech enhancement network, and obtain enhanced speech features of each of the sub-bands.

[0045] The repair speech signal generation unit is configured to generate a deep repair speech signal corresponding to the damage speech signal based on the enhanced speech features of each of the sub-bands.

[0046] In some exemplary embodiments, when extracting speech features of the damage speech signal to obtain damage speech features, the damage speech feature determination unit is specifically configured to perform complex spectrum feature extraction by performing short-time Fourier transform on the damage speech signal; and the complex spectrum feature is used as the damage speech feature.

[0047] In some exemplary embodiments, the repair speech signal generation unit includes:

[0048] The sub-band speech feature splicing unit is configured to splice the enhanced speech features of each of the sub-bands to obtain fused enhanced speech features.

[0049] The inverse short-time Fourier transform unit is configured to perform inverse short-time Fourier transform on the fused enhanced speech features to obtain a deep repair speech signal corresponding to the damage speech signal.

[0050] In some exemplary embodiments, the repair speech feature generation unit includes:

[0051] The encoding unit is configured to input the damage speech features into an encoder of the repair speech generation network to perform encoding processing, and obtain an encoding result.

[0052] The time sequence convolution unit is configured to perform feature extraction on the encoding result through a time sequence convolutional neural network layer of the repair speech generation network to obtain an intermediate time sequence feature vector.

[0053] The long short-term memory network processing unit is configured to perform feature extraction on the intermediate time sequence feature vector through a long short-term memory network layer of the repair speech generation network to obtain a target time sequence feature vector.

[0054] The decoding unit is configured to input the target time sequence feature vector into a decoder of the repair speech generation network to perform speech feature reconstruction, and obtain the generated repair speech features.

[0055] In some example embodiments, the apparatus further comprises a model training unit, which comprises:

[0056] a sample obtaining unit configured to perform obtaining a sample impaired speech signal and a sample speech signal corresponding to the sample impaired speech signal, the sample impaired speech signal being obtained by performing multiple distortions on the sample speech signal;

[0057] a speech feature extracting unit configured to perform extracting speech features of the sample impaired speech signal and the sample speech signal respectively, to obtain sample impaired speech features and corresponding sample speech features;

[0058] a generative adversarial network training unit configured to perform training a speech repair generative adversarial network based on the sample impaired speech features and the corresponding sample speech features, the speech repair generative adversarial network comprising the repair speech generating network, the repair speech generating network being configured to generate sample repair speech features based on the sample impaired speech features;

[0059] a sub-band dividing unit configured to perform dividing the sample repair speech features generated based on the repair speech generating network according to the plurality of sub-bands, to obtain sub-sample repair speech features corresponding to each of the sub-bands;

[0060] a sub-band enhancing unit configured to perform inputting the sub-sample repair speech features corresponding to each of the sub-bands into a to-be-trained multi-band speech enhancement network, performing speech enhancement processing on the corresponding sub-sample repair speech features by speech enhancement sub-networks corresponding to each of the sub-bands in the to-be-trained multi-band speech enhancement network, to obtain sample enhanced speech features of each sub-band;

[0061] a multi-band speech enhancement network training unit configured to perform training the to-be-trained multi-band speech enhancement network based on differences between the sample enhanced speech features of each sub-band and the sub-sample repair speech features of the corresponding sub-band, to obtain the trained multi-band speech enhancement network.

[0062] In some example embodiments, the generative adversarial network training unit comprises:

[0063] a generating unit configured to perform inputting the sample impaired speech features into the to-be-trained repair speech generating network, to obtain generated repair speech features corresponding to the sample impaired speech features;

[0064] The discriminant network training unit is configured to fix network parameters of the to-be-trained repair speech generation network, take maximizing difference between the generated repair speech feature corresponding to the sample damaged speech feature and the sample speech feature corresponding to the sample damaged speech feature as a training target, train the to-be-trained speech repair discriminant network until convergence, and obtain a trained speech repair discriminant network.

[0065] The generation network training unit is configured to fix network parameters of the speech repair discriminant network, take minimizing difference between the generated repair speech feature corresponding to the sample damaged speech feature and the sample damaged speech feature as a training target, train the to-be-trained repair speech generation network until convergence, and obtain a trained repair speech generation network.

[0066] The speech repair generative adversarial network includes the repair speech generation network and the speech repair discriminant network.

[0067] In some example embodiments, the multi-band speech enhancement network training unit includes:

[0068] The sub-band loss determination unit is configured to determine, for each sub-band, a loss value corresponding to the sub-band based on difference between the sample enhanced speech feature of the sub-band and the sub-sample repair speech feature of the sub-band.

[0069] The total loss determination unit is configured to perform weighted summation on the loss values corresponding to the sub-bands to obtain a total loss value.

[0070] The parameter adjustment unit is configured to perform reverse adjustment on the network parameters of each speech enhancement sub-network based on the total loss value, and continue iterative training based on the adjusted network parameters until a preset training end condition is met to obtain a trained multi-band speech enhancement network.

[0071] In some example embodiments, the apparatus further includes:

[0072] The target distortion type determination unit is configured to randomly select multiple target distortion types from multiple preset distortion types, where the multiple preset distortion types represent distortion types existing in a real-time voice communication link.

[0073] The sample damaged speech signal simulation unit is configured to perform mixing of multiple distortion damages on the sample speech signal based on a speech damage simulation mode corresponding to each target distortion type in the multiple target distortion types to obtain a sample damaged speech signal corresponding to the sample speech signal.

[0074] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, which includes:

[0075] a processor;

[0076] a memory for storing the processor-executable instructions;

[0077] The processor is configured to execute the instructions to implement the voice processing method of the first aspect.

[0078] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, when the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device can execute the voice processing method of the first aspect.

[0079] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the voice processing method of the first aspect.

[0080] The embodiments of the present disclosure input the damaged speech features of the damaged speech signal into the repair speech generation network of the speech damage repair model to obtain generated repair speech features, and divide the repair speech features according to a plurality of preset sub-bands to obtain sub-repair speech features corresponding to each sub-band, and then input the sub-repair speech features corresponding to each sub-band into the multi-band speech enhancement network of the speech damage repair model, perform speech enhancement processing on the corresponding sub-repair speech features through the speech enhancement network corresponding to each sub-band in the multi-band speech enhancement network to obtain enhanced speech features of each sub-band, and generate a deep repair speech signal corresponding to the damaged speech signal based on the enhanced speech features of each sub-band, so that the damaged speech simultaneously interfered by multiple distortions can be restored in a two-stage generative repair manner, the deep repair of the multi-distortion damaged speech is realized, and the speech quality and intelligibility can be significantly improved in the scene where multiple distortions exist simultaneously.

[0081] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0082] The accompanying drawings incorporated in and forming a part of the specification illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure without imposing undue limitations on the disclosure.

[0083] Figure 1 is an application environment schematic diagram of a voice processing method according to an exemplary embodiment;

[0084] Figure 2 is a schematic diagram of calling a speech damage repair model to implement deep speech repair processing of a damaged speech signal interfered by multiple distortions according to an exemplary embodiment;

[0085] Figure 3 FIG. 1 is a flowchart illustrating a process of training a voice impairment repair model according to an example embodiment;

[0086] Figure 4a FIG. 2 is a structural diagram of a repair speech generation network according to an example embodiment;

[0087] Figure 4b FIG. 3 is a structural diagram of another repair speech generation network according to an example embodiment;

[0088] Figure 5 FIG. 4 is a flowchart illustrating a voice processing method according to an example embodiment;

[0089] Figure 6 FIG. 5 is a structural block diagram of a voice processing apparatus according to an example embodiment;

[0090] Figure 7 FIG. 6 is a structural block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0091] In order to make the ordinary person skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings.

[0092] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. The embodiments described in the following example embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0093] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties.

[0094] Please refer to Figure 1 which shows an application environment diagram according to an example embodiment, which can include a terminal 110 and a server 120, which can be connected through a wired network or a wireless network.

[0095] The terminal 110 can be a smartphone, a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto. The terminal 110 can be installed with a client software, such as an application (APP), that provides a voice processing function. The application can be a standalone application or a subprogram in an application. A user of the terminal 110 can log in to the application by using pre-registered user information, which can include an account and a password. The application can be a dedicated voice processing application or an application with a voice processing function, such as a social application, a live broadcast application, and a video application.

[0096] The server 120 can be a server that provides a background service for the application in the terminal 110. The background service can be a voice processing service, which can be a deep voice repair service. The server 120 can store a voice damage repair model, such as a voice damage repair model 1200 shown in FIG. 1. Figure 2 As shown, the server 120 can call the voice damage repair model to implement deep voice repair processing of a damaged voice signal interfered by multiple distortions, to obtain a deep repair voice signal, i.e., a clean voice signal, with greatly improved intelligibility, clarity, and richness.

[0097] The voice damage repair model can include a repair voice generation network and a multi-band voice enhancement network connected in sequence. The repair voice generation network can be used to perform generation processing of a repair voice to generate repair voice features. The generated repair voice features are further divided into sub-repair voice features corresponding to each sub-band according to a plurality of different sub-bands, and then used as inputs of the multi-band voice enhancement network. The multi-band voice enhancement network includes a plurality of parallel voice enhancement sub-networks, each of which corresponds to one of the plurality of sub-bands. Each voice enhancement sub-network is used to predict clean voice features of the corresponding sub-band based on the sub-repair voice features of the corresponding sub-band, and the predicted clean voice features of each sub-band are reassembled to generate a final deep repair voice signal, which can also be referred to as a clean voice signal.

[0098] In one specific application scenario, the terminal 110 can send a damaged voice signal to the server 120, and the server 120 can call the voice damage repair model to perform deep repair processing on the damaged voice signal to obtain a deep repair voice signal. In another specific application scenario, the terminal 110 can pre-download the voice damage repair model from the server 120 and store it locally, and then call the locally stored voice damage repair model to process the damaged voice signal to obtain a deep repair voice signal during voice processing.

[0099] In the embodiments of the present disclosure, the server 120 can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, an edge computing node, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0100] Since the voice processing method in the embodiments of the present disclosure is implemented based on the voice damage repair model, the training process of the voice damage repair model will be introduced first.

[0101] The training process of the voice damage repair model in the embodiments of the present disclosure can include two training stages. In the first training stage, a generative adversarial network is trained. Through the mutual game between the generator and the discriminator in the generative adversarial network, the generator can finally generate voice features closer to the real clean voice signal. The generator at the end of the first stage training is used as the repair voice generation network of the voice damage repair model. The voice damage repair using the generative adversarial network contains functions such as denoising dereverberation, AI-EQ, loudness equalization, audio super-resolution (BWE), and packet loss concealment (PLC). For the transient noise, artifacts, and spectral detail damage that cannot be solved by using a single generative adversarial network, the multi-band voice enhancement network in the voice damage repair model is trained in the second training stage using the repair voice generation network trained in the first training stage for noise suppression and spectral detail repair. For details, please refer to Figure 3 The flowchart shown in FIG. 3 is a flowchart of a method for training a voice damage repair model according to an embodiment of the present disclosure. The method can include the following steps:

[0102] In step S301, a sample damaged voice signal and a sample voice signal corresponding to the sample damaged voice signal are obtained. The sample damaged voice signal is obtained by performing multiple distortion processing on the sample voice signal.

[0103] Specifically, in order to train the model, a training data set needs to be obtained first. The training data set includes a plurality of sample damaged voice signals and sample voice signals corresponding to each sample damaged voice signal. In the process of training the voice damage repair model based on the training data set, for any sample damaged voice signal and its corresponding sample voice signal, the steps shown in the embodiment of FIG. 3 are performed. Figure 2

[0104] ​The sample voice signal is a clean voice signal, which can be understood as an undamaged voice signal. The sample voice signal is processed by multiple distortions to become a damaged voice signal, i.e., a sample damaged voice signal, and then a voice damage repair model trained based on the sample damaged voice signal and the corresponding sample voice signal can realize voice damage repair in a multiple distortion scenario.

[0105] Based on this, in an exemplary embodiment, before step S301, the embodiments of the present disclosure can further include multiple distortion processing on the sample voice signal, which can include: randomly selecting multiple target distortion types from multiple preset distortion types; based on a voice damage simulation mode corresponding to each target distortion type in the multiple target distortion types, mixing multiple distortion damages on the sample voice signal to obtain a sample damaged voice signal corresponding to the sample voice signal; wherein the multiple preset distortion types represent distortion types existing in a real-time voice communication link.

[0106] Specifically, the multiple preset distortion types can include environmental noise, reverberation, spectral tinting, clipping, far-field low volume, packet loss, DC bias, coding distortion, 3A pre-processing damage, and dozens of distortion damages. By randomly selecting multiple target distortion damages from the dozens of distortion damages, and mixing multiple distortion damages on the sample voice signal based on the voice damage simulation mode corresponding to each target distortion damage, an analog damaged voice signal interfered by multiple distortions can be obtained.

[0107] For example, environmental noise can be added under different signal-to-noise ratios; different types of reverberation can be added for reverberation; different types of low-pass filters can be used to simulate nonlinear distortions such as spectral leakage, clipping, half-wave rectification, etc.; different code rates of audio codecs can be used to simulate coding distortion; different packet loss rates can be used to simulate packet loss; and fixed amplitude spectrum and random phase information can be used to simulate phase distortion. For the above several distortion types, the sample damaged voice signal mixed with different distortion types can be randomly generated in a ratio of at least 1 and at most 3.

[0108] The above embodiment successfully realizes a data simulation link for simulating voice damage in real-time voice communication, which can include tens of distortion damages such as environmental noise, reverberation, spectral tinting, pop clipping, far-field low volume, packet loss, DC bias, coding distortion, 3A pre-processing damage, and the like, realizes a training data set suitable for voice signal damage repair tasks in a multi-distortion scene, and helps to improve the training accuracy and training efficiency of the voice damage repair model.

[0109] In step S303, voice features of the sample damage speech signal and the sample speech signal are extracted respectively to obtain sample damage speech features and corresponding sample speech features.

[0110] The voice features in the embodiments of the present disclosure can be complex spectrum features, so that the trained voice damage repair model can realize deep repair of the voice signal based on the complex domain. Generally, complex spectrum can include more rich voice features, so that the deep repaired voice signal is more pure.

[0111] In specific implementation, the sample damage speech signal can be subjected to short-time Fourier transform (STFT) to obtain complex spectrum features of the sample damage speech signal, which can be referred to as sample damage complex spectrum features hereinafter, and then the sample damage complex spectrum features are taken as sample damage speech features. Similarly, the sample speech signal can be subjected to short-time Fourier transform (STFT) to obtain complex spectrum features of the sample speech signal, which can be referred to as sample complex spectrum features hereinafter, and then the sample complex spectrum features are taken as sample speech features.

[0112] In step S305, a voice repair generative adversarial network is trained based on the sample damage speech features and the corresponding sample speech features.

[0113] The voice repair generative adversarial network includes a repair voice generation network, which is configured to generate sample repair voice features based on the sample damage speech features.

[0114] In some exemplary embodiments, the above step S305 can include the following when implemented:

[0115] The sample damage speech features are input into the repair voice generation network to be trained to obtain generated repair voice features corresponding to the sample damage speech features.

[0116] fixing network parameters of the trained speech repair generation network to maximize a difference between the generated repair speech feature corresponding to the sample damaged speech feature and the sample speech feature corresponding to the sample damaged speech feature as a training target, training the to-be-trained speech repair generation network until convergence, and obtaining a trained speech repair generation network;

[0117] fixing network parameters of the trained speech repair generation network to maximize a difference between the generated repair speech feature corresponding to the sample damaged speech feature and the sample speech feature corresponding to the sample damaged speech feature as a training target, training the to-be-trained speech repair generation network until convergence, and obtaining a trained speech repair generation network;

[0118] The speech repair generation adversarial network includes the speech repair generation network and the speech repair discrimination network.

[0119] Specifically, the sample damaged speech feature is input into the to-be-trained speech repair generation network for repair speech generation processing, and output generated repair speech features, such as generated repair complex spectrum features, are obtained. When the network parameters of the to-be-trained speech repair generation network are fixed to train the to-be-trained speech repair discrimination network, the input of the to-be-trained speech repair discrimination network includes the generated generated repair speech feature and the sample speech feature (i.e., the clean speech feature), and the discrimination result obtained by the to-be-trained speech repair discrimination network through discrimination processing of the input represents the probability that the input is the sample speech feature (i.e., the clean speech feature). In actual application, the following formula (1) shows an adversarial loss function for maximizing the difference between the generated repair speech feature and the sample speech feature when training the speech repair discrimination network, and minimizing the difference between the generated repair speech feature and the sample speech feature when training the speech repair generation network:

[0120]

[0121] wherein y represents the sample damaged speech feature; x represents the sample speech feature; z represents random training noise data; G represents the to-be-trained speech repair generation network; and D represents the to-be-trained speech repair discrimination network.

[0122] The above embodiments enable the repair speech generation network to accurately learn the nonlinear mapping between the damaged speech feature and the clean speech feature through adversarial learning between the repair speech generation network and the speech repair discrimination network, thereby improving the speech repair capability of the repair speech generation network.

[0123] In some exemplary embodiments, in order to enable the repair speech generation network to more accurately learn the nonlinear mapping between the damaged speech feature and the clean speech feature, the following formula (2) can be used as the adversarial loss function: Figure 4aAs shown, the to-be-trained repaired speech generation network can include an encoder, a time sequence convolutional neural network (such as a compressed time sequence convolutional network, S-TCM) layer, a plurality of stacked bidirectional long short-term memory network (such as a residual time-frequency domain long short-term memory network, Residual TF-LSTM) layers and a decoder connected in sequence, and thus when sample damaged speech features are input into the to-be-trained repaired speech generation network for repaired speech generation processing to obtain output generated repaired speech features, the sample damaged speech features can be input into the encoder for encoding processing to obtain sample encoding results, then the sample encoding results are subjected to feature extraction by the time sequence convolutional neural network layer to obtain sample intermediate time sequence feature vectors, then the sample intermediate time sequence feature vectors are subjected to feature extraction by the plurality of stacked bidirectional long short-term memory network layers to obtain sample target time sequence feature vectors, and finally the sample target time sequence feature vectors are input into the decoder for speech feature reconstruction to obtain generated sample repaired speech features, thereby combining the advantages of the time sequence convolutional network and the LSTM, modeling longer time sequence dependency relationships, and facilitating more accurate learning of a nonlinear mapping relationship from damaged speech features to clean speech features to obtain a repaired speech generation network with better speech repair capability.

[0124] For example, in order to enable the repaired speech generation network to have better feature extraction capability and speech reconstruction capability, and further improve its speech repair capability, as shown in Figure 4b As shown, the above-described encoder can include a densely connected convolutional network layer and a 2D convolutional layer connected in sequence, wherein the densely connected convolutional network layer in the encoder is configured to receive sample damaged speech features input into the encoder, and the 2D convolutional layer is configured to further perform convolution processing on the output of the densely connected convolutional network layer as input to obtain an encoding result output by the encoder. In the densely connected convolutional network structure, the output of each layer is introduced into all subsequent layers, and the densely connected convolutional network structure uses a concatenation structure, which can reduce network parameters, avoid problems such as selective discarding of certain layers or information blocking in residual neural networks. In the densely connected convolutional network structure, each layer obtains additional input from all previous layers and transmits its feature map to all subsequent layers, using a cascading manner, so that each layer receives “collective knowledge” from previous layers, thereby improving the feature extraction capability of the encoder and enabling the speech repair generation network to have a larger receptive field and better feature extraction. Similarly, the decoder is symmetrically arranged with the encoder, and the decoder can also include a 2D convolutional layer and a densely connected convolutional network layer connected in sequence, wherein the 2D convolutional layer in the decoder is configured to perform convolution processing on the output of the bidirectional long short-term memory network layer as input, and the densely connected convolutional network layer in the decoder is configured to perform densely connected convolution processing on the output of the 2D convolutional layer as input to obtain sample repaired speech features output by the decoder.

[0125] In step S307, the sample repaired speech features generated based on the repaired speech generation network are divided according to the preset multiple sub-bands, to obtain sub-sample repaired speech features corresponding to each sub-band.

[0126] In step S309, the sub-sample repaired speech features corresponding to each sub-band are input into the multi-band speech enhancement network to be trained, and the corresponding sub-sample repaired speech features are processed by the speech enhancement sub-networks corresponding to each of the sub-bands in the multi-band speech enhancement network to be trained, to obtain sample enhanced speech features of each sub-band.

[0127] In the embodiments of the present disclosure, after the trained repaired speech generation network is trained based on the generative adversarial network, the first-stage model training is completed, and at this time, the second-stage model training can be entered, in which the multi-band speech enhancement network is trained based on the repaired speech generation network trained in the first stage.

[0128] The multi-band speech enhancement network in the embodiments of the present disclosure includes multiple parallel speech enhancement sub-networks, each of which corresponds to a preset frequency band for optimizing the features of the frequency band, and the frequency bands corresponding to the speech enhancement sub-networks are different, which are referred to as sub-bands in the following. For example, the multi-band speech enhancement network can include speech enhancement sub-networks corresponding to three preset different sub-bands, which can be low band, medium band and high band, respectively, wherein the coverage of each of the low band, medium band and high band can be set based on actual experience. For example, each speech enhancement sub-network is composed of multiple stacked convolutional layers and fully connected layers.

[0129] Specifically, after the repaired speech generation network is trained based on the generative adversarial network in the foregoing step S305, the sample damaged speech features can be re-input into the trained repaired speech generation network for repaired speech generation processing to obtain sample repaired speech features, such as generated sample complex spectrum features. Then, the sample repaired speech features at this time are divided based on the multiple sub-bands corresponding to the multi-band speech enhancement network, each sub-band corresponding to different sound features, so as to obtain sub-sample repaired speech features of each sub-band.

[0130] Then, the sub-sample repaired speech features of each sub-band are input into the speech enhancement sub-network corresponding to the sub-band, and the speech enhancement sub-network corresponding to the sub-band is used to perform speech enhancement processing on the sub-sample repaired speech features of the sub-band to obtain sample enhanced speech features of each sub-band. The speech enhancement processing of each speech enhancement sub-network can be prediction of clean speech features, such as prediction of clean complex spectrum features, and the predicted clean speech features are used as the sample enhanced speech features of the corresponding sub-band.

[0131] In step S311, the multi-band speech enhancement network to be trained is trained based on the difference between the sample enhanced speech features of each sub-band and the sub-sample repaired speech features of the corresponding sub-band, and a trained multi-band speech enhancement network is obtained.

[0132] Specifically, a preset loss function can be used to determine a loss value based on the difference between the sample enhanced speech features of each sub-band and the sub-sample repaired speech features of the corresponding sub-band, the network parameters of each speech enhancement sub-network in the multi-band speech enhancement network to be trained are adjusted based on the loss value, and the iterative training is continued based on the adjusted network parameters until a preset training end condition is met to obtain the trained multi-band speech enhancement network. The preset training end condition can be that the loss value reaches a preset loss threshold, or the number of iterations reaches a preset iteration number threshold, or the difference between the loss values of adjacent two iterations reaches a preset difference threshold. The preset loss function can be a mean square error loss function.

[0133] In some exemplary embodiments, the above step S311 can include the following steps when implemented:

[0134] For each sub-band, a loss value corresponding to the sub-band is determined based on the difference between the sample enhanced speech features of the sub-band and the sub-sample repaired speech features of the sub-band.

[0135] The loss values corresponding to each of the sub-bands are weighted and summed to obtain a total loss value.

[0136] The network parameters of each speech enhancement sub-network are adjusted in reverse based on the total loss value, and the iterative training is continued based on the adjusted network parameters until a preset training end condition is met to obtain the trained multi-band speech enhancement network.

[0137] In a specific implementation, the total loss value Loss can be represented by the following formula (2):

[0138]

[0139] wherein n represents the number of sub-bands, i represents the i-th sub-band, X i-comples represents the sub-sample repaired speech features of the sub-band i, and Xi ′ -comples represents the sample enhanced speech feature of the sub-band i; ||X i-comples -X i ′ -comples || represents the loss value corresponding to the sub-band i; ω i represents the weighted sum weight corresponding to the sub-band i, which can be set based on actual experience, and

[0140] In the above implementation, the loss value corresponding to each sub-band is determined based on the difference between the sample enhanced speech feature of each sub-band and the sub-sample repaired speech feature of the sub-band, and then the network parameters of each speech enhancement sub-network are simultaneously adjusted in reverse based on the weighted sum of the loss values corresponding to each sub-band, and the network parameters are continuously iteratively trained based on the adjusted network parameters, so that the repaired speech feature generated by the repaired speech generation network is more finely enhanced in each sub-band to obtain a multi-band enhanced speech network capable of noise suppression and spectral detail repair.

[0141] It should be noted that after the multi-band enhanced speech network is trained in the embodiment of the present disclosure, the trained speech damage repair model in the embodiment of the present disclosure can be obtained by connecting the trained repaired speech generation network and the trained multi-band speech enhancement network.

[0142] The repaired speech generation network in the speech damage repair model is trained by the generative adversarial network framework in the embodiment of the present disclosure, and on this basis, the multi-band speech enhancement network in the speech damage repair model is more finely enhanced in each sub-band to obtain the repaired speech feature, so that the trained speech damage repair model can perform deep speech repair processing on the damaged speech signal in the multi-distortion scene, such as noise reduction, AI-EQ, loudness equalization, audio super-resolution (BWE), packet loss concealment (PLC), noise suppression (NS) and spectral detail repair, which is beneficial to improve the quality and intelligibility of the speech signal.

[0143] Please refer to Figure 5 , which is a flowchart of speech processing to realize deep speech repair based on the aforementioned trained speech damage repair model in the embodiment of the present disclosure, as shown in Figure 5 , which includes the following steps:

[0144] In step S501, the damaged speech signal is obtained, the speech feature of the damaged speech signal is extracted, and the damaged speech feature is obtained.

[0145] The damaged speech signal can be a speech signal interfered by one distortion or multiple distortions, such as environmental noise, reverberation, spectral coloring, pop distortion, far-field low volume, packet loss, DC bias, coding distortion, and 3A pre-processing distortion.

[0146] For example, the speech feature can be a complex spectrum feature, and the speech feature of the damaged speech signal can be obtained by performing short-time Fourier transform on the damaged speech signal to obtain a complex spectrum feature, which is taken as the damaged speech feature. Since the complex spectrum can include more abundant speech features, deep repair processing of the damaged speech signal based on the complex spectrum feature of the damaged speech signal can make the deep-repaired speech signal purer, which is beneficial to improving the quality and intelligibility of the repaired speech signal.

[0147] In step S503, the damaged speech feature is input into a repaired speech generation network of the speech damage repair model to obtain a generated repaired speech feature.

[0148] The speech damage repair model is obtained by machine learning training based on a sample damaged speech signal and a sample speech signal corresponding to the sample damaged speech signal, and the sample damaged speech signal is obtained by performing multiple distortion processing on the sample speech signal.

[0149] In some example embodiments, the step S503 can include the following steps when implemented:

[0150] The damaged speech feature is input into an encoder of the repaired speech generation network for encoding processing to obtain an encoding result.

[0151] The encoding result is extracted by a time sequence convolutional neural network layer of the repaired speech generation network to obtain an intermediate time sequence feature vector.

[0152] The intermediate time sequence feature vector is extracted by a long short-term memory network layer of the repaired speech generation network to obtain a target time sequence feature vector.

[0153] The target time sequence feature vector is input into a decoder of the repaired speech generation network for speech feature reconstruction to obtain the generated repaired speech feature.

[0154] In specific implementations, the time sequence convolutional neural network layer can be a squeezed temporal convolutional module (S-TCM), and the long short-term memory network layer can be a residual TF-LSTM.

[0155] In actual applications, to further improve the recovery degree of the generated repaired speech feature, the encoding process of the above-mentioned encoder can include: inputting the damaged speech feature into the dense convolutional network layer of the encoder for dense convolutional processing to obtain a dense convolutional processing result, and then performing convolutional processing on the dense convolutional processing result based on the 2D convolutional layer in the encoder, and taking the result of the convolutional processing as the encoding result. Similarly, the decoding process of the above-mentioned decoder can include: inputting the target time sequence feature vector into the 2D convolutional layer of the decoder for decoding to obtain an intermediate decoding result, inputting the intermediate decoding result into the dense convolutional network layer of the decoder for decoding, and outputting the decoding result as the generated repaired speech feature, so that the feature extraction capabilities of the convolutional neural network and the dense convolutional network are combined in the encoder and the decoder, the model has a larger receptive field, and the feature extraction can be better.

[0156] It should be noted that when the damaged speech feature is represented by a complex spectrum feature, the repaired speech feature generated by the repaired speech generation network is a complex spectrum feature of the repaired speech signal.

[0157] In the above-mentioned embodiments, the advantages of the time sequence convolutional network and the long short-term memory network (LSTM) are combined to generate the repaired speech feature, which can obtain a repaired speech feature closer to the clean speech signal, thereby facilitating to improve the recovery degree of the final repaired speech signal and improve the quality and intelligibility of the repaired speech signal.

[0158] In step S505, the repaired speech feature is divided according to a plurality of preset sub-bands to obtain sub-repaired speech features corresponding to the sub-bands.

[0159] Specifically, the repaired speech feature can be divided based on a plurality of sub-bands corresponding to the multi-band speech enhancement network, and then sub-repaired speech features corresponding to the sub-bands are obtained.

[0160] In step S507, the sub-repaired speech features corresponding to the sub-bands are input into the multi-band speech enhancement network of the speech damage repair model, and the speech enhancement sub-networks corresponding to the sub-bands in the multi-band speech enhancement network perform speech enhancement processing on the corresponding sub-repaired speech features to obtain enhanced speech features of the sub-bands.

[0161] Specifically, each speech enhancement sub-network can perform clean speech feature prediction on the sub-repaired speech feature of the corresponding sub-band to obtain a predicted clean speech feature, which is the enhanced speech feature of the sub-band. For example, when the repaired speech feature is represented by a complex spectrum feature, the enhanced speech features of the sub-bands are also represented by complex spectrum features, i.e., the complex spectrum features of the deeply repaired speech signal.

[0162] In step S509, a deep repaired speech signal corresponding to the damaged speech signal is generated based on the enhanced speech features of each of the sub-bands.

[0163] For example, when the deep repaired speech signal corresponding to the damaged speech signal is generated based on the enhanced speech features of each of the sub-bands, the enhanced speech features of each of the sub-bands can be spliced to obtain fused enhanced speech features, and then inverse short-time Fourier transform is performed on the fused enhanced speech features to obtain the deep repaired speech signal corresponding to the damaged speech signal.

[0164] For example, the sub-bands are low frequency bands, medium frequency bands, and high frequency bands, and the enhanced speech features of the low frequency bands, the enhanced speech features of the medium frequency bands, and the enhanced speech features of the high frequency bands can be spliced according to the frequency bands to obtain fused enhanced speech features corresponding to the full frequency bands, and then inverse short-time Fourier transform (ISTFT) processing is performed on the fused enhanced speech features, and the time domain signal after the ISTFT processing is referred to as the deep repaired speech signal.

[0165] The above embodiment splices the enhanced speech features of each sub-band to obtain fused enhanced speech features, and then uses inverse short-time Fourier transform to restore the fused enhanced speech features to a time domain signal, thereby obtaining a deep repaired speech signal corresponding to a damaged speech signal.

[0166] The speech processing method of the embodiment of the present disclosure uses the repair speech generation network in the speech damage repair model to perform first-stage generative repair on the damaged speech signal, and contains functions such as denoising dereverberation, AI-EQ, loudness equalization, audio super-resolution (BWE), and packet loss concealment (PLC), which can recover clean speech features from highly damaged audio. For transient noise, artifacts, and spectral detail damage that cannot be solved by a single generative network, the multi-band speech enhancement network in the speech damage repair model is further combined to perform fine-grained enhancement on the repaired speech features generated in the first stage in multiple sub-bands to achieve noise suppression and spectral detail repair, thereby being able to recover damaged speech that is simultaneously interfered by multiple distortions, achieving deep repair of multi-distortion damaged speech, and significantly improving speech quality and intelligibility in scenarios where multiple distortions exist simultaneously.

[0167] Figure 6 FIG. 6 is a structural block diagram of a speech processing device according to an example embodiment. Referring to FIG. 6, Figure 6 The speech processing device 600 includes:

[0168] The impaired speech feature determination unit 610 is configured to perform obtaining an impaired speech signal, extracting speech features of the impaired speech signal to obtain impaired speech features;

[0169] The repaired speech feature generation unit 620 is configured to perform inputting the impaired speech features into a repaired speech generation network of a speech impairment repair model to obtain generated repaired speech features; the speech impairment repair model is obtained by machine learning training based on a sample impaired speech signal and a sample speech signal corresponding to the sample impaired speech signal, and the sample impaired speech signal is obtained by multiple distortion processing on the sample speech signal;

[0170] The repaired speech feature division unit 630 is configured to perform dividing the repaired speech features according to a plurality of preset sub-bands to obtain sub-repaired speech features corresponding to each sub-band;

[0171] The sub-band enhancement unit 640 is configured to perform inputting the sub-repaired speech features of each sub-band into a multi-band speech enhancement network, performing speech enhancement processing on the corresponding sub-repaired speech features by a speech enhancement sub-network corresponding to each sub-band in the multi-band speech enhancement network to obtain enhanced speech features of each sub-band;

[0172] The repaired speech signal generation unit 650 is configured to perform generating a deep repaired speech signal corresponding to the impaired speech signal based on the enhanced speech features of each sub-band.

[0173] In some exemplary embodiments, the impaired speech feature determination unit 610, when extracting speech features of the impaired speech signal to obtain impaired speech features, is specifically configured to perform obtaining complex spectrum features by performing short-time Fourier transform on the impaired speech signal, and taking the complex spectrum features as the impaired speech features.

[0174] In some exemplary embodiments, the repaired speech signal generation unit 650 includes:

[0175] The sub-band speech feature splicing unit is configured to perform splicing the enhanced speech features of each sub-band to obtain fused enhanced speech features;

[0176] The inverse short-time Fourier transform unit is configured to perform inverse short-time Fourier transform on the fused enhanced speech features to obtain a deep repaired speech signal corresponding to the impaired speech signal.

[0177] In some exemplary embodiments, the repaired speech feature generation unit 620 includes:

[0178] An encoding unit is configured to perform encoding processing on the impaired speech feature input to an encoder of the repaired speech generation network to obtain an encoding result.

[0179] A time sequence convolution unit is configured to perform feature extraction on the encoding result by a time sequence convolution neural network layer of the repaired speech generation network to obtain an intermediate time sequence feature vector.

[0180] A long short-term memory network processing unit is configured to perform feature extraction on the intermediate time sequence feature vector by a long short-term memory network layer of the repaired speech generation network to obtain a target time sequence feature vector.

[0181] A decoding unit is configured to perform speech feature reconstruction on the target time sequence feature vector input to a decoder of the repaired speech generation network to obtain the generated repaired speech feature.

[0182] In some exemplary embodiments, the apparatus 600 further comprises a model training unit, which comprises:

[0183] A sample acquisition unit is configured to perform acquisition of a sample impaired speech signal and a sample speech signal corresponding to the sample impaired speech signal, the sample impaired speech signal being obtained by performing multiple distortion processing on the sample speech signal.

[0184] A speech feature extraction unit is configured to perform extraction of speech features of the sample impaired speech signal and the sample speech signal respectively to obtain a sample impaired speech feature and a corresponding sample speech feature.

[0185] A generative adversarial network training unit is configured to perform training of a speech repair generative adversarial network based on the sample impaired speech feature and the corresponding sample speech feature, the speech repair generative adversarial network comprising the repaired speech generation network, the repaired speech generation network being used to generate a sample repaired speech feature based on the sample impaired speech feature.

[0186] A sub-band division unit is configured to perform division of the sample repaired speech feature generated based on the repaired speech generation network according to the plurality of sub-bands to obtain a sub-sample repaired speech feature corresponding to each sub-band.

[0187] A sub-band enhancement unit is configured to input the sub-sample repaired speech feature corresponding to each sub-band to a to-be-trained multi-band speech enhancement network, perform speech enhancement processing on the corresponding sub-sample repaired speech feature by a speech enhancement sub-network corresponding to each of the sub-bands in the to-be-trained multi-band speech enhancement network, and obtain a sample enhanced speech feature of each sub-band.

[0188] The multi-band speech enhancement network training unit is configured to train the multi-band speech enhancement network to be trained based on a difference between the sample enhanced speech feature of each sub-band and the sub-sample repaired speech feature of the corresponding sub-band, to obtain the trained multi-band speech enhancement network.

[0189] In some example embodiments, the generative adversarial network training unit comprises:

[0190] The generation unit is configured to input the sample damaged speech feature into a repaired speech generation network to be trained, to obtain a generated repaired speech feature corresponding to the sample damaged speech feature.

[0191] The discriminative network training unit is configured to fix the network parameters of the repaired speech generation network to be trained, to maximize a difference between the generated repaired speech feature corresponding to the sample damaged speech feature and the sample speech feature corresponding to the sample damaged speech feature as a training target, to train the speech repair discriminative network to be trained until convergence, to obtain the trained speech repair discriminative network.

[0192] The generation network training unit is configured to fix the network parameters of the speech repair discriminative network, to minimize a difference between the generated repaired speech feature corresponding to the sample damaged speech feature and the sample damaged speech feature as a training target, to train the repaired speech generation network to be trained until convergence, to obtain the trained repaired speech generation network.

[0193] The speech repair generative adversarial network comprises the repaired speech generation network and the speech repair discriminative network.

[0194] In some example embodiments, the multi-band speech enhancement network training unit comprises:

[0195] The sub-band loss determination unit is configured to determine, for each sub-band, a loss value corresponding to the sub-band based on a difference between the sample enhanced speech feature of the sub-band and the sub-sample repaired speech feature of the sub-band.

[0196] The total loss determination unit is configured to perform weighted summation on the loss values corresponding to each of the sub-bands, to obtain a total loss value.

[0197] The parameter adjustment unit is configured to adjust the network parameters of each of the speech enhancement sub-networks in reverse based on the total loss value, and continue iterative training based on the adjusted network parameters until a preset training end condition is met, to obtain the trained multi-band speech enhancement network.

[0198] In some example embodiments, the apparatus 600 further comprises:

[0199] a target distortion type determination unit configured to randomly select a plurality of target distortion types from a plurality of preset distortion types, the plurality of preset distortion types representing distortion types existing in the real-time voice communication link;

[0200] a sample damaged speech signal simulation unit configured to simulate a plurality of distortion damages on the sample speech signal based on a speech damage simulation manner corresponding to each target distortion type in the plurality of target distortion types, to obtain a sample damaged speech signal corresponding to the sample speech signal.

[0201] As to the apparatus in the above embodiments, the specific manners in which the respective modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0202] In an exemplary embodiment, an electronic device is also provided, including a processor, a memory for storing processor-executable instructions, wherein the processor is configured to implement the speech processing method provided in any of the above embodiments when executing the instructions stored in the memory.

[0203] The electronic device can be a terminal, a server or similar computing device. In the following, the electronic device is taken as an example of a server, Figure 7 is a block diagram of an electronic device running a speech processing method according to an exemplary embodiment, as shown in Figure 7 The server 700 can vary greatly in configuration and performance, and can include one or more central processing units (CPUs) 710 (the processor 710 can include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA processing device), a memory 730 for storing data, one or more storage media 720 (such as one or more mass storage devices) for storing application programs 723 or data 722. Among them, the memory 730 and the storage medium 720 can be temporary storage or persistent storage. The programs stored in the storage medium 720 can include one or more modules, each of which can include a series of instruction operations in the server. Further, the central processing unit 710 can be configured to communicate with the storage medium 720 and execute a series of instruction operations in the storage medium 720 on the server 700. The server 700 can also include one or more power supplies 760, one or more wired or wireless network interfaces 750, one or more input / output interfaces 740, and / or one or more operating systems 721, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0204] The input / output interface 740 can be used to receive or transmit data via a network. The network can include a wireless network provided by a communication provider of the server 700. In an example, the input / output interface 740 includes a network interface controller (NIC) that can be connected to other network devices through a base station to communicate with the Internet. In an example, the input / output interface 740 can be a radio frequency (RF) module that is used to communicate with the Internet through a wireless manner.

[0205] Those skilled in the art can understand that Figure 7 The structure shown is only schematic, and does not limit the structure of the electronic device described above. For example, the server 700 can further include more or less components than those shown, or have a different configuration of components than those shown. Figure 7 The structure shown is only schematic, and does not limit the structure of the electronic device described above. For example, the server 700 can further include more or less components than those shown, or have a different configuration of components than those shown. Figure 7 The structure shown is only schematic, and does not limit the structure of the electronic device described above. For example, the server 700 can further include more or less components than those shown, or have a different configuration of components than those shown.

[0206] In an example embodiment, a computer readable storage medium is also provided, which stores instructions when executed by a processor of an electronic device, so that the electronic device can perform the voice processing method provided in any of the above embodiments.

[0207] In an example embodiment, a computer program product is also provided, which includes a computer program, and the computer program is executed by a processor to implement the voice processing method provided in any of the above embodiments.

[0208] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features of the disclosure disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

[0209] It should be understood that the present disclosure is not limited to the precise structures as set forth above and shown in the drawings and that various modifications and changes can be made without departing from the scope thereof, which is set forth in the following claims.​

Claims

1. A voice processing method, characterized by, The method comprises the following steps: obtaining a damaged speech signal, extracting speech features of the damaged speech signal to obtain damaged speech features; inputting the damaged speech features into a repair speech generation network of a speech damage repair model to obtain generated repair speech features; the speech damage repair model is obtained by machine learning training based on a sample damaged speech signal and a sample speech signal corresponding to the sample damaged speech signal, and the sample damaged speech signal is obtained by multiple distortion processing on the sample speech signal; dividing the repair speech features according to a plurality of preset sub-frequency bands to obtain sub-repair speech features corresponding to each sub-frequency band; inputting the sub-repair speech features corresponding to each sub-frequency band into a multi-band speech enhancement network of the speech damage repair model, performing speech enhancement processing on the corresponding sub-repair speech features through a speech enhancement sub-network corresponding to each sub-frequency band in the multi-band speech enhancement network to obtain enhanced speech features of each sub-frequency band; the multi-band speech enhancement network is determined in the following manner: obtaining a sample damaged speech signal and a sample speech signal corresponding to the sample damaged speech signal; the sample damaged speech signal is obtained by multiple distortion processing on the sample speech signal; respectively extracting the speech features of the sample damaged speech signal and the sample speech signal to obtain sample damaged speech features and corresponding sample speech features; training a speech repair generation adversarial network based on the sample damaged speech features and the corresponding sample speech features, the speech repair generation adversarial network comprising the repair speech generation network, and the repair speech generation network being used to generate sample repair speech features based on the sample damaged speech features; dividing the sample repair speech features generated based on the repair speech generation network according to the plurality of sub-frequency bands to obtain sub-sample repair speech features corresponding to each sub-frequency band; inputting the sub-sample repair speech features corresponding to each sub-frequency band into a to-be-trained multi-band speech enhancement network, performing speech enhancement processing on the corresponding sub-sample repair speech features through a speech enhancement sub-network corresponding to each sub-frequency band in the to-be-trained multi-band speech enhancement network to obtain sample enhanced speech features of each sub-frequency band; training the to-be-trained multi-band speech enhancement network based on the difference between the sample enhanced speech features of each sub-frequency band and the sub-sample repair speech features of the corresponding sub-frequency band to obtain the trained multi-band speech enhancement network; generating a deep repair speech signal corresponding to the damaged speech signal based on the enhanced speech features of each sub-frequency band.

2. The method of claim 1, wherein, The method comprises the following steps: obtaining a damaged speech signal, extracting speech features of the damaged speech signal to obtain damaged speech features; 3. The method of claim 2, wherein, obtaining complex spectrum features by performing short-time Fourier transform on the damaged speech signal, and taking the complex spectrum features as the damaged speech features. The method comprises the following steps: splicing the enhanced speech features of each sub-frequency band to obtain fused enhanced speech features; perform inverse short-time Fourier transform on the fused enhanced speech feature to obtain a deep-repair speech signal corresponding to the damaged speech signal.

4. The method of claim 1, wherein, The inputting the damaged speech feature into a speech damage repair model repair speech generation network to obtain generated repair speech features includes: encoding the damaged speech feature input into an encoder of the repair speech generation network to obtain an encoding result; extracting features from the encoding result through a time sequence convolutional neural network layer of the repair speech generation network to obtain an intermediate time sequence feature vector; extracting features from the intermediate time sequence feature vector through a long short-term memory network layer of the repair speech generation network to obtain a target time sequence feature vector; inputting the target time sequence feature vector into a decoder of the repair speech generation network to reconstruct speech features to obtain the generated repair speech features.

5. The method of claim 1, wherein, The training of the speech repair generative adversarial network based on the sample damaged speech feature and the corresponding sample speech feature includes: inputting the sample damaged speech feature into a repair speech generation network to be trained to obtain a generated repair speech feature corresponding to the sample damaged speech feature; fixing the network parameters of the repair speech generation network to be trained to maximize the difference between the generated repair speech feature corresponding to the sample damaged speech feature and the sample speech feature corresponding to the sample damaged speech feature as a training target, training the speech repair discriminative network to be trained until convergence to obtain a trained speech repair discriminative network; fixing the network parameters of the speech repair discriminative network to minimize the difference between the generated repair speech feature corresponding to the sample damaged speech feature and the sample damaged speech feature as a training target, training the repair speech generation network to be trained until convergence to obtain the trained repair speech generation network. The speech repair generative adversarial network includes the repair speech generation network and the speech repair discriminative network.

6. The method of claim 1, wherein, The training of the multi-band speech enhancement network to be trained based on the difference between the sample enhanced speech feature of each sub-band and the sub-sample repair speech feature of the corresponding sub-band includes: for each sub-band, determining a loss value corresponding to the sub-band based on the difference between the sample enhanced speech feature of the sub-band and the sub-sample repair speech feature of the sub-band; performing weighted summation on the loss values corresponding to each of the sub-bands to obtain a total loss value; based on the total loss value, adjusting the network parameters of each speech enhancement sub-network in reverse, and based on the adjusted network parameters, continuing iterative training until a preset training end condition is met to obtain the trained multi-band speech enhancement network.

7. The method of claim 1, wherein, The multi-distortion processing on the sample speech signal includes: randomly selecting a plurality of target distortion types from a plurality of preset distortion types; the plurality of preset distortion types represent distortion types existing in a real-time speech communication link; Based on the speech damage simulation mode corresponding to each target distortion type in the plurality of target distortion types, a plurality of distortion damages are mixed to obtain a sample damage speech signal corresponding to the sample speech signal.

8. A speech processing device, characterized by Comprise: The damage speech feature determination unit is configured to execute the acquisition of the damage speech signal, extract the speech features of the damage speech signal, and obtain the damage speech features; The repair speech feature generation unit is configured to execute the input of the damage speech features into the multi-band speech enhancement network of the speech damage repair model to obtain the generated repair speech features; The speech damage repair model is obtained by machine learning training based on the sample damage speech signal and the sample speech signal corresponding to the sample damage speech signal, and the sample damage speech signal is obtained by multiple distortion processing on the sample speech signal; The repair speech feature division unit is configured to execute the division of the repair speech features according to a plurality of preset sub-bands to obtain sub-repair speech features corresponding to each sub-band; The sub-band enhancement unit is configured to execute the input of the sub-repair speech features corresponding to each sub-band into the multi-band speech enhancement network of the speech damage repair model, and perform speech enhancement processing on the corresponding sub-repair speech features through the speech enhancement sub-network corresponding to each sub-band in the multi-band speech enhancement network to obtain the enhanced speech features of each sub-band; The multi-band speech enhancement network is determined by the following model training unit, and the model training unit comprises: The sample obtaining unit is configured to perform obtaining a sample damaged speech signal and a sample speech signal corresponding to the sample damaged speech signal; the sample damaged speech signal is obtained by performing multiple distortion processing on the sample speech signal; the speech feature extraction unit is configured to perform extracting speech features of the sample damaged speech signal and the sample speech signal respectively, to obtain sample damaged speech features and corresponding sample speech features; the generative adversarial network training unit is configured to perform training a speech repair generative adversarial network based on the sample damaged speech features and the corresponding sample speech features, the speech repair generative adversarial network comprising the repair speech generation network, the repair speech generation network being used for generating sample repair speech features based on the sample damaged speech features; the sub-band division unit is configured to perform dividing the sample repair speech features generated based on the repair speech generation network according to the multiple sub-bands, to obtain sub-sample repair speech features corresponding to each of the sub-bands; the sub-band enhancement unit is configured to perform inputting the sub-sample repair speech features corresponding to each of the sub-bands into a to-be-trained multi-band speech enhancement network, performing speech enhancement processing on the corresponding sub-sample repair speech features by a speech enhancement sub-network corresponding to each of the sub-bands in the to-be-trained multi-band speech enhancement network, to obtain sample enhanced speech features of each sub-band; the multi-band speech enhancement network training unit is configured to perform training the to-be-trained multi-band speech enhancement network based on differences between the sample enhanced speech features of each sub-band and the sub-sample repair speech features of the corresponding sub-band, to obtain the trained multi-band speech enhancement network. The repair speech signal generation unit is configured to perform generating a deep repair speech signal corresponding to the damaged speech signal based on the enhanced speech features of each of the sub-bands.

9. An electronic device, comprising: It comprises: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the speech processing method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When the instructions in the computer readable storage medium are executed by the processor of the electronic device, the electronic device can perform the speech processing method of any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the speech processing method of any one of claims 1 to 7.

Images