Speech processing method and apparatus, storage medium, and computer device

Abstract

The application discloses a speech processing method, characterized in that the method comprises: acquiring initial speech features of a call speech; inputting the initial speech features into a pre-trained speech enhancement model to obtain target speech features output by the speech enhancement model, the speech enhancement model being obtained through step-by-step training based on a deep clustering loss function and a mask inference loss function; and calculating a target speech with noise and reverberation removed according to the target speech features. In this way, the speech enhancement model is trained through the deep clustering loss function and the mask inference loss function, noise and reverberation in the initial speech features can be removed more accurately, clean target speech features are output to obtain the target speech with noise and reverberation removed, and the application embodiment effectively improves the performance of speech enhancement while reducing the model calculation resources by applying artificial intelligence technology.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and more specifically, to a speech processing method, apparatus, storage medium, and computer equipment. Background Technology

[0002] Speech enhancement is essentially speech noise reduction. In daily life, the speech captured by microphones is usually "polluted" speech with different kinds of noise. The main purpose of speech enhancement is to recover the clean speech we want from these "polluted" noisy speech, thereby effectively suppressing various interference signals and enhancing the target speech signal. This not only improves the quality of speech but also helps to improve the performance of speech recognition.

[0003] Speech enhancement has applications in areas such as video conferencing and speech recognition. It serves as a preprocessing module in many speech coding and recognition systems and can generally be divided into near-field speech enhancement and far-field speech enhancement. In complex speech acquisition environments, noise and reverberation coexist. Existing speech enhancement methods employ noise reduction and reverberation détente schemes based on two-level networks. However, the large computational load of these two-level networks makes speech enhancement unable to meet the performance requirements of practical applications. Summary of the Invention

[0004] This application provides a speech processing method, apparatus, storage medium, and computer device, aiming to improve the performance of speech enhancement.

[0005] On one hand, embodiments of this application provide a speech processing method, which includes: acquiring initial speech features of a call speech; inputting the initial speech features into a pre-trained speech enhancement model to obtain target speech features output by the speech enhancement model, wherein the speech enhancement model is obtained by step-by-step training based on a deep clustering loss function and a mask inference loss function; and calculating the target speech with noise and reverberation removed based on the target speech features.

[0006] On the other hand, embodiments of this application also provide a speech processing apparatus, which includes: an acquisition module for acquiring initial speech features of a call speech; an enhancement module for inputting the initial speech features into a pre-trained speech enhancement model to obtain target speech features output by the speech enhancement model, wherein the speech enhancement model is obtained by step-by-step training based on a deep clustering loss function and a mask inference loss function; and a calculation model for calculating the target speech with noise and reverberation removed based on the target speech features.

[0007] On the other hand, this application also provides a computer device, which includes a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are called by the processor, the above-mentioned voice processing method is executed.

[0008] On the other hand, embodiments of this application also provide a computer-readable storage medium storing program code, wherein the above-described speech processing method is executed when the program code is run by a processor.

[0009] On the other hand, embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium and executes the computer instructions, causing the computer to perform the steps in the aforementioned voice processing method.

[0010] This application provides a speech processing method that acquires initial speech features from a call and inputs these features into a pre-trained speech enhancement model to obtain target speech features output by the model. This speech enhancement model is obtained through step-by-step training based on a deep clustering loss function and a mask inference loss function. Furthermore, based on the target speech features, the method calculates the target speech with noise and reverberation removed. Thus, by training the pre-set speech enhancement model using two different loss functions, the model is guided to efficiently remove noise and reverberation from the speech features, improving speech enhancement performance while reducing computational resources. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 The illustration shows a schematic diagram of a commonly used noise reduction and de-reverberation method provided in an embodiment of this application.

[0013] Figure 2 A schematic diagram of the architecture of a speech processing system provided in an embodiment of this application is shown.

[0014] Figure 3 A schematic flowchart of a speech processing method provided in an embodiment of this application is shown.

[0015] Figure 4 The illustration shows an application scenario diagram of a speech processing method provided in an embodiment of this application.

[0016] Figure 5 A schematic diagram of the architecture of a speech enhancement model provided in an embodiment of this application is shown.

[0017] Figure 6 A flowchart illustrating another speech processing method provided in an embodiment of this application is shown.

[0018] Figure 7 A schematic diagram of a speech feature extraction process provided in an embodiment of this application is shown.

[0019] Figure 8 A schematic diagram of the architecture of a preset enhancement network provided in an embodiment of this application is shown.

[0020] Figure 9 A block diagram of a voice processing device provided in an embodiment of this application is shown.

[0021] Figure 10 This is a block diagram of a computer device provided in an embodiment of this application.

[0022] Figure 11 This is a block diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0023] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0024] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0025] In daily life, we often encounter problems with voice communication under noise interference. Examples include using mobile phones in cars or trains, dealing with ambient noise, and the noisy, distant speech captured by microphones during multi-person video conferences. Therefore, speech enhancement technology is needed to extract the purest possible original speech from noisy speech signals. Depending on the call scenario, users can make calls using client applications, including near-field and far-field calls. However, near-field calls are only suitable for one-on-one or small-group calls at close range, and the audio and video experience is generally mediocre.

[0026] To improve user experience, industrial research focuses on long-distance calling in large-screen communication devices. However, long-distance calls suffer from lower signal-to-noise ratios due to the greater distance, and the speech is often accompanied by noise and reverberation. Therefore, it is necessary to utilize higher-performance far-field speech enhancement to reduce noise and dedeverage. Currently, mainstream speech enhancement solutions in the industry typically employ two models for noise reduction and dedeverage respectively. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 Two commonly used noise reduction and dereverberation methods are shown: noise reduction followed by dereverberation and dereverberation followed by noise reduction.

[0027] For example, a microphone array can be divided into different subsets. Each subset passes through a first-level speech enhancement network to obtain the enhanced speech from each microphone. The enhanced speech is then combined and passed through a second-level speech enhancement network to obtain the final output. However, this two-level network-based speech enhancement scheme requires a large amount of computation during training, which is not suitable for the performance requirements of actual product applications. Reducing the number of network parameters to decrease the computational load will lead to a deterioration in the speech enhancement performance.

[0028] To address the aforementioned problems, the inventors, through research, proposed the speech processing method provided in this application. This method acquires the initial speech features of a call and inputs these features into a pre-trained speech enhancement model to obtain the target speech features output by the model. This speech enhancement model is obtained through step-by-step training based on a deep clustering loss function and a mask inference loss function, thereby fusing two models (two-level networks) into a single model to reduce the computational cost of model training. Furthermore, based on the target speech features, the target speech with noise and reverberation removed is calculated. Thus, by training the pre-set speech enhancement model using different loss functions, the model is guided to efficiently remove noise and reverberation from the initial speech features, improving speech enhancement performance while reducing model computational resources.

[0029] The following section will introduce one application scenario of the speech processing method involved in this application. Figure 2 A schematic diagram of the architecture of a voice processing system is shown. In some embodiments, the voice processing system 300 is applied in a remote video conferencing scenario. The voice processing system 300 may include a near-end client 310, a far-end client 330, and a server 350. The near-end client 310, the far-end client 330, and the server 350 are connected via a network. As one implementation, the near-end client 310 and the far-end client 330 may be large-screen terminals for video conferencing, and the server 350 may be a cloud server.

[0030] For example, the remote client 330 can collect the initial speech with noise and reverberation emitted by the participants and transmit it to the server 350. After receiving the initial speech, the server 350 can use a pre-trained speech enhancement model to reduce noise and remove reverberation from the initial speech to obtain enhanced clean speech (target speech), and then transmit the clean speech to the near client 310. Optionally, the speech enhancement model can also be configured on either the near client 310 or the remote client 330 according to the needs of the actual application scenario.

[0031] It should be noted that the above-mentioned voice processing system 300 is merely an example. The architecture and application scenarios of the voice processing system described in this embodiment are intended to more clearly illustrate the technical solutions of this embodiment and do not constitute a limitation on the technical solutions provided by this embodiment. As those skilled in the art will know, with the evolution of voice processing system architecture and the emergence of new application scenarios, the technical solutions provided by this embodiment are also applicable to similar technical problems.

[0032] Please see Figure 3 , Figure 3 A schematic flowchart of a speech processing method according to an embodiment of this application is shown. In a specific embodiment, the speech processing method is applied to, for example... Figure 9 The speech processing device 500 and the computer device 600 equipped with the speech processing device 500 are shown. Figure 10 ).

[0033] The following will use a computer device as an example to illustrate the specific process of this embodiment. Of course, it is understood that the computer device used in this embodiment can be a server or a terminal, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, blockchain, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these.

[0034] The following will address Figure 3 The process shown combines Figure 4 The application scenarios shown will be explained in detail. Figure 4This illustration shows an application scenario of a speech processing method provided in an embodiment of this application. In this scenario, the speech processing method can be applied to a specific speech enhancement system. This system can deploy a speech enhancement model 411 on a cloud server 410. The cloud server 410 can communicate with two conference terminals (a first conference terminal 430 and a second conference terminal 450) in two meeting rooms. The first conference terminal 430 and the second conference terminal 450 can collect the speech of participants in their respective meeting rooms and upload the collected speech to the cloud server 410. The cloud server 410 then enhances the speech to obtain clean speech. Finally, the cloud server 410 transmits the clean speech to the corresponding conference terminal for playback. The speech processing method may specifically include the following steps:

[0035] Step S110: Obtain the initial voice features of the call voice.

[0036] In this embodiment, the computer device can acquire the initial speech features of the call speech that needs to be enhanced. The initial speech features are acoustic features obtained from the conversion of the call speech, such as logarithmic power spectrum (LPS) and Mel-frequency cepstral coefficients (MFCC), etc., which are not limited here.

[0037] Because speech data cannot be directly input into models for training like image data, and because it lacks significant feature changes over long time, it is difficult to learn the features of speech data. Furthermore, speech time-domain data typically consists of a 16K sampling rate (16,000 samples per second), directly inputting these time-domain samples would result in excessively large training data volumes and make it difficult to achieve practically meaningful results. Therefore, in speech processing tasks, speech data is usually converted into acoustic features to serve as the model's input or output.

[0038] As one implementation method, after acquiring the call speech, frame segmentation and windowing processing can be performed on the speech to obtain initial speech features. For example, frame segmentation and windowing processing can be performed sequentially on the call speech acquired by all microphones to obtain speech signal frames. A Fast Fourier Transform (FFT) is then performed on the speech signal frames, and the discrete power spectrum after the FFT is obtained. Logarithmic calculation is then performed on the obtained discrete power spectrum to obtain the logarithmic power spectrum as the initial speech features. By performing frame segmentation and windowing processing on the call speech, the call speech can be transformed from a non-stationary time-varying signal in the time domain to a stationary signal in the frequency domain, facilitating model training.

[0039] exist Figure 4 In the video conferencing scenario shown, the distance between participants and the conferencing terminal results in noise and reverberation in the participants' voices captured by the conferencing terminal. Therefore, the speech processing method provided in this application embodiment can be used to enhance the participants' voices, thereby removing noise and reverberation.

[0040] For example, the second conference terminal 450 collects the voices of the participants 420 in the conference room through the microphone, that is, the voice of the call, and sends the voice of the call to the cloud server 410 through the network. Then, after receiving the voice of the call, the cloud server 410 performs frame processing, windowing processing and Fourier transform on the voice of the call to obtain the initial voice features.

[0041] Step S120: Input the initial speech features into the pre-trained speech enhancement model to obtain the target speech features output by the speech enhancement model.

[0042] In practical applications, microphone arrays capture speech containing both noise and reverberation. Considering the two-stage networks used for noise reduction and dereverberation, the large number of parameters in each network during training requires significant computational resources. Furthermore, reducing the number of parameters in each network degrades the model's performance in both noise reduction and dereverberation. Therefore, the two-stage networks can be merged into a single network. The merged model has fewer parameters than the two individual networks, significantly reducing the computational load during training and improving the model's performance in speech enhancement.

[0043] In this embodiment, the speech enhancement model can generate target speech features corresponding to the call speech based on the initial input speech features; that is, clean speech features after speech enhancement, removing noise and reverberation. Please refer to [link to relevant documentation]. Figure 5 , Figure 5 A schematic diagram of the architecture of a speech enhancement model is shown. This speech enhancement model may include a hidden layer, a deep clustering layer, a speech mask inference layer, and a noise mask inference layer.

[0044] The deep clustering layer, speech mask inference layer, and noise mask inference layer can all be linear layers, with their inputs derived from the output of the hidden layer. The hidden layer can calculate intermediate features based on the initial speech features of the input, and these intermediate features serve as intermediate values ​​in the speech enhancement process.

[0045] The speech mask inference layer can perform mask inference (MI) based on intermediate features to obtain target speech features with noise and reverberation removed. The noise mask inference layer can perform mask inference based on intermediate features to obtain speech features with noise. The deep clustering layer can assist the speech mask inference layer and the noise mask inference layer in noise reduction and de-reverberation by performing deep clustering (DC) on the obtained intermediate features. Preferably, the hidden layer can be a Long Short-Term Memory (LSTM) network, because speech features have short-term stationary temporal sequences, which is consistent with the long short-term memory capability of LSTM.

[0046] As one implementation method, the model can be trained step-by-step during training using the deep clustering loss function corresponding to the deep clustering layer and the mask inference loss functions corresponding to the speech mask inference layer and the noise mask inference layer, respectively. For example, in the first step, a denoising model can be trained based on the deep clustering loss function and the mask inference loss function. Training stops once the denoising model converges. The mask inference loss function for the speech mask inference layer uses clean speech labels with reverberation but no noise. In the second step, a denoising model is trained. The denoising model trained in the first step is used as the denoising model, and training is performed based on the deep clustering loss function and the mask inference loss function. Training stops once the denoising model converges. The mask inference loss function for the speech mask inference layer uses clean speech labels with neither noise nor reverberation. Thus, the final denoising model, i.e., the speech enhancement model, has the ability to perform both denoising and denoising simultaneously.

[0047] It should be noted that the deep clustering layer of the speech enhancement model uses a binary loss based on time-frequency clustering. Due to the regularization properties of the deep clustering loss, it can better guide the speech mask inference layer and noise mask inference layer to remove noise and reverberation from the speech during training, thereby effectively improving the model's performance in speech enhancement. Step-by-step training allows both the denoising and dereverberation tasks to achieve optimal training results in their respective independent training processes, contributing to improved capabilities of the speech enhancement model in both denoising and dereverberation.

[0048] Therefore, the speech enhancement model obtained through the above training can obtain intermediate features through multiple LSTM layers. Furthermore, the speech mask inference layer can perform mask inference based on these intermediate features to calculate the speech mask, i.e., the target speech features. For example, as follows... Figure 4In the video conferencing scenario shown, after the cloud server 410 obtains the initial speech features, it can input these features into the speech enhancement model 411. The speech mask inference layer of the speech enhancement model 411 can calculate the speech mask, i.e., the target speech features, based on the intermediate features obtained through multiple LSTM layers. In practical speech enhancement applications, since only the target speech features output by the speech mask inference layer need to be used to recover the speech, the computational load of the speech enhancement process is effectively reduced.

[0049] Step S130: Calculate the target speech after removing noise and reverberation based on the target speech features.

[0050] As one implementation method, an inverse feature transform can be performed on the acquired target speech features to calculate the target speech with noise and reverberation removed. For example, an inverse Fourier transform can be performed on the target speech features to transform them from the frequency domain to the time domain, thereby obtaining the enhanced time-domain speech, i.e., the target speech. For example, as... Figure 4 In the video conferencing scenario shown, after the cloud server 410 obtains the target speech features output by the speech enhancement model 411, it can convert the target speech features, i.e. the clean speech features, into target speech through inverse Fourier transform, thereby obtaining clean speech with noise and reverberation removed. Furthermore, the cloud server 410 can send the clean speech to the first conference terminal 430, and the speaker of the first conference terminal 430 will play the speech of the participants 420 without noise and reverberation.

[0051] In this embodiment, initial speech features of the call speech can be obtained and input into a pre-trained speech enhancement model to obtain the target speech features output by the speech enhancement model. This speech enhancement model is obtained through step-by-step training based on a deep clustering loss function and a mask inference loss function. Furthermore, based on the target speech features, the target speech with noise and reverberation removed is calculated. Thus, by training the pre-set speech enhancement model using different loss functions, the model is guided to efficiently remove noise and reverberation from the initial speech features, thereby improving the performance of speech enhancement while reducing model computational resources.

[0052] Based on the methods described in the above embodiments, the following examples will provide further detailed explanations.

[0053] In this embodiment, the explanation will be based on the example of the voice processing device being specifically integrated into a computer device.

[0054] Please see Figure 6 , Figure 6This application illustrates another speech processing method provided by an embodiment of the present application. In a specific embodiment, this video processing speech processing method utilizes, for example... Figure 8 The preset enhancement network is shown below. The following will address... Figure 5 The process shown will be explained in detail.

[0055] This application incorporates Artificial Intelligence (AI) technology. AI technology utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines capable of reacting in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0056] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0057] The solutions provided in this application involve technologies such as artificial intelligence speech technology. Key technologies in speech technology include Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and Voiceprint Recognition (VPR). Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech being one of the most promising methods for future human-computer interaction.

[0058] The following will combine Figure 6 The process shown and Figure 8 The network architecture diagram shown is explained in detail. This speech processing method may specifically include the following steps:

[0059] Step S210: The computer device acquires the training sample set.

[0060] The speech processing method provided in this application includes training a preset enhancement network. It is worth noting that the training of the preset enhancement network can be performed in advance based on the acquired training sample data set. Subsequently, whenever speech enhancement is needed on the initial speech features of the call speech, the target speech features for removing noise and reverberation can be calculated using the trained speech enhancement model, without having to train the preset enhancement network again each time speech enhancement is performed.

[0061] Optionally, the wsj0-2mix (Wall Street Journal) dataset is used to determine the training sample set, which contains 30 hours of speech training data and 10 hours of speech training data. By randomly selecting different speaker voices from the corresponding sets and mixing them with a random relative signal-to-noise ratio (SNR) between 0dB and 10dB, noisy and reverberant speech can be generated for use in network training.

[0062] As one implementation, the step of the computer device acquiring the training sample set may include:

[0063] (1) The computer device acquires the first sample speech.

[0064] (2) The computer equipment extracts speech features from the first sample speech to obtain noisy speech features.

[0065] (3) The computer device acquires the second sample speech.

[0066] (4) The computer device extracts speech features from the second sample speech to obtain the first clean speech label and the second clean speech label.

[0067] (5) The computer equipment determines the deep clustering label based on the first sample speech and the second sample speech.

[0068] The first sample speech consists of noisy and reverberant speech captured by a microphone. The second sample speech consists of clean speech with reverberation but no noise, and clean speech without reverberation but no noise. The deep clustering label is the ratio of the feature sizes of the first and second sample speech at each time-frequency point.

[0069] For example, computer devices can directly collect noisy and reverberant speech through a microphone. For instance, in a video conference, the speech of participants collected through the microphone of a large-screen conference terminal can be used as the first sample speech. In the actual training process, technicians can directly obtain the first sample speech from the pre-built noise-reduced training corpus.

[0070] Furthermore, the computer device can extract speech features from the acquired first sample speech; please refer to [link to relevant documentation]. Figure 7 , Figure 7 The diagram illustrates a speech feature extraction process, in which a microphone captures a conversation speech containing noise and reverberation, i.e., the first sample speech. Multi-frame speech signals were obtained through frame segmentation and windowing processes, respectively. n represents the total number of frames, and t represents the time domain. The computer can then perform an FFT on each frame of the speech signal, transforming it from the time domain to the frequency domain to obtain the corresponding discrete power spectrum. Finally, the logarithm of the obtained discrete power spectrum is calculated to obtain the logarithmic power spectrum. f represents the frequency domain space. By concatenating the features from all microphones, the final noise speech feature xnoisy can be obtained. Optionally, noise speech labels can be generated based on the noise speech features, i.e.

[0071] Computer equipment can also obtain clean speech as a reference from the noise-reduced training corpus as the second sample speech. To facilitate step-by-step training of the preset enhancement network, clean speech with reverberation but without noise and clean speech without reverberation can be obtained. Then, speech features are extracted from the clean speech with reverberation but without noise to obtain the first clean speech label, and speech features are extracted from the clean speech without reverberation but without noise to obtain the second clean speech label. In the actual calculation process, the noise speech label... First Clean Voice Tag And the second clean voice tag The mathematical expression for this is an eigenvector (Embedding), where the length of the eigenvector is the dimension of the feature.

[0072] In one implementation, a computer device can determine deep clustering labels by comparing the speech energy of a first sample speech and a second sample speech at each time-frequency point. Since speech signals change over time, their energy also changes over time. Therefore, when calculating the energy of a digitized speech signal, the overall energy is not calculated; instead, the energy at each time-frequency point is calculated frame by frame. For example, a computer device can use the energy ratio of noise-free speech with reverberation to noisy speech as a deep clustering label, or it can use the energy ratio of noise-free speech without reverberation to noisy speech as a deep clustering label. This deep clustering label is used to calculate the deep clustering loss function.

[0073] Step S220: The computer device acquires the preset enhanced network.

[0074] Considering the stringent latency (i.e., real-time requirements) requirements for the industrial application of speech enhancement technology, it is necessary to minimize the number of parameters in the speech enhancement model. However, this would significantly reduce the effectiveness of the speech enhancement. Therefore, this application proposes merging two-level networks into a single network, enabling the speech enhancement model to simultaneously perform noise reduction and dereverberation, thereby improving the speech enhancement effect without reducing the number of model parameters.

[0075] Please see Figure 8 , Figure 8 A schematic diagram of a pre-defined augmentation network architecture is shown. This pre-defined augmentation network includes hidden layers, deep clustering layers, and a mask inference layer. It is a multi-head output network with shared weights at the bottom layer. The introduction of the deep clustering layer assists the speech mask inference layer and the noise mask inference layer in performing mask inference, enabling them to effectively distinguish noise and reverberation in speech during network training. The hidden layers can utilize LSTM or a bidirectional long short-term memory (Bi-LSTM) network. Figure 8 The hidden layer shown is LSTM, and the mask inference layer includes a speech mask layer (Clean-MI) and a noise mask layer (Noise-MI).

[0076] The speech masking inference layer calculates the speech mask, i.e., the clean speech label, while the noise masking inference layer calculates the noise and reverberation mask, i.e., the noisy speech label. It should be noted that in practical applications, only the mask output by the speech masking inference layer is needed to recover the speech; therefore, the computational load of the speech enhancement process is not increased, thus improving the efficiency of speech enhancement.

[0077] Step S230: The computer device trains the preset enhancement network step by step through the training sample set to remove noise and reverberation until the preset enhancement network meets the preset conditions, and the trained target enhancement network is obtained as the speech enhancement model.

[0078] The target augmentation network, i.e., the speech augmentation model, obtained after training, needs to perform two augmentation tasks simultaneously: noise reduction and dereverberation. If these two augmentation tasks are trained at the same time, the pre-set augmentation network training will not achieve optimal training results. Therefore, a step-by-step training approach can be adopted, performing the training process for the two tasks separately.

[0079] Specifically, this application provides two step-by-step training methods. For example, noise removal training can be performed first, followed by reverberation removal training, or vice versa. The purpose of noise removal training is to enable the network to reduce noise, while the purpose of reverberation removal training is to enable the network to remove reverberation. This allows both enhancement tasks to achieve optimal training results during their individual training processes, thereby improving the performance of the speech enhancement model in speech enhancement.

[0080] In some embodiments, the step of training a preset enhancement network stepwise with a training sample set for noise removal and reverberation removal until the preset enhancement network meets preset conditions may include:

[0081] (1) The computer device inputs the noisy speech features into the hidden layer and generates intermediate training features through the hidden layer.

[0082] (2) The computer device inputs the intermediate training features into the deep clustering layer and generates clustering training labels through the deep clustering layer.

[0083] (3) The computer device inputs the intermediate training features into the speech mask inference layer, and generates clean speech training features through the speech mask inference layer.

[0084] (4) The computer device inputs the intermediate training features into the noise mask inference layer and generates noisy speech training features through the noise mask inference layer.

[0085] (5) The computer equipment constructs a target loss function based on clean speech labels, noisy speech labels, deep clustering annotations, clean speech training features, noisy speech training features and clustering training annotations, and performs noise removal training and reverberation removal training on the preset enhancement network step by step according to the target loss function until the preset enhancement network meets the preset conditions.

[0086] The intermediate training features are intermediate values ​​generated by the hidden layers of the pre-defined augmentation network. These values ​​can be used as shared values ​​and input to the deep clustering layer, speech mask inference layer, and noise mask inference layer, respectively, thereby achieving weight sharing at the bottom layer and reducing the number of network parameters. The speech mask inference layer and the noise mask inference layer can generate clean speech training features y based on the intermediate training features. clean Training features for noisy speech y noise Deep clustering layers can generate clustering training labels y based on intermediate training features. dc .

[0087] In one implementation, the computer device constructs a target loss function based on clean speech labels, noisy speech labels, deep clustering annotations, clean speech training features, noisy speech training features, and clustering training annotations. The step of training a preset enhancement network step-by-step for noise removal and reverberation removal based on the target loss function until the preset enhancement network meets preset conditions may include:

[0088] (5.1) The computer device determines the first loss function based on the cluster training labels and deep cluster labels.

[0089] The first loss function is the deep clustering loss function. For example, the first loss function... y dc For cluster training labeling, This is for deep clustering annotation.

[0090] (5.2) The computer device determines the second loss function based on the clean speech training features and clean speech labels.

[0091] For the two step-by-step training methods, two different second loss functions can be determined based on different clean speech labels.

[0092] Optionally, computer equipment can train features based on clean speech. clean And the first clean voice tag Determine the noise removal loss function The noise removal loss function is used as the second loss function.

[0093] Optionally, computer equipment can train features based on clean speech. clean Second Clean Voice Tags Determine the noise removal loss function The noise removal loss function is used as the second loss function.

[0094] (5.3) The computer equipment determines the third loss function based on the training features of the noisy speech and the noisy speech labels.

[0095] For example, the third loss function Among them, y noise Training features for noisy speech. For noisy speech tags.

[0096] Among them, the second loss function Loss clean and the third loss function This is the mask inference loss function.

[0097] (5.4) The computer device constructs the target loss function of the preset augmentation network based on the first loss function, the second loss function and the third loss function, and performs noise removal training and reverberation removal training on the preset augmentation network step by step according to the target loss function until the preset augmentation network meets the preset conditions.

[0098] For example, the computer device can be based on the first loss function Loss dc The second loss function Loss clean and the third loss function Loss noise Construct the target loss function Loss of the pre-defined augmentation network, as shown in the following formula:

[0099]

[0100] Where α, β, and γ are weight parameters. Further, the target loss function Loss is used to train the preset augmentation network step-by-step for noise removal and reverberation removal until the preset augmentation network meets preset conditions. Optionally, the preset augmentation network can be trained based on multi-task learning, thereby jointly using the deep clustering loss function and the mask inference loss function to learn both the noise reduction and reverberation removal tasks simultaneously. The two tasks can share the information they have learned during the learning process by sharing parameters, resulting in a better generalization effect for the trained target augmentation network.

[0101] Noise typically refers to unwanted sounds in certain situations, such as human chatter and various sudden noises. Reverberation refers to the persistence of sound even after a sound source in a room has stopped emitting sound. Considering that different application scenarios have different needs for speech enhancement—for example, in multi-person conference rooms, the main focus is on removing noise from the sound collected by the conference terminal, while in professional recording studios, the main focus is on removing reverberation from the sound collected by the recording equipment—different step-by-step training methods can be used based on the actual scenario in which the final speech enhancement model will be used.

[0102] In some embodiments, application scenario attributes can be obtained based on the actual scenario in which the final speech enhancement model is used, and a corresponding distributed training strategy can be determined based on the application scenario attributes. Further, based on the distributed training strategy, a target loss function for the preset enhancement network is constructed according to the first loss function, the second loss function, and the third loss function. The preset enhancement network is then trained step-by-step for noise removal and reverberation removal according to this target loss function until the preset enhancement network meets preset conditions.

[0103] The application scenario attribute is used to characterize the actual scenario in which the speech enhancement model is applied, such as a scenario attribute focusing on noise reduction or a scenario attribute focusing on dereverberation. The distributed training strategy includes a first distributed training strategy and a second distributed training strategy. The first distributed training strategy, for scenarios focusing on noise reduction, trains first on noise removal and then on dereverberation removal. The second distributed training strategy, for scenarios focusing on dereverberation, trains first on dereverberation removal and then on noise removal.

[0104] As one implementation method, in applications where noise removal is the goal, such as multi-person video conferences, the conference terminal collects not only the speaker's voice but also the voices of other speakers. Noise reduction processing is needed for the audio collected by the conference terminal. To achieve this, noise removal training can be performed first, followed by reverberation removal training. The computer device can determine the target loss function of the preset enhancement network based on a first distribution training strategy, according to a first loss function, a second loss function, and a third loss function. This second loss function is determined by the noise removal loss function. The preset enhancement network is then iteratively trained for noise removal based on the target loss function until it meets preset conditions, resulting in a noise removal network that only performs noise reduction.

[0105] Furthermore, the computer device can determine the target loss function of the noise removal network based on the first loss function, the second loss function, and the third loss function. This second loss function is determined by the reverberation removal loss function. Then, the noise removal network is iteratively trained for reverberation removal based on the target loss function until the noise removal network meets preset conditions. In this way, performing separate noise removal training first avoids interference from reverberation factors during the training process, thereby enabling the generated target enhancement network to have better noise reduction performance.

[0106] As another implementation, in applications where reverberation removal is the primary objective, such as recording studios where high sound quality is required, removing unnecessary reverberation is particularly important. Therefore, reverberation removal training can be performed first, followed by noise removal training. The computer equipment can determine the target loss function of the preset enhancement network based on a second distribution training strategy, according to a first loss function, a second loss function, and a third loss function. This second loss function is determined by the reverberation removal loss function. The preset enhancement network is then iteratively trained for reverberation removal based on the target loss function until it meets preset conditions, resulting in a reverberation removal network that only performs the function of removing reverberation.

[0107] Furthermore, the computer device can determine the target loss function of the reverberation removal network based on the first loss function, the second loss function, and the third loss function. This second loss function is determined by the noise removal loss function. The reverberation removal network is then iteratively trained for noise removal based on the target loss function until it meets preset conditions. In this way, performing separate reverberation removal training first avoids interference from noise factors during the training process, thereby enabling the generated target enhancement network to have better dereverberation performance.

[0108] Preferably, when noise is precisely defined, the concept of noise essentially includes reverberation. Therefore, when there are no special requirements for the application scenario of the speech enhancement model, the preset enhancement network can be trained for noise removal first, and then trained for reverberation removal. This allows the network to learn the ability to remove reverberation on the basis of an excellent noise reduction network. In this way, the optimal training effect can be achieved in both training processes, thereby improving the performance of the speech enhancement model in speech enhancement.

[0109] It should be noted that the preset conditions can be: the total loss value of the target loss function is less than a preset value, the total loss value of the target loss function no longer changes, or the preset number of training iterations has been reached. Optionally, an optimizer can be used to optimize the target loss function, and the learning rate, batch size, and epochs during training can be set based on experimental experience.

[0110] Understandably, after iteratively training the network to be trained (preset augmentation network / noise removal network / reverberation removal network) for multiple training cycles based on the training sample data set, where each training cycle includes multiple iterations, continuously optimizing the parameters of the network to be trained, the total loss value will gradually decrease until it reaches a fixed value or is less than the preset value. At this point, it indicates that the network to be trained has converged. Alternatively, it can be determined that the preset augmentation network / noise removal network / reverberation removal network has converged after the number of training iterations reaches a preset number.

[0111] The pre-defined augmentation network is trained using multi-task learning. Although a combination of deep clustering loss and mask inference loss is used for training, the mask inference loss is only used in the verification process of the target augmentation network, i.e., the speech augmentation model selection. When the speech augmentation model is running, the output of the mask inference branch is used as the mask after speech augmentation, i.e., the target speech feature.

[0112] Step S240: The computer device acquires the initial speech features of the call speech.

[0113] Step S250: The computer device inputs the initial speech features into the hidden layer and generates intermediate features through the hidden layer.

[0114] Step S260: The computer device inputs the intermediate features into the speech mask inference layer, generates clean speech features through the speech mask inference layer, and uses the clean speech features as the target speech features.

[0115] In one implementation, after acquiring the voice call, the computer device can extract voice features from the voice call, including performing frame segmentation, windowing, and Fourier transform on the voice call to obtain initial voice features. Further, the computer device can input the initial voice features into the hidden layer of the voice enhancement network to generate intermediate features through the hidden layer. Further, the computer device can input the intermediate features into the voice mask inference layer to generate clean voice features through the voice mask inference layer, and use the clean voice features as the target voice features.

[0116] Step S270: The computer device performs inverse feature transformation on the target speech features to calculate the target speech with noise and reverberation removed.

[0117] As one implementation method, after acquiring the target speech features, the computer device can perform an inverse feature transformation on the target speech features to convert the target speech features (mask) in the frequency domain space to the target speech in the time domain space. Optionally, the inverse feature transformation can be an inverse Fourier transform. In this embodiment, a training sample set and a preset enhancement network can be acquired. The preset enhancement network is then trained step by step for noise removal and reverberation removal using the training sample set until the preset enhancement network meets preset conditions. The trained target enhancement network is then obtained as a speech enhancement model. Further, the initial speech features are input into the hidden layer, intermediate features are generated through the hidden layer, and the intermediate features are input into the speech mask inference layer. The speech mask inference layer generates clean speech features, which are then used as the target speech features. An inverse feature transformation is then performed on the target speech features to calculate the target speech with noise and reverberation removed. Thus, only the target speech features output by the speech mask inference layer of the speech enhancement model need to be used to recover the speech, avoiding increasing the computational load of the speech enhancement process and improving the efficiency of speech enhancement.

[0118] Please see Figure 9 The diagram illustrates a structural block diagram of a speech processing device 500 provided in an embodiment of this application. The speech processing device 500 includes: an acquisition module 510 for acquiring initial speech features of a call; an enhancement module 520 for inputting the initial speech features into a pre-trained speech enhancement model to obtain target speech features output by the speech enhancement model, wherein the speech enhancement model is obtained through step-by-step training based on a deep clustering loss function and a mask inference loss function; and a calculation model 530 for calculating the target speech with noise and reverberation removed based on the target speech features.

[0119] In some embodiments, the speech processing device 500 may further include: a sample acquisition module, a network acquisition module, and a model training module. The sample acquisition module is used to acquire a training sample set, which includes noisy speech features, clean speech labels, noisy speech labels, and deep clustering annotations. The network acquisition module is used to acquire a preset enhancement network, which includes hidden layers, deep clustering layers, and mask inference layers. The network training module is used to perform noise removal training and reverberation removal training on the preset enhancement network step by step using the training sample set until the preset enhancement network meets preset conditions, thereby obtaining the trained target enhancement network as a speech enhancement model.

[0120] In some embodiments, the mask inference layer includes a speech mask inference layer and a noise mask inference layer. The network training module may include: a hiding unit for inputting noisy speech features into the hiding layer and generating intermediate training features through the hiding layer; a deep clustering unit for inputting intermediate training features into the deep clustering layer and generating clustering training labels through the deep clustering layer; a speech inference unit for inputting intermediate training features into the speech mask inference layer and generating clean speech training features through the speech mask inference layer; a noise inference unit for inputting intermediate training features into the noise mask inference layer and generating noisy speech training features through the noise mask inference layer; and a network training unit for constructing a target loss function based on clean speech labels, noisy speech labels, deep clustering labels, clean speech training features, noisy speech training features, and clustering training labels, and performing noise removal training and reverberation removal training on the preset enhancement network step by step according to the target loss function until the preset enhancement network meets the preset conditions.

[0121] In some embodiments, the network training unit includes: a first subunit, configured to determine a first loss function based on clustering training labels and deep clustering labels; a second subunit, configured to determine a second loss function based on clean speech training features and clean speech labels; a third subunit, configured to determine a third loss function based on noisy speech training features and noisy speech labels; and a training subunit, configured to construct a target loss function for a preset augmentation network based on the first, second, and third loss functions, and to perform noise removal training and reverberation removal training on the preset augmentation network step by step according to the target loss function until the preset augmentation network meets preset conditions.

[0122] In some embodiments, the second subunit may be specifically used to: determine a noise removal loss function based on clean speech training features and a first clean speech label; use the noise removal loss function as a second loss function, where the first clean speech label is a speech label obtained based on noise-free reverberant speech.

[0123] In some embodiments, the second subunit may also be specifically used to: determine a reverberation removal loss function based on clean speech training features and a second clean speech label; use the reverberation removal loss function as the second loss function, where the second clean speech label is a speech label obtained based on noise-free and reverberation-free speech.

[0124] In some embodiments, the training subunit may be specifically used to: determine a target loss function for a preset augmentation network based on a first loss function, a second loss function, and a third loss function; iteratively train the preset augmentation network for noise removal based on the target loss function until the preset augmentation network meets a preset condition to obtain a noise removal network, wherein the second loss function is determined by the noise removal loss function; determine a target loss function for the noise removal network based on the first loss function, a reverberation removal loss function, and a third loss function; iteratively train the noise removal network for reverberation removal based on the target loss function until the noise removal network meets a preset condition, wherein the second loss function is determined by the reverberation removal loss function.

[0125] In some embodiments, the training subunit may be specifically used to: determine a target loss function for a preset enhancement network based on a first loss function, a second loss function, and a third loss function; iteratively train the preset enhancement network for reverberation removal based on the target loss function until the preset enhancement network meets a preset condition to obtain a reverberation removal network, wherein the second loss function is determined by the reverberation removal loss function; determine a target loss function for the reverberation removal network based on the first loss function, the second loss function, and the third loss function; iteratively train the reverberation removal network for noise removal based on the target loss function until the reverberation removal network meets a preset condition, wherein the second loss function is determined by the noise removal loss function.

[0126] In some embodiments, the sample acquisition module may be specifically used to: acquire a first sample speech, wherein the first sample speech is noisy speech collected by a microphone; extract speech features from the first sample speech to obtain noisy speech features; acquire a second sample speech, wherein the second sample speech includes clean speech with reverberation and clean speech without noise and without reverberation; extract speech features from the second sample speech to obtain a first clean speech label and a second clean speech label; and determine a deep clustering label based on the first sample speech and the second sample speech.

[0127] In some embodiments, the speech enhancement model includes a hidden layer, a deep clustering layer, a speech mask inference layer, and a noise mask inference layer. The enhancement module 520 can be specifically used to: input initial speech features into the hidden layer and generate intermediate features through the hidden layer; input the intermediate features into the speech mask inference layer and generate clean speech features through the speech mask inference layer, and use the clean speech features as the target speech features.

[0128] The computational model 530 can be specifically used to perform inverse feature transformation on the target speech features to calculate the target speech with noise and reverberation removed.

[0129] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0130] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0131] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0132] The solution provided in this application can acquire initial speech features of a call and input these features into a pre-trained speech enhancement model to obtain the target speech features output by the model. This speech enhancement model is obtained through step-by-step training based on a deep clustering loss function and a mask inference loss function. Furthermore, based on the target speech features, the target speech with noise and reverberation removed is calculated. Thus, by training the pre-set speech enhancement model using different loss functions, the model is guided to efficiently remove noise and reverberation from speech, improving speech enhancement performance while reducing computational resources.

[0133] like Figure 10 As shown in the figure, this application embodiment also provides a computer device 600, which includes a processor 610, a memory 620, a power supply 630, and an input unit 640. The memory 620 stores computer program instructions. When the computer program instructions are called by the processor 610, they can execute various method steps provided in the above embodiments. Those skilled in the art will understand that the structure of the computer device shown in the figures does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0134] The processor 610 may include one or more processing cores. The processor 610 connects to various parts of the entire battery management system using various interfaces and lines. It executes instructions, programs, code sets, or instruction sets stored in the memory 620, calls data stored in the memory 620, performs various functions and processes data within the battery management system, and performs various functions and processes data within the computer device, thereby providing overall control of the computer device. Optionally, the processor 610 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 610 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 610 and may be implemented separately using a communication chip.

[0135] The memory 620 may include random access memory (RAM) or read-only memory (ROM). The memory 620 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 620 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described below, etc. The data storage area may also store data created during the use of the computer device (such as phonebook and audio / video data). Accordingly, the memory 620 may also include a memory controller to provide the processor 610 with access to the memory 620.

[0136] The power supply 630 can be logically connected to the processor 610 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 630 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0137] The input unit 640 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0138] Although not shown, the computer device 600 may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 610 in the computer device loads the executable files corresponding to the processes of one or more applications into the memory 620 according to the following instructions, and the processor 610 runs the applications stored in the memory 620, thereby implementing the various method steps provided in the foregoing embodiments.

[0139] like Figure 11 As shown, this application embodiment also provides a computer-readable storage medium 700, which stores computer program instructions 710, which can be called by a processor to execute the methods described in the above embodiments.

[0140] Computer-readable storage media can be electronic storage devices such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, computer-readable storage media include non-volatile computer-readable storage medium. Computer-readable storage medium 700 has storage space for program code that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in a suitable form.

[0141] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations of the above embodiments.

[0142] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Although this application has disclosed preferred embodiments as above, it is not intended to limit this application. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the technical solution of this application. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of this application without departing from the scope of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. A speech processing method, characterized in that, The method includes: The initial speech features and speech enhancement model of the call speech are obtained; the speech enhancement model includes a hidden layer, a deep clustering layer, a speech mask inference layer, and a noise mask inference layer; the speech enhancement model is obtained by step-by-step training based on the deep clustering loss function and the mask inference loss function; The initial speech features are input into the hidden layer, and intermediate features are generated through the hidden layer. The intermediate features are input into the speech mask inference layer, and clean speech features are generated through the speech mask inference layer. The clean speech features are then used as the target speech features. The target speech features are subjected to inverse feature transformation to calculate the target speech with noise and reverberation removed.

2. The method according to claim 1, characterized in that, The speech enhancement model is trained through the following steps: Obtain a training sample set, which includes noisy speech features, clean speech labels, noisy speech labels, and deep clustering annotations; Obtain a preset enhancement network, which includes a hidden layer, a deep clustering layer, and a mask inference layer; The preset enhancement network is trained step by step using the training sample set to remove noise and reverberation until the preset enhancement network meets the preset conditions, and the trained target enhancement network is obtained as the speech enhancement model.

3. The method according to claim 2, characterized in that, The mask inference layer includes a speech mask inference layer and a noise mask inference layer. The step-by-step training of the preset enhancement network on the training sample set for noise removal and reverberation removal, until the preset enhancement network meets preset conditions, includes: The noisy speech features are input into the hidden layer, and intermediate training features are generated through the hidden layer. The intermediate training features are input into the deep clustering layer, and clustering training labels are generated through the deep clustering layer. The intermediate training features are input into the speech mask inference layer, and clean speech training features are generated through the speech mask inference layer. The intermediate training features are input into the noise mask inference layer, and the noise mask inference layer generates noisy speech training features. A target loss function is constructed based on the clean speech label, the noisy speech label, the deep clustering annotation, the clean speech training features, the noisy speech training features, and the clustering training annotation. The preset enhancement network is then trained step by step for noise removal and reverberation removal based on the target loss function until the preset enhancement network meets the preset conditions.

4. The method according to claim 3, characterized in that, The step of constructing a target loss function based on the clean speech label, the noisy speech label, the deep clustering annotation, the clean speech training features, the noisy speech training features, and the clustering training annotation, and then training the preset enhancement network step by step for noise removal and reverberation removal based on the target loss function until the preset enhancement network meets preset conditions, includes: The first loss function is determined based on the clustering training labels and the deep clustering labels; Based on the clean speech training features and the clean speech labels, a second loss function is determined; Based on the noisy speech training features and the noisy speech labels, a third loss function is determined; Based on the first loss function, the second loss function, and the third loss function, a target loss function for the preset enhancement network is constructed. The preset enhancement network is then trained step by step for noise removal and reverberation removal based on the target loss function until the preset enhancement network meets the preset conditions.

5. The method according to claim 4, characterized in that, The clean speech label includes a first clean speech label, and the step of determining a second loss function based on the clean speech training features and the clean speech label includes: Based on the clean speech training features and the first clean speech label, determine the noise removal loss function; The noise removal loss function is used as the second loss function, and the first clean speech label is a speech label obtained based on speech with reverberation but no noise.

6. The method according to claim 4, characterized in that, The clean speech label includes a second clean speech label, and the step of determining the second loss function based on the clean speech training features and the clean speech label includes: Based on the clean speech training features and the second clean speech label, determine the reverberation removal loss function; The reverberation removal loss function is used as the second loss function, and the second clean speech label is a speech label obtained based on speech without noise and reverberation.

7. The method according to claim 5 or 6, characterized in that, The step involves constructing a target loss function for the preset enhancement network based on the first loss function, the second loss function, and the third loss function, and then training the preset enhancement network step-by-step for noise removal and reverberation removal based on the target loss function until the preset enhancement network meets preset conditions, including: Obtain application scenario attributes; Determine the corresponding distribution training strategy based on the application scenario attributes; Based on the distributed training strategy, a target loss function for the preset augmentation network is constructed according to the first loss function, the second loss function, and the third loss function. The preset augmentation network is then trained step by step for noise removal and reverberation removal according to the target loss function until the preset augmentation network meets the preset conditions.

8. The method according to claim 7, characterized in that, The distributed training strategy includes a first distributed training strategy. Based on the distributed training strategy, a target loss function for the preset augmentation network is constructed according to the first loss function, the second loss function, and the third loss function. The preset augmentation network is then trained step-by-step for noise removal and reverberation removal according to the target loss function until the preset augmentation network meets preset conditions, including: When the distribution training strategy is the first distribution training strategy, the target loss function of the preset augmentation network is determined according to the first loss function, the second loss function and the third loss function, and the preset augmentation network is iteratively trained for noise removal according to the target loss function until the preset augmentation network meets the preset conditions to obtain the noise removal network, wherein the second loss function is determined by the noise removal loss function; Based on the first loss function, the second loss function, and the third loss function, a target loss function for the noise removal network is determined, and the noise removal network is iteratively trained for reverberation removal based on the target loss function until the noise removal network meets a preset condition. The second loss function is determined by the reverberation removal loss function.

9. The method according to claim 7, characterized in that, The distributed training strategy includes a second distributed training strategy. Based on the distributed training strategy, a target loss function for the preset augmentation network is constructed according to the first loss function, the second loss function, and the third loss function. The preset augmentation network is then trained step-by-step for noise removal and reverberation removal according to the target loss function until the preset augmentation network meets preset conditions, including: When the distribution training strategy is the second distribution training strategy, the target loss function of the preset enhancement network is determined according to the first loss function, the second loss function and the third loss function, and the preset enhancement network is iteratively trained for reverberation removal according to the target loss function until the preset enhancement network meets the preset conditions to obtain the reverberation removal network, wherein the second loss function is determined by the reverberation removal loss function; Based on the first loss function, the second loss function, and the third loss function, a target loss function for the reverberation removal network is determined, and the reverberation removal network is iteratively trained for noise removal based on the target loss function until the reverberation removal network meets a preset condition. The second loss function is determined by the noise removal loss function.

10. The method according to claim 2, characterized in that, The acquisition of the training sample set includes: Acquire a first sample speech, which is a noisy and reverberant speech collected based on a microphone; Speech features are extracted from the first sample speech to obtain noisy speech features; Acquire a second sample speech, which includes clean speech with reverberation but no noise and clean speech without reverberation but no noise. Speech features are extracted from the second sample speech to obtain a first clean speech label and a second clean speech label; Based on the first sample speech and the second sample speech, determine the deep clustering label.

11. A voice processing device, characterized in that, The device includes: The acquisition module is used to acquire the initial speech features of the call speech and the speech enhancement model; the speech enhancement model includes a hidden layer, a deep clustering layer, a speech mask inference layer, and a noise mask inference layer; the speech enhancement model is obtained by step-by-step training based on the deep clustering loss function and the mask inference loss function; The enhancement module is used to input the initial speech features into the hidden layer, generate intermediate features through the hidden layer; input the intermediate features into the speech mask inference layer, generate clean speech features through the speech mask inference layer, and use the clean speech features as the target speech features; A computational model is used to perform inverse feature transformation on the target speech features to calculate the target speech with noise and reverberation removed.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 10.

13. A computer device, characterized in that, include: Memory; One or more processors are coupled to the memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes computer instructions stored in a storage medium, a processor of a computer device reads the computer instructions from the storage medium, and the processor executes the computer instructions to cause the computer device to perform the method as described in any one of claims 1 to 10.

Images