Speech processing method, speech processing model training method and speech denoising method

Abstract

The embodiment of the invention provides a voice processing method, a voice processing model training method and a voice denoising method, and the voice processing method comprises the steps: determining a to-be-processed voice; inputting the to-be-processed voice into a voice processing model, and performing frequency domain conversion processing on the to-be-processed voice in the voice processing model to obtain a voice spectrogram of the to-be-processed voice; dividing the voice spectrogram according to a preset bandwidth to obtain a plurality of sub-voice spectrograms corresponding to the to-be-processed voice; and performing de-noising processing on each sub-speech spectrogram in the plurality of sub-speech spectrograms, and obtaining a de-noised target speech according to a processing result. The full-band sub-speech spectrogram is considered, and the integrity and smoothness of the full-band speech signal are ensured while the to-be-processed speech is denoised, so that the high quality of the denoised target speech is ensured.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a speech processing method, a speech processing model training method, and a speech denoising method. Background Art

[0002] With the development of artificial intelligence, intelligent voice has gradually been applied to various tasks, such as voice wake-up, voice broadcasting, and intelligent voice assistants. For example, smart homes can perform tasks based on user voice commands, and intelligent robots can even engage in conversations with users and answer their questions. However, voice signals may contain interference such as ambient noise, which can prevent accurate recognition and processing of received voice signals, hindering the execution of subsequent intelligent voice tasks and thus affecting the user experience. Therefore, an effective technical solution is urgently needed to address these issues. Summary of the Invention

[0003] In view of this, embodiments of this specification provide a speech processing method. One or more embodiments of this specification also relate to a speech processing device, a speech processing model training method, a speech processing model training device, a speech denoising method, a speech denoising device, a speech recognition method, a speech recognition device, a computing device, a computer-readable storage medium, and a computer program product to address technical deficiencies in the prior art.

[0004] According to a first aspect of the embodiments of this specification, a speech processing method is provided, including:

[0005] Determine the speech to be processed;

[0006] Inputting the speech to be processed into a speech processing model, and performing frequency domain conversion processing on the speech to be processed in the speech processing model to obtain a speech spectrogram of the speech to be processed;

[0007] Dividing the speech spectrogram according to a preset bandwidth to obtain a plurality of sub-speech spectrograms corresponding to the speech to be processed;

[0008] Denoising is performed on each of the multiple sub-speech spectrograms, and a denoised target speech is obtained according to the processing result.

[0009] According to a second aspect of the embodiments of this specification, there is provided a speech processing device, including:

[0010] A determination module, configured to determine a speech to be processed;

[0011] An input module is configured to input the speech to be processed into a speech processing model, and in the speech processing model, perform frequency domain conversion processing on the speech to be processed to obtain a speech spectrogram of the speech to be processed;

[0012] a division module configured to divide the speech spectrogram according to a preset bandwidth to obtain a plurality of sub-speech spectrograms corresponding to the speech to be processed;

[0013] The denoising module is configured to perform denoising processing on each of the multiple sub-speech spectrograms and obtain the denoised target speech according to the processing result.

[0014] According to a third aspect of the embodiments of this specification, a speech processing model training method is provided, which is applied to a cloud-side device, including:

[0015] Determining a noisy speech sample and a denoised speech label corresponding to the noisy speech sample;

[0016] Inputting the noisy speech sample into a speech processing model, and performing frequency domain conversion processing on the noisy speech sample in the speech processing model to obtain a sample speech spectrogram of the noisy speech sample;

[0017] Dividing the sample speech spectrogram according to a preset bandwidth to obtain a plurality of sample sub-speech spectrograms corresponding to the noisy speech sample;

[0018] Performing denoising processing on each of the plurality of sample sub-speech spectrograms and obtaining a predicted denoised speech according to the processing result;

[0019] The speech processing model is trained according to the denoised speech label and the predicted denoised speech until a speech processing model that meets a training stop condition is obtained.

[0020] According to a fourth aspect of the embodiments of this specification, a speech processing model training device is provided, which is applied to a cloud-side device, including:

[0021] A determination module is configured to determine a noisy speech sample and a denoised speech label corresponding to the noisy speech sample;

[0022] An input module is configured to input the noisy speech sample into a speech processing model, and in the speech processing model, perform frequency domain conversion processing on the noisy speech sample to obtain a sample speech spectrogram of the noisy speech sample;

[0023] a dividing module configured to divide the sample speech spectrogram according to a preset bandwidth to obtain a plurality of sample sub-speech spectrograms corresponding to the noisy speech sample;

[0024] A denoising module is configured to perform denoising processing on each of the plurality of sample sub-speech spectrograms and obtain a predicted denoised speech according to the processing result;

[0025] The training module is configured to train the speech processing model according to the denoised speech label and the predicted denoised speech until a speech processing model that meets the training stop condition is obtained.

[0026] According to a fifth aspect of the embodiments of this specification, a speech denoising method is provided, which is applied to a cloud-side device, including:

[0027] Receive noisy voice sent by the client;

[0028] Inputting the noisy speech into a speech processing model, and performing frequency domain conversion processing on the noisy speech in the speech processing model to obtain a speech spectrogram of the noisy speech;

[0029] Dividing the speech spectrogram according to a preset bandwidth to obtain a plurality of sub-speech spectrograms corresponding to the noisy speech;

[0030] Performing denoising processing on each of the plurality of sub-speech spectrograms, and obtaining a denoised target speech according to the processing result;

[0031] The target voice is sent to the client.

[0032] According to a sixth aspect of the embodiments of this specification, a speech denoising apparatus is provided, which is applied to a cloud-side device, including:

[0033] A receiving module, configured to receive noisy speech sent by a client;

[0034] An input module is configured to input the noisy speech into a speech processing model, and in the speech processing model, perform frequency domain conversion processing on the noisy speech to obtain a speech spectrogram of the noisy speech;

[0035] a division module configured to divide the speech spectrogram according to a preset bandwidth to obtain a plurality of sub-speech spectrograms corresponding to the noisy speech;

[0036] A denoising module is configured to perform denoising processing on each of the multiple sub-speech spectrograms and obtain a denoised target speech according to the processing result;

[0037] The sending module is configured to send the target voice to the client.

[0038] According to a seventh aspect of the embodiments of this specification, a speech recognition method is provided, which is applied to a cloud-side device, including:

[0039] receiving a user's voice signal;

[0040] Inputting the speech signal into a speech processing model, and performing frequency domain conversion processing on the speech signal in the speech processing model to obtain a speech spectrogram of the speech signal;

[0041] Dividing the speech spectrogram according to a preset bandwidth to obtain a plurality of sub-speech spectrograms corresponding to the speech signal;

[0042] Performing denoising processing on each of the multiple sub-speech spectrograms, and obtaining a denoised speech signal according to the processing result;

[0043] Recognize the de-noised voice signal and generate a reply message according to the recognition result.

[0044] According to an eighth aspect of the embodiments of this specification, a speech recognition apparatus is provided, which is applied to a cloud-side device, including:

[0045] A receiving module configured to receive a user's voice signal;

[0046] An input module is configured to input the speech signal into a speech processing model, and in the speech processing model, perform frequency domain conversion processing on the speech signal to obtain a speech spectrogram of the speech signal;

[0047] a division module configured to divide the speech spectrogram according to a preset bandwidth to obtain a plurality of sub-speech spectrograms corresponding to the speech signal;

[0048] A denoising module is configured to perform denoising processing on each of the multiple sub-speech spectrograms and obtain a denoised speech signal according to the processing result;

[0049] The recognition module is configured to recognize the denoised voice signal and generate reply information according to the recognition result.

[0050] According to a ninth aspect of the embodiments of this specification, a computing device is provided, including:

[0051] memory and processor;

[0052] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above method are implemented.

[0053] According to a tenth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and the computer program / instruction implements the steps of the above method when executed by a processor.

[0054] According to an eleventh aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0055] One embodiment of the present specification provides a speech processing method, comprising: determining a speech to be processed; inputting the speech to be processed into a speech processing model, performing frequency domain conversion processing on the speech to be processed in the speech processing model to obtain a speech spectrogram of the speech to be processed; dividing the speech spectrogram according to a preset bandwidth to obtain multiple sub-speech spectrograms corresponding to the speech to be processed; performing denoising processing on each of the multiple sub-speech spectrograms, and obtaining a denoised target speech according to the processing results.

[0056] In the above method, when denoising the speech to be processed, a speech processing model can be called, in which a speech spectrogram of the speech to be processed is determined, and the speech spectrogram is divided into multiple sub-speech spectrograms according to a preset bandwidth. Denoising is performed on each sub-speech spectrogram, so that the denoised target speech is obtained according to the processing results. In the process of denoising the speech to be processed, each sub-speech spectrogram of the preset bandwidth is processed, taking into account the sub-speech spectrograms of the full frequency band. While denoising the speech to be processed, the integrity and smoothness of the full-band speech signal are guaranteed, and the environmental noise and reverberation present in the speech to be processed are eliminated, thereby ensuring the high quality of the denoised target speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a schematic diagram of an application scenario of a speech processing method provided by an embodiment of this specification;

[0058] Figure 2 This is a flow chart of a speech processing method provided by one embodiment of this specification;

[0059] FIG3( a ) is a schematic diagram of the structure of a speech processing model in a speech processing method provided in one embodiment of this specification;

[0060] FIG3( b ) is a schematic diagram of a processing flow of a frequency band division unit in a speech processing model provided in one embodiment of this specification;

[0061] FIG3( c ) is a schematic diagram of a processing flow of a sequence encoding unit in a speech processing model provided in one embodiment of this specification;

[0062] FIG3( d ) is a schematic diagram of a processing flow of a mask estimation unit in a speech processing model provided in one embodiment of this specification;

[0063] Figure 4 This is a schematic diagram of comparison results of speech processed by a speech processing method provided in one embodiment of this specification;

[0064] Figure 5 This is a flowchart of a speech processing method provided by one embodiment of this specification;

[0065] Figure 6 This is a structural diagram of a speech processing device provided by one embodiment of this specification;

[0066] Figure 7 This is a flowchart of a speech processing model training method provided by one embodiment of this specification;

[0067] Figure 8 This is a structural diagram of a speech processing model training device provided by one embodiment of this specification;

[0068] Figure 9 This is a flow chart of a speech denoising method provided by one embodiment of this specification;

[0069] Figure 10 This is a structural diagram of a speech denoising device provided by an embodiment of this specification;

[0070] Figure 11 This is a flow chart of a speech recognition method provided by one embodiment of this specification;

[0071] Figure 12 This is a schematic diagram of the structure of a speech recognition device provided by one embodiment of this specification;

[0072] Figure 13 This is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0073] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0074] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0075] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0076] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0077] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. A large model can also be called a cornerstone model / foundation model. It is pre-trained on a large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as a large language model (LLM) and a multi-modal pre-training model.

[0078] When large models are used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. Large models can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0079] First, the terms involved in one or more embodiments of this specification are explained.

[0080] AI: Artificial Intelligence.

[0081] SE: Speech Enhancement, speech enhancement.

[0082] Deep Learning: Deep learning.

[0083] RNN: Recurrent Neural network.

[0084] LSTM: Long Short Term Memory Networks, long short-term memory network.

[0085] BSRNN: Band-Split Recurrent Neural network, band-split recurrent neural network.

[0086] MLP: MultiLayer Perceptron, multilayer perceptron.

[0087] TFM: Time-Frequency Mask, time-frequency mask.

[0088] STFT: Short Time Fourier Transform, short time Fourier transform.

[0089] ISTFT: Inverse Short Time Fourier Transform, inverse short time Fourier transform.

[0090] MOS: Mean Opinion Score, average opinion score.

[0091] PESQ: Perceptual Evaluation of Speech Quality, speech quality perception evaluation.

[0092] STOI: Short Time Objective Intelligibility, short-time target intelligibility.

[0093] SNR: Signal-to-noise Ratio.

[0094] MSE: Mean Square Error.

[0095] SDR: Signal-to-Distortion Ratio, signal-distortion ratio.

[0096] In practical applications, as the artificial intelligence industry matures, intelligent voice is gradually being applied to a variety of tasks, such as voice wake-up, voice broadcasting, and intelligent voice assistants. Today, more and more people are becoming accustomed to the companionship and assistance provided by AI voice. As a key pre-processing step for intelligent voice tasks, the role of speech enhancement is self-evident. This technology not only helps human listeners better understand the content, but also facilitates more accurate recognition and transcription by machine listeners. It can reduce environmental noise while preventing the loss of speech and human voices, ensuring the fluency and integrity of speech.

[0097] Although current speech enhancement algorithms are simple to implement and require minimal computation, they all have limitations. Specifically, they struggle to suppress non-stationary, sudden noises, such as the sudden sounds of keyboard tapping, car horns, and doors and windows closing. Traditional noise reduction algorithms struggle to eliminate these noises completely. Second, after enhancement, traditional algorithms leave a significant amount of residual noise, which can lead to poor subjective listening and even affect the intelligibility of speech information, compromising subsequent tasks like recognition and translation. Third, traditional algorithms contain numerous assumptions, making them difficult to adapt to complex and changing real-world scenarios. Fourth, in low signal-to-noise ratio conditions, traditional noise reduction methods are nearly ineffective. Fifth, traditional speech enhancement algorithms are unable to eliminate ambient reverberation.

[0098] In practical applications, leveraging the high precision, high performance, and powerful model representation capabilities of deep learning technology offers new insights into this field. The input to a speech enhancement neural network is typically a time-frequency map of the audio signal or features extracted from it. The output is either a time-frequency map or a time-frequency mask (TFM). Due to the wider dynamic range of the time-frequency map, the model requires complex nonlinear operations to eliminate the frequency components of noise. Furthermore, increasing the number of convolutional and pooling layers enhances the nonlinear processing capabilities of the neural network, but this also introduces more distortion. While these models can significantly improve the quality of the desired speech signal, they typically only consider narrowband and wideband signals, limiting their performance across the full frequency range. First, different audio pickup devices have different frequency responses. Some devices accurately capture the high-frequency information of the speech signal, while others may introduce significant distortion in the high-frequency range. This unpredictable device-dependent distortion in high-frequency components can degrade the performance of full-band speech enhancement. Secondly, the model's training objectives, such as the signal-to-noise ratio (SNR) and frequency-domain mean square error (MSE), are not closely related to human auditory perception and other objective perceptual indicators, and are therefore ineffective in eliminating environmental reverberation. Therefore, an effective technical solution is urgently needed to address these issues.

[0099] In this specification, a speech processing method is provided. This specification also relates to a speech processing device, a speech processing model training method, a speech processing model training device, a speech denoising method, a speech denoising device, a speech recognition method, a speech recognition device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0100] See also Figure 1 , Figure 1 A schematic diagram of an application scenario of a speech processing method provided by an embodiment of this specification is shown.

[0101] Figure 1 It includes a dialogue robot 102 and a cloud-side device 104.

[0102] In a specific implementation, a user can communicate with the conversational robot 102. When the user speaks, the conversational robot 102 can receive the voice signal emitted by the user and send the voice signal to the cloud-side device 104. The cloud-side device 104 can input the voice signal into the voice processing model. In the voice processing model, the voice signal is converted into the frequency domain to obtain a voice spectrogram of the voice signal. The voice spectrogram is divided according to a preset bandwidth to obtain multiple sub-speech spectrograms corresponding to the noisy speech. Each of the multiple sub-speech spectrograms is denoised. Based on the processing results, a denoised voice signal is obtained. The denoised voice signal is recognized, and a reply message is generated based on the recognition result in response to the user's voice signal. The cloud-side device 104 can then send the reply message to the conversational robot 102, which then feeds the reply message back to the user through a conversation. By denoising the user's voice signal, the conversational robot ensures the accuracy of the recognition of the user's voice signal, enabling conversational interaction between the user and the conversational robot.

[0103] The cloud-side device 104 can be understood as a server that provides various services, including physical servers and cloud servers. For example, a server that provides communication services to multiple clients, a server that provides background training to support models used on clients, and a server that processes data sent by clients. It should be noted that the cloud-side device 104 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The cloud-side device 104 can also be a server for a distributed system, or a server that integrates a blockchain. The cloud-side device 104 can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0104] It is worth noting that the speech processing method provided in the embodiments of this specification can be executed by the cloud-side device 104. In other embodiments of this specification, the speech processing model can be deployed in the dialogue robot 102, so that the dialogue robot 102 can also have similar functions as the cloud-side device 104, thereby executing the speech processing method provided in the embodiments of this specification; in other embodiments, the speech processing method provided in the embodiments of this specification can also be jointly executed by the dialogue robot 102 and the cloud-side device 104.

[0105] See also Figure 2 , Figure 2A flow chart of a speech processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0106] Step 202: Determine the speech to be processed.

[0107] Specifically, the voice processing method provided in one embodiment of this specification can be applied to the field of intelligent voice. For example, for an intelligent robot, it can denoise the received voice signal, generate a reply message based on the denoised voice, and play the reply message in voice form to complete an intelligent conversation with the user. Alternatively, when a user issues a voice command to a smart home, the voice command can be denoised, recognized after the denoised voice command, and the task issued by the user to the smart home can be executed based on the denoised voice command.

[0108] The speech to be processed can be understood as a speech signal that carries noise and requires speech enhancement. The speech to be processed can be speech received by the intelligent robot when the user is speaking, or it can be speech sent by the user through the client. This embodiment of the specification is not limited to this.

[0109] Step 204: Input the speech to be processed into a speech processing model, and perform frequency domain conversion processing on the speech to be processed in the speech processing model to obtain a speech spectrogram of the speech to be processed.

[0110] The speech processing model can be understood as a neural network model used for speech enhancement. In practical applications, the speech processing model can be a full-band speech enhancement model based on a band-recurrent neural network. Frequency domain conversion can be understood as converting the speech to be processed from the time domain to the frequency domain, that is, from a speech signal waveform to a complex-valued spectrogram. Therefore, the speech spectrogram of the speech to be processed can be understood as a complex-valued spectrogram, i.e., a frequency domain representation of the speech to be processed.

[0111] Specifically, the speech to be processed can be input into a speech processing model, and the speech processing model is used to perform frequency domain conversion on the speech to be processed, converting the speech to be processed from a time domain waveform diagram to a frequency domain representation, thereby obtaining a speech spectrum diagram of the speech to be processed.

[0112] In practical applications, the speech processing model may include a Fourier transform unit, which can be used to perform a Fourier transform on the speech to be processed to achieve frequency domain conversion of the speech to be processed. Furthermore, in one embodiment of the present specification, a short-time Fourier transform can be performed on the speech to be processed based on a preset Fourier transform length and frame shift to obtain a speech spectrogram of the speech to be processed. In practical applications, the Fourier transform length can be set to 2048, and the frame shift can be set to 512. In addition, different Fourier transform lengths and frame shifts can be set according to actual needs, and this embodiment of the present specification does not limit this.

[0113] Furthermore, the dimension of the speech to be processed is (B, t). The speech to be processed is input into the speech processing model, and the speech processing model performs frequency domain conversion on the speech to be processed. After obtaining a speech spectrogram of the speech to be processed, the dimension of the speech spectrogram of the speech to be processed changes to (B, F, T, 2), where B is the number of speech to be processed. During model training, multiple speech to be processed can be trained simultaneously. During model inference, B can be set to 1, meaning only one piece of data is inferred at a time. T represents the total number of samples for each speech to be processed. It is understood that the speech to be processed input into the speech processing model is a speech signal, an analog signal. For processing, this analog signal needs to be converted to a digital signal, and then the speech signal can be sampled. For example, in an audio signal with a sampling rate of 48kHz, the number of samples of 1 second of speech to be processed is 48,000. F can be used to represent the frequency dimension, F = fft_size / 2+1 represents the number of frequency points of each frame of speech to be processed, T can be used to represent the time dimension, T = t / hop_size+1 represents the total number of frames of speech to be processed, where fft_size is the Fourier transform length, hop_size is the frame shift, and 2 can be used to represent the two dimensions of amplitude spectrum (i.e., real part) and phase spectrum (i.e., imaginary part).

[0114] It can be understood that after the frequency domain conversion processing is performed on the speech to be processed, the dimension of the speech to be processed is increased, which facilitates the subsequent processing of the speech to be processed according to multiple dimensions of the speech to be processed, further ensuring the comprehensiveness and accuracy of denoising.

[0115] In practical applications, before using the speech processing model to denoise the speech to be processed, model training is required to enable the speech processing model to learn denoising capabilities during the model training process. The specific implementation method is as follows:

[0116] Before inputting the to-be-processed speech into the speech processing model, the method further includes:

[0117] Determining a noisy speech sample and a denoised speech label corresponding to the noisy speech sample;

[0118] Inputting the noisy speech sample into a speech processing model, and performing frequency domain conversion processing on the noisy speech sample in the speech processing model to obtain a sample speech spectrogram of the noisy speech sample;

[0119] Dividing the sample speech spectrogram according to a preset bandwidth to obtain a plurality of sample sub-speech spectrograms corresponding to the noisy speech sample;

[0120] Performing denoising processing on each of the plurality of sample sub-speech spectrograms and obtaining a predicted denoised speech according to the processing result;

[0121] The speech processing model is trained according to the denoised speech label and the predicted denoised speech until a speech processing model that meets a training stop condition is obtained.

[0122] The noisy speech sample can be understood as a speech signal containing environmental noise, which can be used as a training sample during the speech processing model training process. The denoised speech label corresponding to the noisy speech sample can be understood as a speech signal identical to the noisy speech sample but without the noise, which can be used as a training label during the speech processing model training process. These training samples and training labels can be used to perform supervised training on the speech processing model. The training stopping condition can be understood as the model loss value reaching a preset loss value threshold and / or the number of training times reaching a preset threshold.

[0123] Based on this, a noisy speech sample and a denoised speech label corresponding to the noisy speech sample can be obtained from a training database, and the noisy speech sample can be input into a speech processing model. In the speech processing model, the noisy speech sample can be converted into a frequency domain to obtain a sample speech spectrogram of the noisy speech sample, and the sample speech spectrogram can be divided according to a preset bandwidth to obtain multiple sample sub-speech spectrograms corresponding to the noisy speech sample. Denoising is performed on each sample sub-speech spectrogram, and the predicted denoised speech output by the speech processing model is obtained according to the processing result. The speech processing model is trained according to the denoised speech label and the predicted denoised speech until a speech processing model that meets the training stop condition is obtained.

[0124] It is understood that the training process of the speech processing model is consistent with the inference process of the speech processing model. In other words, the frequency domain conversion process, partitioning process, and denoising process described above are consistent with the frequency domain conversion process, partitioning process, and denoising process during the inference process of the speech processing model. Please refer to the description of the inference process of the speech processing model for details.

[0125] In summary, by training the speech processing model, the speech processing model can learn the denoising ability during the training process, ensuring the denoising of the processed speech in subsequent applications.

[0126] Furthermore, in order to ensure that the speech processing model can eliminate environmental noise and reverberation to a large extent without damaging the speech signal, the model loss values ​​calculated using multiple loss calculation rules can be used for model training. The specific implementation method is as follows:

[0127] The step of training the speech processing model according to the denoised speech label and the predicted denoised speech until a speech processing model that satisfies a training stop condition is obtained includes:

[0128] Calculating the denoised speech label and the predicted denoised speech according to at least one loss calculation rule to obtain a model loss value;

[0129] The speech processing model is trained according to the model loss value until a speech processing model that meets the training stop condition is obtained.

[0130] The loss calculation rule can be understood as the rule for calculating the model loss value, and the model loss value can be understood as the loss function used for model training.

[0131] Specifically, when the speech processing model is trained according to the model loss value, each network layer (i.e., each unit) included in the speech processing model can be trained, that is, the parameters of each network layer included in the speech processing model can be adjusted according to the model loss value.

[0132] In one embodiment of the present specification, the at least one loss calculation rule includes a rule for calculating speech resolution loss, a rule for calculating time domain waveform loss, and a rule for calculating speech signal distortion loss. The model loss values ​​calculated by the rule for calculating speech resolution loss, the rule for calculating time domain waveform loss, and the rule for calculating speech signal distortion loss are different.

[0133] Based on this, the denoised speech labels and predicted denoised speech can be calculated according to the rules for calculating speech resolution loss to obtain a first model loss value. The denoised speech labels and predicted denoised speech can be calculated according to the rules for calculating time domain waveform loss to obtain a second model loss value. The denoised speech labels and predicted denoised speech can be calculated according to the rules for calculating speech signal distortion loss to obtain a third model loss value. The first model loss value can be understood as speech resolution loss, the second model loss value can be understood as time domain waveform loss, and the third model loss value can be understood as speech signal distortion loss. In addition, the speech processing model is trained according to the first model loss value, the second model loss value, and the third model loss value until a speech processing model that meets the training stop condition is obtained.

[0134] In one embodiment of the present specification, a target model loss value can be calculated based on the first model loss value, the second model loss value, and the third model loss value, and the speech processing model can be trained based on the target model loss value.

[0135] In addition, the model loss value may be calculated according to one of the loss calculation rules, or according to any two loss calculation rules, or according to three loss calculation rules. This specification does not limit this.

[0136] In practical applications, the loss function calculated according to the rules for calculating speech resolution loss is shown in the following formula (1). This loss function can be a multi-resolution STFT loss function, which can effectively capture the time-frequency distribution of the real waveform. The loss function calculated according to the rules for calculating time domain waveform loss is shown in the following formula (2). This loss function can be a time domain waveform loss function, which can calculate the distance between the real waveform and the generated waveform. The loss function calculated according to the rules for calculating speech signal distortion loss is shown in the following formula (3). It can calculate the distortion ratio of the SDR signal. SDR is an indicator of audio quality. It can calculate the ratio between the original signal energy and the distortion (that is, the energy difference between the original signal and the generated signal). It can be understood that the larger the SDR value, the less distortion, that is, the generated audio signal is closer to the real audio signal.

[0137]

[0138] L time =E (x，s) [|xs|] (2)

[0139]

[0140] Among them, L sc Can be used to express spectral convergence loss, L mag It can be used to represent the logarithmic STFT magnitude spectrum loss. x can be used to represent the speech signal output after denoising by the speech processing model, s can be used to represent the denoised speech label (i.e., the clean speech signal) used to train the speech processing model, E can be used to represent the expectation, r can be used to represent the different resolutions set, and R can be used to represent the total number of different resolutions set.

[0141] In summary, by combining the above multiple loss calculation rules to train the speech processing model, the speech denoising performance of the speech processing model can be guaranteed, making the speech signal predicted by the speech processing model in subsequent applications closer to the real speech signal.

[0142] Step 206: Divide the speech spectrogram according to a preset bandwidth to obtain a plurality of sub-speech spectrograms corresponding to the speech to be processed.

[0143] Specifically, after performing frequency domain conversion processing on the speech to be processed to obtain a speech spectrogram of the speech to be processed, the speech spectrogram can be divided according to a preset bandwidth to obtain multiple sub-speech spectrograms corresponding to the speech to be processed.

[0144] In practical applications, the speech processing model includes a frequency band division unit, which can be used to divide the speech spectrum of the speech to be processed. Figure X The spectrum is divided into a series of sub-band spectrograms (ie, sub-speech spectrograms) with a predefined bandwidth (ie, preset bandwidth).

[0145] For example, the speech spectrogram of the speech to be processed can be divided into 41 sub-speech spectrograms according to the preset bandwidth (i.e., band_widths) equal to [5, 6, 6, 6, 6, 6, 6, 6, 6, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 24, 24, 24, 24, 24, 24, 24, 42, 42, 42, 42, 42, 42, 82, 82, 130], where the sum of the preset bandwidths is F.

[0146] The preset bandwidth may be related to the voice type of the speech to be processed. Specifically, before dividing the speech spectrogram according to the preset bandwidth and obtaining multiple sub-speech spectrograms corresponding to the speech to be processed, the method further includes:

[0147] Determining the speech type of the speech to be processed;

[0148] According to the voice type, a preset bandwidth corresponding to the voice type is determined.

[0149] The speech types of the speech to be processed may include speech types, musical speech types, and the like. It is understood that for speech types, the focus is usually on the clarity of the speech to be processed, that is, whether the user can hear it clearly. Therefore, for speech types, the focus is on the low-frequency signals of the speech to be processed. For musical speech types, however, the focus is on losslessness of the speech to be processed. Therefore, both the high-frequency and low-frequency signals of the speech to be processed need to be considered.

[0150] Based on this, the voice type of the voice to be processed can be determined, and the corresponding preset bandwidth can be determined according to the voice type.

[0151] For example, for the speech type of speech to be processed, when the sub-band division is performed according to the preset bandwidth, the density of the low-frequency part of the speech to be processed can be set relatively high, while for the musical speech type of speech to be processed, when the sub-band division is performed according to the preset bandwidth, the high-frequency part and the low-frequency part of the speech to be processed can be evenly divided.

[0152] In summary, by adjusting the preset bandwidth accordingly according to different speech types, targeted processing of different types of speech to be processed is achieved, which facilitates achieving optimized denoising performance on specific types of speech to be processed.

[0153] Step 208: performing denoising processing on each of the plurality of sub-speech spectrograms, and obtaining a denoised target speech according to the processing result.

[0154] Specifically, after the speech spectrogram of the speech to be processed is divided into a plurality of sub-speech spectrograms according to a preset bandwidth, denoising processing can be performed on each sub-speech spectrogram, and the denoised target speech can be obtained according to the processing result.

[0155] In a specific implementation, performing denoising on each of the multiple sub-speech spectrograms and obtaining the denoised target speech according to the processing result includes:

[0156] Processing each of the plurality of sub-speech spectrograms to obtain a speech spectrogram mask;

[0157] The speech spectrogram is subjected to denoising processing according to the speech spectrogram mask, and the denoised target speech is obtained according to the processing result.

[0158] The speech spectrogram mask can be used to describe the noise level of the speech signal. If the speech signal is noise-free, the value of the speech spectrogram mask is 1; if the speech signal is noisy, the value of the speech spectrogram mask is 0. Based on the speech spectrogram mask, the complex-valued spectrogram obtained after Fourier transform (i.e., the speech spectrogram) can be denoised to obtain the full-band spectrogram of the speech to be processed, thereby obtaining the denoised target speech.

[0159] Based on this, the speech processing model can be used to process each sub-speech spectrogram in multiple sub-speech spectrograms to obtain the speech spectrogram mask of the speech to be processed, and the speech spectrogram can be denoised according to the speech spectrogram mask, and the denoised target speech can be obtained according to the processing results.

[0160] In practical applications, the speech spectrogram mask may be a full-band complex-valued TF mask.

[0161] In summary, by calculating the speech spectrogram mask, the subsequent denoising processing of the speech spectrogram is realized, thereby achieving denoising of the processed speech.

[0162] Furthermore, processing each of the multiple sub-speech spectrograms can be understood as performing feature processing and mask processing on each of the multiple sub-speech spectrograms. The specific implementation method is as follows: processing each of the multiple sub-speech spectrograms to obtain a speech spectrogram mask includes:

[0163] Performing feature processing on each of the multiple sub-speech spectrograms to obtain speech spectrogram features;

[0164] Processing the speech spectrogram features according to at least two dimensions to obtain target sub-speech spectrogram features corresponding to each sub-speech spectrogram;

[0165] Masking is performed on the target sub-speech spectrogram features corresponding to each sub-speech spectrogram to obtain a speech spectrogram mask.

[0166] Among them, the speech spectrogram feature can be understood as the full-band feature tensor of the speech to be processed.

[0167] Based on this, feature processing can be performed on each of the multiple sub-speech spectrograms to obtain the speech spectrogram features of the speech to be processed, and the speech spectrogram features can be processed according to at least two dimensions to obtain the target sub-speech spectrogram features corresponding to each sub-speech spectrogram, and mask processing can be performed on the target sub-speech spectrogram features corresponding to each sub-speech spectrogram to obtain the speech spectrogram mask of the speech to be processed.

[0168] It is understandable that after masking the target sub-speech spectrogram features corresponding to each sub-speech spectrogram, the obtained results can also be merged to obtain the speech spectrogram features of the speech to be processed. The specific implementation method is as follows: the feature processing of each sub-speech spectrogram in the multiple sub-speech spectrograms to obtain the speech spectrogram features includes:

[0169] Performing feature extraction on each of the multiple sub-speech spectrograms to obtain initial sub-speech spectrogram features corresponding to each sub-speech spectrogram;

[0170] The initial sub-speech spectrogram features corresponding to the sub-speech spectrograms are merged to obtain a speech spectrogram feature.

[0171] The initial sub-speech spectrogram features can be understood as real-valued sub-band features of each sub-speech spectrogram.

[0172] Specifically, the frequency band division unit in the speech processing model can be used to extract features from each sub-speech spectrogram, generate initial sub-speech spectrogram features corresponding to each sub-speech spectrogram, and merge the initial sub-speech spectrogram features corresponding to multiple sub-speech spectrograms to generate a full-band feature tensor of the speech to be processed.

[0173] In practical applications, the frequency band division unit in the speech processing model may include a normalized network layer (Norm) and a fully connected network layer (FC). Specifically, the real and imaginary parts of each sub-speech spectrogram may be connected and input into the normalized network layer and the fully connected layer to generate the real-valued sub-band features of each sub-speech spectrogram, and the real-valued sub-band features of all sub-speech spectrograms are merged to generate the transformed full-band feature tensor Z. At this time, the dimension of the speech to be processed changes from (B, F, T, 2) to (B, T, K, N), where K is the number of sub-speech spectrograms and N is the sub-band feature dimension. The specific implementation method is as follows: the speech processing model includes a normalized network layer and a fully connected network layer; the feature extraction of each sub-speech spectrogram in the multiple sub-speech spectrograms to obtain the initial sub-speech spectrogram features corresponding to each sub-speech spectrogram includes:

[0174] The normalized network layer and the fully connected network layer are used to perform feature extraction on each of the multiple sub-speech spectrograms to obtain initial sub-speech spectrogram features corresponding to each sub-speech spectrogram.

[0175] In summary, by generating the initial sub-speech spectrogram features corresponding to each sub-speech spectrogram, it is convenient to subsequently generate a full-band feature tensor and perform sequence encoding on the full-band feature tensor, thereby further realizing subsequent denoising.

[0176] Furthermore, since the width of each sub-speech spectrogram is different, feature processing can be performed on each sub-speech spectrogram through the corresponding normalization network layer and fully connected network layer. The specific implementation method is as follows:

[0177] The utilizing the normalized network layer and the fully connected network layer to extract features from each of the multiple sub-speech spectrograms comprises:

[0178] In the normalized network layer and the fully connected network layer, determining a sub-normalized network layer and a sub-fully connected network layer corresponding to each of the multiple sub-speech spectrograms;

[0179] The sub-normalized network layer and the sub-fully connected network layer corresponding to each sub-speech spectrogram are used to perform feature extraction on each sub-speech spectrogram to obtain initial sub-speech spectrogram features corresponding to each sub-speech spectrogram.

[0180] Specifically, the width of each sub-speech spectrogram can be determined in the normalized network layer and the fully connected network layer of the speech processing model, and the sub-normalized network layer and the sub-fully connected network layer corresponding to each sub-speech spectrogram can be determined based on the width. The sub-normalized network layer and the sub-fully connected network layer corresponding to each sub-speech spectrogram can be used to extract features of each sub-speech spectrogram to obtain the initial sub-speech spectrogram features corresponding to each sub-speech spectrogram.

[0181] For example, the sub-speech spectrum can be Figure 1 The width of the sub-speech spectrum is determined Figure 1 The corresponding sub-normalized network layer 1 and sub-fully connected network layer 1 are used to perform the sub-speech spectrum Figure 1 Perform feature processing to obtain the sub-speech spectrum Figure 1 The corresponding initial sub-speech spectrogram feature is 1. Similarly, the above operation is performed on each sub-speech spectrogram to obtain the initial sub-speech spectrogram features corresponding to all sub-speech spectrograms.

[0182] In summary, by setting corresponding normalized network layers and fully connected network layers for each sub-speech spectrogram of different widths, personalized processing of different bandwidths is taken into account to further ensure the denoising performance of the speech processing model.

[0183] Specifically, the processing of the speech spectrogram features according to at least two dimensions to obtain target sub-speech spectrogram features corresponding to each sub-speech spectrogram includes:

[0184] Encoding the speech spectrogram features according to the time dimension to obtain first sub-speech spectrogram features corresponding to each sub-speech spectrogram;

[0185] Encoding the speech spectrogram features according to the frequency domain dimension to obtain second sub-speech spectrogram features corresponding to each sub-speech spectrogram;

[0186] According to the first sub-speech spectrogram feature and the second sub-speech spectrogram feature, target sub-speech spectrogram features corresponding to the respective sub-speech spectrograms are determined.

[0187] Specifically, the speech processing model further includes a sequence encoding unit, which can encode the speech spectrogram features according to the time dimension and the frequency domain dimension, thereby obtaining target sub-speech spectrogram features corresponding to each sub-speech spectrogram.

[0188] In specific implementation, each sub-speech spectrogram feature included in the speech spectrogram feature can be encoded according to the time dimension (i.e., sequence-level modeling) to obtain the first sub-speech spectrogram feature corresponding to each sub-speech spectrogram; each sub-speech spectrogram feature included in the speech spectrogram feature can be encoded according to the frequency domain dimension (i.e., sub-band-level modeling) to obtain the second sub-speech spectrogram feature corresponding to each sub-speech spectrogram; the first sub-speech spectrogram feature and the second sub-speech spectrogram feature can be fused to obtain the target sub-speech spectrogram feature corresponding to each sub-speech spectrogram.

[0189] In practical applications, frequency-domain modeling can be performed on the time dimension T. To save model size, parallel processing across subbands is allowed, where K sub-speech spectrogram features (i.e., subband features) can share the same RNN network layer. In other words, multiple sub-speech spectrogram features can be encoded in parallel along the time dimension using the same RNN network layer. Subband-level modeling can be performed on the sub-band dimension K. As can be understood, the intra-band features of each sub-band have dependencies. The RNN network layer can include a normalization module, an LSTM network layer, and an FC network layer. Furthermore, a residual connection is added between the input and output of the FC network layer to pass information such as phase and alignment, ensuring that low-dimensional features of the input audio are not lost. Furthermore, the speech processing model can include multiple RNN network layers, thereby building a deeper architecture. By modeling temporal correlations, the speech processing model's ability to learn temporal correlations and enhance feature correlations is enhanced.

[0190] Furthermore, after obtaining the target sub-speech spectrogram features corresponding to each sub-speech spectrogram, the target sub-speech spectrogram features corresponding to multiple sub-speech spectrograms can be merged to obtain a merged target speech spectrogram feature. This facilitates the subsequent input of the target speech spectrogram features into the mask estimation module for mask estimation, effectively utilizing the information between the target sub-speech spectrogram features corresponding to each sub-speech spectrogram, and thus enabling parallel processing of each target sub-speech spectrogram feature included in the full band of target speech spectrogram features, thereby improving the model training and deduction speeds.

[0191] In practical applications, masking the target sub-speech spectrogram features corresponding to each sub-speech spectrogram to obtain a speech spectrogram mask includes:

[0192] Performing mask estimation on target sub-speech spectrogram features corresponding to each sub-speech spectrogram to obtain sub-speech spectrogram masks corresponding to each sub-speech spectrogram;

[0193] The sub-speech spectrogram masks corresponding to the sub-speech spectrograms are merged to obtain a speech spectrogram mask.

[0194] The sub-speech spectrogram mask can be understood as the TF mask of each sub-speech spectrogram.

[0195] Specifically, the speech processing model also includes a mask estimation unit, which can be used to perform mask estimation on the target sub-speech spectrogram features corresponding to each sub-speech spectrogram to obtain the sub-speech spectrogram mask corresponding to each sub-speech spectrogram, and merge the sub-speech spectrogram masks corresponding to each sub-speech spectrogram to obtain the speech spectrogram mask of the speech to be processed.

[0196] In practical applications, the mask estimation unit may include a normalized network layer and a multi-layer perceptron, which may be used to perform mask estimation on the target sub-speech spectrogram features corresponding to each sub-speech spectrogram to obtain the sub-speech spectrogram mask corresponding to each sub-speech spectrogram.

[0197] It can be understood that each sub-speech spectrogram can correspond to a sub-normalized network layer and a sub-multi-layer perceptron. The sub-normalized network layer and sub-multi-layer perceptron corresponding to each sub-speech spectrogram can be determined, and the corresponding sub-normalized network layer and sub-multi-layer perceptron can be used to perform mask estimation on the target sub-speech spectrogram features corresponding to each sub-speech spectrogram, thereby obtaining the sub-speech spectrogram mask corresponding to each sub-speech spectrogram.

[0198] Specifically, performing denoising processing on the speech spectrogram according to the speech spectrogram mask, and obtaining the denoised target speech according to the processing result, includes:

[0199] Obtaining a denoised speech spectrogram according to the speech spectrogram mask and the speech spectrogram;

[0200] The denoised speech spectrogram is subjected to time domain conversion processing to obtain the denoised target speech.

[0201] Specifically, the speech spectrogram mask and the speech spectrogram can be multiplied to obtain a denoised speech spectrogram, and the denoised speech spectrogram can be inverse Fourier transformed to convert it into a time domain representation (ie, a waveform), thereby obtaining the denoised target speech.

[0202] In one embodiment of the present specification, an inverse short-time Fourier transform can be performed on the denoised speech spectrogram to convert it into a time domain representation. In practical applications, the speech processing model can further include an inverse Fourier transform unit, which can be used to perform a Fourier transform on the denoised speech spectrogram to achieve time domain conversion of the denoised speech spectrogram.

[0203] Refer to Figure 3(a), which is a structural diagram of a speech processing model in a speech processing method provided by an embodiment of the present specification. As shown in Figure 3(a), the speech processing model may include a frequency band division unit 302, a sequence encoding unit 304 and a mask estimation unit 306. In addition, the speech processing model may also include a Fourier transform unit, which converts the speech to be processed into a speech spectrum graph. Specifically, after the speech to be processed is input into the preprocessing model, the speech to be processed is Fourier transformed, and the speech to be processed is converted into a speech spectrum graph. The frequency band division unit 302 can divide the speech spectrum graph of the speech to be processed. Specifically, Figure 3(b) is a processing flow diagram of the frequency band division unit in the speech processing model provided by an embodiment of the present specification, which can divide the speech spectrum graph of the speech to be processed. Figure X The frequency band division unit 302 may include a normalization network layer (Norm) and a fully connected network layer (FC). Specifically, the real part and the imaginary part of each sub-speech spectrogram may be connected and input into the normalization network layer and the fully connected layer to generate the sub-speech spectrogram features (i.e., real-valued sub-band features) of each sub-speech spectrogram, and the real-valued sub-band features of all sub-speech spectrograms may be merged to generate the transformed speech spectrogram features Z (i.e., full-band feature tensor).

[0204] FIG3( c ) is a schematic diagram of the processing flow of the sequence encoding unit in the speech processing model provided by one embodiment of the present specification. Referring to FIG3( c ), the sequence encoding unit 304 can encode the speech spectrogram feature Z according to the time dimension and the frequency domain dimension. In the specific encoding process, the speech spectrogram feature can be split into sub-speech spectrogram features corresponding to each sub-speech spectrogram and encoded, thereby obtaining the target sub-speech spectrogram feature corresponding to each sub-speech spectrogram. The target sub-speech spectrogram features corresponding to each sub-speech spectrogram are then merged to obtain the merged target speech spectrogram feature Q. The sequence encoding unit 304 can include multiple RNN network structures, each of which includes a normalized network layer, a long short-term memory network, and a fully connected network layer. The speech spectrogram feature Z can be sequentially input into the multiple RNN network structures, and the RNN network structures encode the speech spectrogram feature Z according to the time dimension and the frequency domain dimension. For example, taking the illustrated sequence encoding unit 304 as an example, which includes two RNN network structures, the speech spectrogram feature Z can be input into the first RNN network structure, and pass through the normalization network layer, the long short-term memory network, and the fully connected network layer in sequence. In order to prevent information loss, the features output by the first RNN network structure and the speech spectrogram feature Z can be input into the second RNN network structure, and then pass through the normalization network layer, the long short-term memory network, and the fully connected network layer in sequence, thereby outputting the target sub-speech spectrogram features corresponding to each sub-speech spectrogram. Specifically, the speech spectrogram features can be encoded according to the time dimension to obtain the first sub-speech spectrogram features corresponding to each sub-speech spectrogram; the speech spectrogram features can be encoded according to the frequency domain dimension to obtain the second sub-speech spectrogram features corresponding to each sub-speech spectrogram; the first sub-speech spectrogram features and the second sub-speech spectrogram features can be fused to obtain the target sub-speech spectrogram features corresponding to each sub-speech spectrogram.

[0205] Figure 3 (d) is a schematic diagram of the processing flow of the mask estimation unit in the speech processing model provided by an embodiment of this specification. Referring to Figure 3 (d), the mask estimation unit 306 may include multiple normalized network layers and multiple multilayer perceptrons. It can be understood that in the multiple normalized network layers and multiple multilayer perceptrons, each sub-speech spectrogram corresponds to each normalized network layer + multilayer perceptron. This is to achieve personalized processing of the target sub-speech spectrogram features at different bandwidths. The mask estimation unit 306 can perform mask estimation on the merged target speech spectrogram feature Q to obtain the sub-speech spectrogram mask (i.e., TF mask) corresponding to each sub-speech spectrogram, and merge each sub-speech spectrogram mask to obtain the speech spectrogram mask m (i.e., full-band complex-valued TF mask). During specific implementation, the mask estimation unit 306 can perform mask estimation on the target sub-speech spectrogram feature corresponding to each sub-speech spectrogram included in the target speech spectrogram feature, thereby obtaining the sub-speech spectrogram mask (i.e., TF mask) corresponding to each sub-speech spectrogram.

[0206] Afterwards, the speech spectrogram mask m can be inverse Fourier transformed to convert it into a time domain representation, thereby obtaining the denoised target speech.

[0207] It can be understood that in one embodiment of the present specification, the outputs of the frequency band division unit 302, the sequence encoding unit 304 and the mask estimation unit 306 are all full-band, thereby realizing parallel processing between multiple sub-bands included in the full-band, thereby improving the training speed and the deduction speed.

[0208] In practical applications, the speech processing methods proposed in the embodiments of this specification can be evaluated using a variety of metrics. For example, the evaluation can be performed using the following metrics: The metric can be a mean opinion score (MAS), where a large number of listeners evaluate the quality of sentences read aloud by male and female speakers. Listeners can score each sentence according to the following criteria: 1 for very poor, 2 for poor, 3 for fair, 4 for good, and 5 for very good. The average score from each listener is calculated, with a higher score indicating better speech quality. Another metric can be a perceptual speech quality assessment, which uses the PESQ algorithm to calculate a predicted value for speech quality assessment based on the attenuated signal and the original reference signal. A higher score indicates better speech quality. Another metric can be a short-term target intelligibility measurement (STEM), where a larger value indicates higher speech intelligibility and better speech quality. The following table compares speech quality assessment metrics for speech denoising using different algorithms.

[0209]

[0210] It can be seen that the speech processing method proposed in the embodiments of this specification scores better than other solutions in multiple measurement indicators, and significantly improves the quality and intelligibility of the enhanced speech.

[0211] Furthermore, the speech processing method proposed in the embodiments of this specification is used to denoise noisy speech in real scenes, and the denoised speech is compared. The real scenes include but are not limited to offices, restaurants, streets, roads and corridors. Figure 4 , Figure 4 FIG2 shows a schematic diagram of the comparison results of speech processed by a speech processing method provided according to an embodiment of this specification. Figure 4 As shown, according to the comparison of noisy speech, real clean speech, and the speech obtained after denoising the noisy speech using the speech processing method proposed in the embodiment of the specification, it can be seen that the spectral structure of the denoised speech remains relatively complete, the high-frequency part is not damaged, and the noise reduction and reverberation reduction capabilities are strong.

[0212] In summary, in the above method, when denoising the speech to be processed, a speech processing model can be called, in which the speech spectrogram of the speech to be processed is determined, and the speech spectrogram is divided into multiple sub-speech spectrograms according to a preset bandwidth, and each sub-speech spectrogram is denoised, so as to obtain the denoised target speech according to the processing result. In the process of denoising the speech to be processed, the full-band sub-speech spectrogram is taken into account, and the integrity and smoothness of the full-band speech signal are guaranteed while denoising the speech to be processed, and the elimination of environmental noise and reverberation in the speech to be processed is guaranteed, thereby ensuring the high quality of the denoised target speech.

[0213] The following combined Figure 5 , taking the application of the speech processing method provided in this specification in speech denoising as an example, the speech processing method is further explained. Figure 5 A flowchart of a speech processing method provided in one embodiment of the present specification is shown, which specifically includes the following steps.

[0214] Step 502: Short-time Fourier transform.

[0215] Specifically, a short-time Fourier transform (SFT) can be performed on the noisy speech signal (i.e., the speech to be processed) input to the speech processing model to obtain a frequency domain representation of the noisy speech signal (i.e., a complex-valued spectrogram). Here, the dimension of the speech to be processed is changed from (B, t) to (B, F, T, 2).

[0216] Step 504: Frequency band division.

[0217] Specifically, based on the preset bandwidth, the complex-valued spectrogram is divided into multiple subband spectrograms, and the real and imaginary parts of each subband spectrogram are connected and passed to the normalization network layer and the fully connected network layer to generate real-valued subband features for each subband spectrogram. The real-valued subband features of all subband spectrograms are merged to generate a transformed full-band feature tensor. Here, the dimension of the speech to be processed is changed to (B, T, K, N), where K is the number of sub-speech spectrograms (i.e., subband spectrograms) and N is the subband feature dimension.

[0218] Step 506: Sequence encoding.

[0219] Specifically, the full-band feature tensor is sequence-encoded according to the time dimension and the frequency domain dimension. Here, the dimension of the speech to be processed is changed to (B, N, K, T).

[0220] Step 508: Mask estimation.

[0221] Specifically, the subband feature tensor of each subband spectrogram included in the full-band feature tensor is passed to the normalization network layer and the multilayer perceptron to generate a TF mask corresponding to each subband spectrogram. The TF masks corresponding to all subband spectrograms are then merged to generate a full-band complex-valued TF mask. Here, the dimension of the speech to be processed is changed to (B, F, T, 2).

[0222] Step 510: Inverse short-time Fourier transform.

[0223] Specifically, the full-band complex-valued TF mask and the complex-valued spectrogram are multiplied together to obtain a full-band spectrogram. This full-band spectrogram is then inversely short-time Fourier transformed into a time-domain representation, and the denoised speech signal is determined based on this time-domain representation. Here, the dimension of the speech to be processed is changed to (B, t).

[0224] In summary, in the above method, when denoising the speech to be processed, a speech processing model can be called, in which the speech spectrogram of the speech to be processed is determined, and the speech spectrogram is divided into multiple sub-speech spectrograms according to a preset bandwidth, and each sub-speech spectrogram is denoised, so as to obtain the denoised target speech according to the processing result. In the process of denoising the speech to be processed, the full-band sub-speech spectrogram is taken into account, and the integrity and smoothness of the full-band speech signal are guaranteed while denoising the speech to be processed, and the elimination of environmental noise and reverberation in the speech to be processed is guaranteed, thereby ensuring the high quality of the denoised target speech.

[0225] Corresponding to the above method embodiment, this specification also provides a speech processing device embodiment, Figure 6 FIG. 1 shows a schematic diagram of the structure of a speech processing device provided by an embodiment of this specification. Figure 6As shown, the device includes:

[0226] A determination module 602 is configured to determine a speech to be processed;

[0227] An input module 604 is configured to input the speech to be processed into a speech processing model, and in the speech processing model, perform frequency domain conversion processing on the speech to be processed to obtain a speech spectrogram of the speech to be processed;

[0228] A division module 606 is configured to divide the speech spectrogram according to a preset bandwidth to obtain a plurality of sub-speech spectrograms corresponding to the speech to be processed;

[0229] The denoising module 608 is configured to perform denoising processing on each of the multiple sub-speech spectrograms and obtain a denoised target speech according to the processing result.

[0230] In an optional embodiment, the denoising module 608 is further configured to:

[0231] Processing each of the plurality of sub-speech spectrograms to obtain a speech spectrogram mask;

[0232] The speech spectrogram is subjected to denoising processing according to the speech spectrogram mask, and the denoised target speech is obtained according to the processing result.

[0233] In an optional embodiment, the denoising module 608 is further configured to:

[0234] Performing feature processing on each of the multiple sub-speech spectrograms to obtain speech spectrogram features;

[0235] Processing the speech spectrogram features according to at least two dimensions to obtain target sub-speech spectrogram features corresponding to each sub-speech spectrogram;

[0236] Masking is performed on the target sub-speech spectrogram features corresponding to each sub-speech spectrogram to obtain a speech spectrogram mask.

[0237] In an optional embodiment, the denoising module 608 is further configured to:

[0238] Performing feature extraction on each of the multiple sub-speech spectrograms to obtain initial sub-speech spectrogram features corresponding to each sub-speech spectrogram;

[0239] The initial sub-speech spectrogram features corresponding to the sub-speech spectrograms are merged to obtain a speech spectrogram feature.

[0240] In an optional embodiment, the denoising module 608 is further configured to:

[0241] The normalized network layer and the fully connected network layer of the speech processing model are used to extract features of each sub-speech spectrogram in the multiple sub-speech spectrograms to obtain initial sub-speech spectrogram features corresponding to each sub-speech spectrogram.

[0242] In an optional embodiment, the speech processing model includes a normalized network layer and a fully connected network layer;

[0243] The denoising module 608 is further configured to:

[0244] In the normalized network layer and the fully connected network layer, determining a sub-normalized network layer and a sub-fully connected network layer corresponding to each of the multiple sub-speech spectrograms;

[0245] The sub-normalized network layer and the sub-fully connected network layer corresponding to each sub-speech spectrogram are used to perform feature extraction on each sub-speech spectrogram to obtain initial sub-speech spectrogram features corresponding to each sub-speech spectrogram.

[0246] In an optional embodiment, the denoising module 608 is further configured to:

[0247] Performing mask estimation on target sub-speech spectrogram features corresponding to each sub-speech spectrogram to obtain sub-speech spectrogram masks corresponding to each sub-speech spectrogram;

[0248] The sub-speech spectrogram masks corresponding to the sub-speech spectrograms are merged to obtain a speech spectrogram mask.

[0249] In an optional embodiment, the denoising module 608 is further configured to:

[0250] Obtaining a denoised speech spectrogram according to the speech spectrogram mask and the speech spectrogram;

[0251] The denoised speech spectrogram is subjected to time domain conversion processing to obtain the denoised target speech.

[0252] In an optional embodiment, the denoising module 608 is further configured to:

[0253] Encoding the speech spectrogram features according to the time dimension to obtain first sub-speech spectrogram features corresponding to each sub-speech spectrogram;

[0254] Encoding the speech spectrogram features according to the frequency domain dimension to obtain second sub-speech spectrogram features corresponding to each sub-speech spectrogram;

[0255] According to the first sub-speech spectrogram feature and the second sub-speech spectrogram feature, target sub-speech spectrogram features corresponding to the respective sub-speech spectrograms are determined.

[0256] In an optional embodiment, the dividing module 606 is further configured to:

[0257] Determining the speech type of the speech to be processed;

[0258] According to the voice type, a preset bandwidth corresponding to the voice type is determined.

[0259] In an optional embodiment, the apparatus further includes a training module configured to:

[0260] Determining a noisy speech sample and a denoised speech label corresponding to the noisy speech sample;

[0261] Inputting the noisy speech sample into a speech processing model, and performing frequency domain conversion processing on the noisy speech sample in the speech processing model to obtain a sample speech spectrogram of the noisy speech sample;

[0262] Dividing the sample speech spectrogram according to a preset bandwidth to obtain a plurality of sample sub-speech spectrograms corresponding to the noisy speech sample;

[0263] Performing denoising processing on each of the plurality of sample sub-speech spectrograms and obtaining a predicted denoised speech according to the processing result;

[0264] The speech processing model is trained according to the denoised speech label and the predicted denoised speech until a speech processing model that meets a training stop condition is obtained.

[0265] In an optional embodiment, the training module is further configured to:

[0266] Calculating the denoised speech label and the predicted denoised speech according to at least one loss calculation rule to obtain a model loss value;

[0267] The speech processing model is trained according to the model loss value until a speech processing model that meets the training stop condition is obtained.

[0268] In an optional embodiment, the at least one loss calculation rule includes a rule for calculating speech resolution loss, a rule for calculating time domain waveform loss, and a rule for calculating speech signal distortion loss, and the model loss values ​​calculated by the rule for calculating speech resolution loss, the rule for calculating time domain waveform loss, and the rule for calculating speech signal distortion loss are different.

[0269] In summary, in the above-mentioned device, when denoising the speech to be processed, a speech processing model can be called, in which the speech spectrogram of the speech to be processed is determined, and the speech spectrogram is divided into multiple sub-speech spectrograms according to a preset bandwidth, and each sub-speech spectrogram is denoised, so as to obtain the denoised target speech according to the processing result. In the process of denoising the speech to be processed, the full-band sub-speech spectrogram is taken into account, and the integrity and smoothness of the full-band speech signal are guaranteed while denoising the speech to be processed, and the elimination of environmental noise and reverberation present in the speech to be processed is guaranteed, thereby ensuring the high quality of the denoised target speech.

[0270] The above is a schematic diagram of a speech processing device according to this embodiment. It should be noted that the technical solution of the speech processing device and the technical solution of the speech processing method described above are based on the same concept. For details not described in detail in the technical solution of the speech processing device, please refer to the description of the technical solution of the speech processing method described above.

[0271] See also Figure 7 , Figure 7 A flowchart of a speech processing model training method provided according to an embodiment of this specification is shown, which specifically includes the following steps.

[0272] Step 702: Determine a noisy speech sample and a denoised speech label corresponding to the noisy speech sample;

[0273] Step 704: inputting the noisy speech sample into a speech processing model, and performing frequency domain conversion processing on the noisy speech sample in the speech processing model to obtain a sample speech spectrogram of the noisy speech sample;

[0274] Step 706: Divide the sample speech spectrogram according to a preset bandwidth to obtain a plurality of sample sub-speech spectrograms corresponding to the noisy speech sample;

[0275] Step 708: performing denoising processing on each of the plurality of sample sub-speech spectrograms, and obtaining a predicted denoised speech according to the processing result;

[0276] Step 710: Train the speech processing model according to the denoised speech label and the predicted denoised speech until a speech processing model that meets a training stop condition is obtained.

[0277] In an optional embodiment, the training of the speech processing model according to the denoised speech label and the predicted denoised speech until a speech processing model that satisfies a training stop condition is obtained includes:

[0278] Calculating the denoised speech label and the predicted denoised speech according to at least one loss calculation rule to obtain a model loss value;

[0279] The speech processing model is trained according to the model loss value until a speech processing model that meets the training stop condition is obtained.

[0280] In summary, in the above method, during the training of the speech processing model, a sample speech spectrogram of the noisy speech sample can be determined in the speech processing model, and the sample speech spectrogram is divided into multiple sample sub-speech spectrograms according to a preset bandwidth. Each sample sub-speech spectrogram is denoised, and predicted denoised speech is obtained based on the processing results. By taking into account the full-band sub-speech spectrograms, the integrity and fluency of the full-band speech signal are maintained while denoising the processed speech, and the ambient noise and reverberation present in the processed speech are eliminated, thereby ensuring the high quality of the denoised target speech.

[0281] The above is a schematic scheme of a speech processing model training method of this embodiment. It should be noted that the technical scheme of this speech processing model training method and the technical scheme of the above-mentioned speech processing method are based on the same concept. For details not described in detail in the technical scheme of the speech processing model training method, please refer to the description of the technical scheme of the above-mentioned speech processing method.

[0282] Corresponding to the above method embodiment, this specification also provides an embodiment of a speech processing model training device, Figure 8 FIG. 1 shows a schematic diagram of the structure of a speech processing model training device provided by an embodiment of this specification. Figure 8 As shown, the device includes:

[0283] A determination module 802 is configured to determine a noisy speech sample and a denoised speech label corresponding to the noisy speech sample;

[0284] An input module 804 is configured to input the noisy speech sample into a speech processing model, and in the speech processing model, perform frequency domain conversion processing on the noisy speech sample to obtain a sample speech spectrogram of the noisy speech sample;

[0285] A division module 806 is configured to divide the sample speech spectrogram according to a preset bandwidth to obtain a plurality of sample sub-speech spectrograms corresponding to the noisy speech sample;

[0286] A denoising module 808 is configured to perform denoising processing on each of the plurality of sample sub-speech spectrograms and obtain a predicted denoised speech according to the processing result;

[0287] The training module 810 is configured to train the speech processing model according to the denoised speech label and the predicted denoised speech until a speech processing model that meets the training stop condition is obtained.

[0288] In summary, in the above-mentioned device, during the training of the speech processing model, a sample speech spectrogram of the noisy speech sample can be determined in the speech processing model, and the sample speech spectrogram is divided into multiple sample sub-speech spectrograms according to a preset bandwidth. Each sample sub-speech spectrogram is denoised, thereby obtaining a predicted denoised speech based on the processing results. Taking into account the full-band sub-speech spectrograms, the integrity and smoothness of the full-band speech signal are maintained while denoising the processed speech, ensuring the elimination of environmental noise and reverberation present in the processed speech, thereby ensuring the high quality of the denoised target speech.

[0289] The above is a schematic diagram of a speech processing model training device according to this embodiment. It should be noted that the technical solution of the speech processing model training device and the technical solution of the speech processing method described above are based on the same concept. For details not described in detail in the technical solution of the speech processing model training method, please refer to the description of the technical solution of the speech processing method described above.

[0290] See also Figure 9 , Figure 9 A flowchart of a speech denoising method provided according to an embodiment of the present specification is shown, which is applied to a cloud-side device and specifically includes the following steps.

[0291] Step 902: receiving noisy speech sent by the client;

[0292] Step 904: inputting the noisy speech into a speech processing model, performing frequency domain conversion processing on the noisy speech in the speech processing model to obtain a speech spectrogram of the noisy speech;

[0293] Step 906: Divide the speech spectrogram according to a preset bandwidth to obtain a plurality of sub-speech spectrograms corresponding to the noisy speech;

[0294] Step 908: performing denoising processing on each of the plurality of sub-speech spectrograms, and obtaining a denoised target speech according to the processing result;

[0295] Step 910: Send the target voice to the client.

[0296] Specifically, the user can send the noisy speech that needs to be denoised to the cloud-side device through the client. The cloud-side device receives the noisy speech and inputs the noisy speech into the speech processing model. In the speech processing model, the noisy speech is converted into the frequency domain to obtain a speech spectrogram of the noisy speech. The speech spectrogram is divided according to the preset bandwidth to obtain multiple sub-speech spectrograms corresponding to the noisy speech, and denoising is performed on each of the multiple sub-speech spectrograms. According to the processing results, the denoised target speech is obtained, and the target speech is sent to the client. It is fed back to the user through the client to complete the denoising of the noisy speech for the user.

[0297] In summary, when denoising the noisy speech sent by the client, a speech processing model can be called, in which the speech spectrogram of the noisy speech is determined, and the speech spectrogram is divided into multiple sub-speech spectrograms according to a preset bandwidth. Denoising is performed on each sub-speech spectrogram, so that the denoised target speech is obtained based on the processing results, and the target speech is returned to the client. In the process of denoising the noisy speech, the full-band sub-speech spectrogram is taken into account. While denoising the noisy speech, the integrity and smoothness of the full-band speech signal are guaranteed, and the ambient noise and reverberation present in the noisy speech are eliminated, thereby ensuring the high quality of the denoised target speech.

[0298] The above is a schematic scheme of a speech denoising method of this embodiment. It should be noted that the technical scheme of this speech denoising method and the technical scheme of the above-mentioned speech processing method are based on the same concept. For details not described in detail in the technical scheme of the speech denoising method, please refer to the description of the technical scheme of the above-mentioned speech processing method.

[0299] Corresponding to the above method embodiment, this specification also provides a speech denoising device embodiment, which is applied to a cloud-side device. Figure 10 FIG1 shows a schematic diagram of the structure of a speech denoising device provided by an embodiment of this specification. Figure 10 As shown, the device includes:

[0300] The receiving module 1002 is configured to receive noisy speech sent by the client;

[0301] An input module 1004 is configured to input the noisy speech into a speech processing model, and in the speech processing model, perform frequency domain conversion processing on the noisy speech to obtain a speech spectrogram of the noisy speech;

[0302] A division module 1006 is configured to divide the speech spectrogram according to a preset bandwidth to obtain a plurality of sub-speech spectrograms corresponding to the noisy speech;

[0303] A denoising module 1008 is configured to perform denoising processing on each of the plurality of sub-speech spectrograms and obtain a denoised target speech according to the processing result;

[0304] The sending module 1010 is configured to send the target voice to the client.

[0305] In summary, when denoising the noisy speech sent by the client, a speech processing model can be called, in which the speech spectrogram of the noisy speech is determined, and the speech spectrogram is divided into multiple sub-speech spectrograms according to a preset bandwidth. Denoising is performed on each sub-speech spectrogram, so that the denoised target speech is obtained based on the processing results, and the target speech is returned to the client. In the process of denoising the noisy speech, the full-band sub-speech spectrogram is taken into account. While denoising the noisy speech, the integrity and smoothness of the full-band speech signal are guaranteed, and the ambient noise and reverberation present in the noisy speech are eliminated, thereby ensuring the high quality of the denoised target speech.

[0306] The above is a schematic diagram of a speech denoising device according to this embodiment. It should be noted that the technical solution of the speech denoising device and the technical solution of the aforementioned speech processing method are based on the same concept. For details not described in detail in the technical solution of the speech denoising device, please refer to the description of the technical solution of the aforementioned speech processing method.

[0307] See also Figure 11 , Figure 11 A flowchart of a speech recognition method provided according to an embodiment of the present specification is shown, which is applied to a cloud-side device and specifically includes the following steps.

[0308] Step 1102: Receive a user's voice signal;

[0309] Step 1104: inputting the speech signal into a speech processing model, and performing frequency domain conversion processing on the speech signal in the speech processing model to obtain a speech spectrogram of the speech signal;

[0310] Step 1106: Divide the speech spectrogram according to a preset bandwidth to obtain a plurality of sub-speech spectrograms corresponding to the speech signal;

[0311] Step 1108: performing denoising processing on each of the multiple sub-speech spectrograms, and obtaining a denoised speech signal according to the processing result;

[0312] Step 1110: Recognize the denoised voice signal and generate reply information based on the recognition result.

[0313] Specifically, the cloud-side device can be connected to the intelligent robot for communication. When the cloud-side device receives the voice signal emitted by the user when speaking, the voice signal is input into the voice processing model. In the voice processing model, the voice signal is converted into the frequency domain to obtain a voice spectrum of the voice signal. The voice spectrum is divided according to a preset bandwidth to obtain multiple sub-voice spectrums corresponding to the noisy voice, and each of the multiple sub-voice spectrums is denoised. According to the processing result, a denoised voice signal is obtained, the denoised voice signal is recognized, and a reply message is generated for the voice signal emitted by the user according to the recognition result.

[0314] Afterwards, the cloud-side device can send the reply information to the intelligent robot, which then feeds the reply information back to the user through a dialogue, thus achieving a dialogue interaction between the user and the intelligent robot.

[0315] In summary, when denoising a user's voice signal, a voice processing model can be called, in which the voice spectrogram of the voice signal is determined, and the voice spectrogram is divided into multiple sub-speech spectrograms according to a preset bandwidth. Each sub-speech spectrogram is denoised, thereby obtaining a denoised voice signal based on the processing result, and generating a reply message for the user's voice signal based on the recognition result. In the process of denoising noisy speech, the full-band sub-speech spectrogram is taken into account. While denoising the noisy speech, the integrity and smoothness of the full-band voice signal are guaranteed, and the ambient noise and reverberation present in the voice signal are eliminated, thereby ensuring the high quality of the denoised voice signal.

[0316] The above is a schematic scheme of a speech recognition method of this embodiment. It should be noted that the technical scheme of this speech recognition method and the technical scheme of the above-mentioned speech processing method are based on the same concept. For details not described in detail in the technical scheme of the speech recognition method, please refer to the description of the technical scheme of the above-mentioned speech processing method.

[0317] Corresponding to the above method embodiment, this specification also provides a speech recognition device embodiment, which is applied to a cloud-side device. Figure 12 FIG1 shows a schematic diagram of the structure of a speech recognition device provided by an embodiment of this specification. Figure 12 As shown, the device includes:

[0318] The receiving module 1202 is configured to receive a user's voice signal;

[0319] The input module 1204 is configured to input the speech signal into a speech processing model, and in the speech processing model, perform frequency domain conversion processing on the speech signal to obtain a speech spectrogram of the speech signal;

[0320] A division module 1206 is configured to divide the speech spectrogram according to a preset bandwidth to obtain a plurality of sub-speech spectrograms corresponding to the speech signal;

[0321] A denoising module 1208 is configured to perform denoising processing on each of the plurality of sub-speech spectrograms and obtain a denoised speech signal according to the processing result;

[0322] The recognition module 1210 is configured to recognize the denoised voice signal and generate reply information according to the recognition result.

[0323] In summary, when denoising a user's voice signal, a voice processing model can be called, in which the voice spectrogram of the voice signal is determined, and the voice spectrogram is divided into multiple sub-speech spectrograms according to a preset bandwidth. Each sub-speech spectrogram is denoised, thereby obtaining a denoised voice signal based on the processing result, and generating a reply message for the user's voice signal based on the recognition result. In the process of denoising noisy speech, the full-band sub-speech spectrogram is taken into account. While denoising the noisy speech, the integrity and smoothness of the full-band voice signal are guaranteed, and the ambient noise and reverberation present in the voice signal are eliminated, thereby ensuring the high quality of the denoised voice signal.

[0324] The above is a schematic diagram of a speech recognition device according to this embodiment. It should be noted that the technical solution of the speech recognition device and the technical solution of the speech processing method described above are of the same concept. For details not described in detail in the technical solution of the speech recognition device, please refer to the description of the technical solution of the speech processing method described above.

[0325] Figure 13 13 shows a block diagram of a computing device 1300 according to one embodiment of the present disclosure. Components of the computing device 1300 include, but are not limited to, a memory 1310 and a processor 1320. The processor 1320 is connected to the memory 1310 via a bus 1330, and a database 1350 is used to store data.

[0326] The computing device 1300 also includes an access device 1340 that enables the computing device 1300 to communicate via one or more networks 1360. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1340 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0327] In one embodiment of the present application, the above components of the computing device 1300 and Figure 13 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 13 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0328] Computing device 1300 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 1300 may also be a mobile or stationary server.

[0329] The processor 1320 is configured to execute the following computer program / instructions, which implement the steps of the above method when executed by the processor.

[0330] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the computing device embodiment is generally similar to the method embodiment, so the description is relatively simple. For relevant parts, refer to the description of the method embodiment.

[0331] An embodiment of the present specification further provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0332] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the computer-readable storage medium embodiment is generally similar to the method embodiment, so its description is relatively simple. For relevant portions, refer to the description of the method embodiment.

[0333] An embodiment of the present specification further provides a computer program product, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0334] The above is an illustrative solution of a computer program product of this embodiment. It should be noted that the technical solution of the computer program product and the technical solution of the above method belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the above method.

[0335] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0336] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0337] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0338] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0339] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A speech processing method, comprising: Determine the speech to be processed; Inputting the speech to be processed into a speech processing model, and performing frequency domain conversion processing on the speech to be processed in the speech processing model to obtain a speech spectrogram of the speech to be processed; Dividing the speech spectrogram according to a preset bandwidth to obtain a plurality of sub-speech spectrograms corresponding to the speech to be processed; Denoising is performed on each of the multiple sub-speech spectrograms, and a denoised target speech is obtained according to the processing result.

2. The speech processing method according to claim 1, wherein the denoising process is performed on each of the plurality of sub-speech spectrograms and the denoised target speech is obtained according to the processing result, comprising: Processing each of the plurality of sub-speech spectrograms to obtain a speech spectrogram mask; The speech spectrogram is subjected to denoising processing according to the speech spectrogram mask, and the denoised target speech is obtained according to the processing result.

3. The speech processing method according to claim 2, wherein the processing of each of the plurality of sub-speech spectrograms to obtain a speech spectrogram mask comprises: Performing feature processing on each of the multiple sub-speech spectrograms to obtain speech spectrogram features; Processing the speech spectrogram features according to at least two dimensions to obtain target sub-speech spectrogram features corresponding to each sub-speech spectrogram; Masking is performed on the target sub-speech spectrogram features corresponding to each sub-speech spectrogram to obtain a speech spectrogram mask.

4. The speech processing method according to claim 3, wherein the step of performing feature processing on each of the plurality of sub-speech spectrograms to obtain speech spectrogram features comprises: Performing feature extraction on each of the multiple sub-speech spectrograms to obtain initial sub-speech spectrogram features corresponding to each sub-speech spectrogram; The initial sub-speech spectrogram features corresponding to the sub-speech spectrograms are merged to obtain a speech spectrogram feature.

5. The speech processing method according to claim 4, wherein the speech processing model comprises a normalized network layer and a fully connected network layer; The extracting features of each sub-speech spectrogram from the plurality of sub-speech spectrograms to obtain initial sub-speech spectrogram features corresponding to each sub-speech spectrogram includes: The normalized network layer and the fully connected network layer are used to perform feature extraction on each of the multiple sub-speech spectrograms to obtain initial sub-speech spectrogram features corresponding to each sub-speech spectrogram.

6. The speech processing method according to claim 5, wherein the step of extracting features from each of the plurality of sub-speech spectrograms using the normalized network layer and the fully connected network layer comprises: In the normalized network layer and the fully connected network layer, determining a sub-normalized network layer and a sub-fully connected network layer corresponding to each of the multiple sub-speech spectrograms; The sub-normalized network layer and the sub-fully connected network layer corresponding to each sub-speech spectrogram are used to perform feature extraction on each sub-speech spectrogram to obtain initial sub-speech spectrogram features corresponding to each sub-speech spectrogram.

7. The speech processing method according to claim 3, wherein the masking process is performed on the target sub-speech spectrogram features corresponding to each sub-speech spectrogram to obtain a speech spectrogram mask, comprising: Performing mask estimation on target sub-speech spectrogram features corresponding to each sub-speech spectrogram to obtain sub-speech spectrogram masks corresponding to each sub-speech spectrogram; The sub-speech spectrogram masks corresponding to the sub-speech spectrograms are merged to obtain a speech spectrogram mask.

8. The speech processing method according to claim 2, wherein the denoising process is performed on the speech spectrogram according to the speech spectrogram mask, and the denoised target speech is obtained according to the processing result, comprising: Obtaining a denoised speech spectrogram according to the speech spectrogram mask and the speech spectrogram; The denoised speech spectrogram is subjected to time domain conversion processing to obtain the denoised target speech.

9. The speech processing method according to claim 3, wherein processing the speech spectrogram features according to at least two dimensions to obtain target sub-speech spectrogram features corresponding to each sub-speech spectrogram comprises: Encoding the speech spectrogram features according to the time dimension to obtain first sub-speech spectrogram features corresponding to each sub-speech spectrogram; Encoding the speech spectrogram features according to the frequency domain dimension to obtain second sub-speech spectrogram features corresponding to each sub-speech spectrogram; According to the first sub-speech spectrogram feature and the second sub-speech spectrogram feature, target sub-speech spectrogram features corresponding to the respective sub-speech spectrograms are determined.

10. The speech processing method according to claim 1, before dividing the speech spectrogram according to a preset bandwidth to obtain a plurality of sub-speech spectrograms corresponding to the speech to be processed, further comprising: Determining the speech type of the speech to be processed; According to the voice type, a preset bandwidth corresponding to the voice type is determined.

11. The speech processing method according to claim 1, before inputting the speech to be processed into the speech processing model, further comprising: Determining a noisy speech sample and a denoised speech label corresponding to the noisy speech sample; Inputting the noisy speech sample into a speech processing model, and performing frequency domain conversion processing on the noisy speech sample in the speech processing model to obtain a sample speech spectrogram of the noisy speech sample; Dividing the sample speech spectrogram according to a preset bandwidth to obtain a plurality of sample sub-speech spectrograms corresponding to the noisy speech sample; Performing denoising processing on each of the plurality of sample sub-speech spectrograms and obtaining a predicted denoised speech according to the processing result; The speech processing model is trained according to the denoised speech label and the predicted denoised speech until a speech processing model that meets a training stop condition is obtained.

12. The speech processing method according to claim 11, wherein the step of training the speech processing model according to the denoised speech label and the predicted denoised speech until a speech processing model that satisfies a training stop condition is obtained comprises: Calculating the denoised speech label and the predicted denoised speech according to at least one loss calculation rule to obtain a model loss value; The speech processing model is trained according to the model loss value until a speech processing model that meets the training stop condition is obtained.

13. The speech processing method according to claim 12, wherein the at least one loss calculation rule includes a rule for calculating speech resolution loss, a rule for calculating time domain waveform loss, and a rule for calculating speech signal distortion loss, and the model loss values ​​calculated by the rule for calculating speech resolution loss, the rule for calculating time domain waveform loss, and the rule for calculating speech signal distortion loss are different.

14. A speech processing model training method, applied to a cloud-side device, comprising: Determining a noisy speech sample and a denoised speech label corresponding to the noisy speech sample; Inputting the noisy speech sample into a speech processing model, and performing frequency domain conversion processing on the noisy speech sample in the speech processing model to obtain a sample speech spectrogram of the noisy speech sample; Dividing the sample speech spectrogram according to a preset bandwidth to obtain a plurality of sample sub-speech spectrograms corresponding to the noisy speech sample; Performing denoising processing on each of the plurality of sample sub-speech spectrograms and obtaining a predicted denoised speech according to the processing result; The speech processing model is trained according to the denoised speech label and the predicted denoised speech until a speech processing model that meets a training stop condition is obtained.

15. The speech processing model training method according to claim 14, wherein the training of the speech processing model according to the denoised speech labels and the predicted denoised speech until a speech processing model that satisfies a training stop condition is obtained comprises: Calculating the denoised speech label and the predicted denoised speech according to at least one loss calculation rule to obtain a model loss value; The speech processing model is trained according to the model loss value until a speech processing model that meets the training stop condition is obtained.

16. A speech denoising method, applied to a cloud-side device, comprising: Receive noisy voice sent by the client; Inputting the noisy speech into a speech processing model, and performing frequency domain conversion processing on the noisy speech in the speech processing model to obtain a speech spectrogram of the noisy speech; Dividing the speech spectrogram according to a preset bandwidth to obtain a plurality of sub-speech spectrograms corresponding to the noisy speech; Performing denoising processing on each of the plurality of sub-speech spectrograms, and obtaining a denoised target speech according to the processing result; The target voice is sent to the client.

17. A speech recognition method, applied to a cloud-side device, comprising: receiving a user's voice signal; Inputting the speech signal into a speech processing model, and performing frequency domain conversion processing on the speech signal in the speech processing model to obtain a speech spectrogram of the speech signal; Dividing the speech spectrogram according to a preset bandwidth to obtain a plurality of sub-speech spectrograms corresponding to the speech signal; Performing denoising processing on each of the multiple sub-speech spectrograms, and obtaining a denoised speech signal according to the processing result; Recognize the de-noised voice signal and generate a reply message according to the recognition result.

18. A computing device comprising: memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 17 are implemented.

19. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction is executed by a processor to implement the steps of the method according to any one of claims 1 to 17.

20. A computer program product comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 17.

Images