Speech enhancement method, related device, equipment and storage medium

Abstract

The invention discloses a speech enhancement method, a related device, equipment and a storage medium, and the method comprises the steps: obtaining a target speech; processing the target voice based on a voice enhancement model to obtain enhanced voice of the target voice after voice enhancement; wherein the speech enhancement model firstly takes a partially-shielded first sample time-frequency spectrum as a training sample and takes the recovered first sample time-frequency spectrum as a training target to carry out pre-training, and then takes an unshielded second sample time-frequency spectrum as a training sample and takes the recovered third sample time-frequency spectrum as a training target to carry out post-training; the second sample time-frequency spectrum is obtained by converting a sample noisy speech, the sample noisy speech is labeled with a sample enhanced speech, and the third sample time-frequency spectrum is obtained by converting the sample enhanced speech. According to the scheme, the speech enhancement effect can be improved, especially under the condition of less supervision data.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a speech enhancement method and related apparatus, devices and storage media. Background Technology

[0002] As the most natural entry point for communication activities such as human-to-human and human-computer interaction, speech usually needs to combat anomalies such as noise, echo, reverberation, and interference in the environment, thus making it necessary to enhance speech.

[0003] Traditional speech enhancement solutions typically rely on supervised training of speech enhancement models using labeled data to achieve speech enhancement. However, in real-world scenarios, labeled supervised data, especially high-quality supervised data, is often scarce, thus impacting the performance of speech enhancement models. Therefore, improving speech enhancement performance, particularly when supervised data is limited, has become a pressing issue. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a speech enhancement method and related apparatus, devices, and storage media that can improve speech enhancement effects, especially when there is limited supervised data.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a speech enhancement method, comprising: acquiring target speech; processing the target speech based on a speech enhancement model to obtain enhanced speech after speech enhancement; wherein the speech enhancement model is first pre-trained using a partially occluded first sample time spectrum as a training sample and recovering the first sample time spectrum as a training objective, and then post-trained using an unoccluded second sample time spectrum as a training sample and recovering a third sample time spectrum as a training objective, wherein the second sample time spectrum is obtained by converting the sample noisy speech, the sample noisy speech is labeled with sample enhanced speech, and the third sample time spectrum is obtained by converting the sample enhanced speech.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a speech enhancement device, comprising: a speech acquisition module and a speech processing module. The speech acquisition module is used to acquire target speech; the speech processing module is used to process the target speech based on a speech enhancement model to obtain enhanced speech after speech enhancement. The speech enhancement model is pre-trained using a partially occluded first sample time spectrum as a training sample and recovering the first sample time spectrum as the training objective. It is then post-trained using an unoccluded second sample time spectrum as a training sample and recovering a third sample time spectrum as the training objective. The second sample time spectrum is obtained by converting noisy sample speech, which is labeled with enhanced sample speech. The third sample time spectrum is obtained by converting enhanced sample speech.

[0007] To address the aforementioned technical problems, a third aspect of this application provides an electronic device comprising at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor executes the program instructions to implement the speech enhancement method of the first aspect described above.

[0008] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the speech enhancement method of the first aspect described above.

[0009] The above scheme acquires the target speech and processes it based on a speech enhancement model to obtain enhanced speech. The speech enhancement model is pre-trained using the partially obscured first sample's time-spectrum as training samples and recovering the first sample's time-spectrum as the training objective. It is then post-trained using the unobscured second sample's time-spectrum as training samples and recovering the third sample's time-spectrum as the training objective. The second sample's time-spectrum is obtained by converting the noisy sample speech, which is then labeled with the enhanced sample speech. The third sample's time-spectrum is also obtained by converting the enhanced sample speech. Therefore, the speech enhancement model is first trained using unsupervised data... Pre-training, using the partially obscured first sample's time-spectrum as training samples and aiming to recover it, forces the speech enhancement model to extensively learn to recover signals from contextual connections during pre-training. Consequently, subsequent training requires only a small amount of supervised data (i.e., noisy sample speech and its labeled enhanced sample speech) to transfer the pre-trained modeling capabilities to speech enhancement. This eliminates the need for massive amounts of supervised data for supervised training; the speech enhancement model's capabilities can be improved through a pre-training followed by post-training approach. Therefore, it enhances speech enhancement performance, especially when supervised data is limited. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an embodiment of the speech enhancement method of this application; Figure 2 This is a schematic diagram of a pre-training process according to an embodiment of this application; Figure 3 This is a schematic diagram of the framework of one embodiment of the training in this application; Figure 4 This is a schematic diagram of the framework of an embodiment of the speech enhancement device of this application; Figure 5 This is a schematic diagram of the framework of an embodiment of the electronic device of this application; Figure 6 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0011] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0012] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0013] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the slash " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper indicates two or more objects.

[0014] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the speech enhancement method of this application. Specifically, it may include the following steps: Step S11: Obtain the target speech.

[0015] Specifically, the target speech is the speech signal to be enhanced in this embodiment of the disclosure. For example, the target speech can be acquired in real time, such as during a meeting, the content of the speaker's speech can be acquired in real time, and the real-time acquired speech signal can be used as the target speech to be enhanced; or, the target speech can be acquired non-real time, such as after an interview, the conversation between the interview and the interview can be used as the target speech to be enhanced. Here, there is no limitation on whether the target speech is acquired in real time or non-real time, nor is there any limitation on the acquisition environment of the target speech.

[0016] Step S12: Process the target speech based on the speech enhancement model to obtain the enhanced speech after speech enhancement.

[0017] In this embodiment, the speech enhancement model is first pre-trained using the partially occluded first sample time spectrum as training samples and recovering the first sample time spectrum as the training objective. Then, it is post-trained using the unoccluded second sample time spectrum as training samples and recovering the third sample time spectrum as the training objective. The second sample time spectrum is obtained by converting the noisy sample speech, which is annotated with sample-enhanced speech. The third sample time spectrum is obtained by converting the sample-enhanced speech. It should be noted that the noisy sample speech and its annotated sample-enhanced speech can have the same speech information. The main difference is that the noisy sample speech, in addition to speech information, can also carry abnormal information such as reverberation, echo, interference, and noise. In other words, sample-enhanced speech is the clean speech obtained by removing abnormal information such as reverberation, echo, interference, and noise from the noisy sample speech, while noisy sample speech is the noisy speech obtained by incorporating abnormal information such as reverberation, echo, interference, and noise into the clean speech used as sample-enhanced speech. For example, in practical applications, clean speech can be obtained first through studio recording or large language model generation as sample-enhanced speech. Then, abnormal information such as reverberation, echo, interference, and noise can be added to the sample-enhanced speech to obtain sample-noisy speech. Of course, the above example is only one possible way to obtain sample-noisy speech and its annotated sample-enhanced speech in practical applications. The specific methods for obtaining sample-noisy speech and its annotated sample-enhanced speech are not limited here, nor will they be listed one by one.

[0018] In one implementation scenario, the speech enhancement model can adopt an encoder-decoder architecture. The encoder and decoder can each be constructed using Transformers; for details, please refer to the technical specifications of Transformers, which will not be elaborated upon here. Of course, as another possible implementation example, the speech enhancement model can also use other model architectures; therefore, no limitation is placed on the network architecture of the speech enhancement model here.

[0019] In one implementation scenario, the first sample time spectrum can be obtained by converting sample speech data. The sample speech data can be noisy or noiseless unsupervised data; that is, the speech quality of the sample speech data is not limited in this embodiment. It can carry abnormal signals such as noise, interference, reverberation, and echo, or it can be free of any abnormal signals. Furthermore, the sample speech data may not contain annotation information. That is, when the sample speech data carries abnormal signals, it is not required to annotate the corresponding clean speech (i.e., the clean speech data after removing abnormal signals). Therefore, there are no strict restrictions on the sample speech data used in the pre-training stage, allowing for the collection of massive amounts of unsupervised data for the pre-training stage. In addition, the sample speech data can be transformed using short-time Fourier transform and other transformation operations to obtain the first sample time spectrum. Of course, other time spectra described below can also be obtained by converting relevant speech data using short-time Fourier transform; their acquisition methods will not be elaborated further when discussing other time spectra. It should be noted that the time spectrum has two dimensions: the time domain and the frequency domain. In the time domain, the time spectrum can be divided into frame signals at different times, while in the frequency domain, each frame signal can contain spectral information at several frequency points. For details, please refer to the technical details of time spectrograms, such as spectrograms, which will not be elaborated here. For example, the spectral information of some frequency points in a portion of the first sample sub-spectrum in the first sample time spectrum can be masked. Here, the first sample sub-spectrum is the frame signal at different times in the time domain dimension of the first sample time spectrum. Furthermore, the specific proportions of the partial first sample sub-spectrum and the specific proportions of some frequency points can be referred to the relevant descriptions below, which will not be elaborated here.

[0020] In one implementation scenario, taking an encoder-decoder architecture for speech enhancement as an example, after obtaining the first sample time-spectrum, partial masking can be performed based on it to obtain the masked time-spectrum. Then, the encoder in the speech enhancement model encodes the masked time-spectrum to obtain a spectrogram representation, and further encodes the positional information of the masked portion in the first sample time-spectrum to obtain a positional representation. Based on this, the decoder in the speech enhancement model decodes the spectrogram and positional representations to obtain the first predicted time-spectrum. The network parameters of the speech enhancement model can then be adjusted based on the difference between the first predicted time-spectrum and the first sample time-spectrum. This approach, by partially masking the first sample time-spectrum and sequentially using time-spectrum encoding and encoding / decoding the positional information of the masked portion to recover the predicted time-spectrum, and by adjusting the network parameters based on the difference between the first sample time-spectrum and the predicted time-spectrum, forces the speech enhancement model to extensively learn to recover signals from contextual relationships during pre-training.

[0021] In a specific implementation scenario, to partially mask the spectrum of the first sample, a first sample sub-spectrum with a frequency not less than a first proportional threshold can be selected as the target sample sub-spectrum. Then, frequency ranges with frequencies not less than a second proportional threshold in each target sample sub-spectrum are masked to obtain the masked spectrum. As a possible implementation example, the first proportional threshold can be set according to the actual application needs. For example, to force the speech enhancement model to learn its modeling ability as accurately as possible, the first proportional threshold can be set appropriately large; if the accuracy requirement for the speech enhancement model's modeling ability is relatively relaxed, the first proportional threshold can be set moderately. In practical applications, the specific size of the first proportional threshold should be such that the speech enhancement model can learn its modeling ability, such as 90%, 95%, or even 100% (i.e., at this point, all first sample sub-spectrums in the first sample's time-frequency spectrum need to be partially masked). As one possible implementation example, the second proportional threshold can also be set according to the actual application needs. For example, to force the speech enhancement model to learn its modeling ability as accurately as possible, the second proportional threshold can be set appropriately large; if the accuracy of the speech enhancement model's modeling ability learning is relatively lenient, then the second proportional threshold can be set more moderately. In practical applications, the specific size of the second proportional threshold should be such that it can provide the necessary contextual information for the speech enhancement model to learn its modeling ability, such as 70%, 75%, 80%, etc. It should be noted that the above examples are only a few possible examples in practical applications, and the specific sizes of the first and second proportional thresholds are not limited here. Furthermore, the above examples of the first and second proportional thresholds can be freely combined as needed in practical applications, and will not be listed one by one here. The above method selects a first sample sub-spectrum with a frequency not lower than a first proportion threshold in the spectrum of the first sample, and uses these sub-spectrums as target sub-spectrums. Then, it masks the frequency ranges of each target sub-spectrum with a frequency not lower than a second proportion threshold, obtaining the spectrum when the samples are masked. This method can both force the speech enhancement model to learn its modeling ability during the pre-training stage through partial masking and provide the necessary contextual information for the speech enhancement model to learn its modeling ability, thus helping to ensure the effectiveness of pre-training as much as possible. For easier understanding, please refer to [reference needed]. Figure 2 , Figure 2 This is a schematic diagram illustrating the pre-training process of an embodiment of this application. Figure 2 As shown, the vertical spectra divided by dashed lines in the first sample's time spectrum are the first sample sub-spectrums. The portions obscured by the black-filled rectangles in the first sample sub-spectrum represent the obscured frequency ranges within that sub-spectrum. It should be noted that... Figure 2 The time-frequency spectrum shown is the spectrum of the sample when it is occluded, and the spectrum of the first sample when it is not occluded (for simplicity). Figure 2(The temporal spectrum of the speech before it was masked and fed into the speech enhancement model is not illustrated.) Of course... Figure 2 The example shown is only one possible example of partial spectral occlusion for the first sample. Other possible occlusion scenarios are not limited here, nor will they be listed one by one.

[0022] In a specific implementation scenario, after partially masking the first sample time spectrum, for each frame of the first sample sub-spectrum in the first sample time spectrum, it is possible to detect whether the first sample sub-spectrum is partially masked based on the sample masking time spectrum. In response to the first sample sub-spectrum being partially masked, the spectral information and position information of the unmasked part in the first sample sub-spectrum can be encoded based on the encoder to obtain the first sub-coded representation of the unmasked part in the first sample sub-spectrum. In response to the first sample sub-spectrum not being masked, the spectral information and position information of the first sample sub-spectrum are encoded based on the encoder to obtain the first sub-coded representation of the first sample sub-spectrum, wherein the spectrum coding representation includes the first sub-coded representation of each unmasked part in the first sample time spectrum. It should be noted that the spectrum of the first sample can be regarded as image data, and the position information of the unmasked part can be regarded as the image coordinates of the unmasked part in the image data. For example, the position information can be represented by the image coordinates of the top left and bottom right vertices (or top right and bottom left vertices) of the unmasked part. Similarly, the position information of the first sample sub-map can be regarded as the image coordinates of the top left and bottom right vertices (or top right and bottom left vertices) of the first sample sub-map in the image data. There are no restrictions on the way the position information is expressed here. Furthermore, when encoding the spectral and positional information of the unmasked portion based on the encoder, the spectral and positional information of the unmasked portion can be superimposed first as the information to be encoded in the unmasked portion. Then, the encoder is used to encode the information to be encoded in the unmasked portion to obtain the first sub-coded representation of the unmasked portion. Similarly, when encoding the first sample sub-map of the unmasked portion based on the encoder, the spectral and positional information of the first sample sub-map of the unmasked portion can be superimposed first as the information to be encoded in the unmasked first sample sub-map of the unmasked portion. Then, the encoder is used to encode the information to be encoded in the unmasked first sample sub-map of the unmasked portion to obtain the first sub-coded representation of the unmasked first sample sub-map of the unmasked portion. For easier understanding, please refer to [further details omitted]. Figure 2 ,like Figure 2 The vertical graph shown before the encoder represents the unmasked portion, while the green-filled vertical rectangle after the encoder represents the first sub-encoder. Of course, Figure 2The above diagram illustrates only one possible scenario for the encoder before and after encoding in practical applications. Other possible scenarios are not limited here, nor will they be listed one by one. In the above method, for each frame's first sample sub-spectrum in the first sample time-frequency spectrum, the encoder detects whether the first sample sub-spectrum is partially occluded based on the sample occlusion time-frequency spectrum. In response to partial occlusion, the encoder encodes the spectral and positional information of the unoccluded portion of the first sample sub-spectrum to obtain the first sub-coded representation of the unoccluded portion. Similarly, in response to unoccluded portions, the encoder encodes the spectral and positional information of the first sample sub-spectrum to obtain the first sub-coded representation of the first sample sub-spectrum. Since the spectral coded representation includes the first sub-coded representations of each unoccluded portion in the first sample time-frequency spectrum, positional information can be implicitly incorporated during the encoding process, facilitating subsequent establishment of contextual relationships for accurate decoding.

[0023] In a specific implementation scenario, the occluded portion can be encoded based on its location information to obtain a location-coded representation. Similarly, using the aforementioned location information, the frequency spectrum of the first sample can be considered as image data. Therefore, the location information of the occluded portion can be viewed as the image coordinates of the top-left and bottom-right vertices (or, the top-right and bottom-left vertices) of the occluded portion in the image data. The specific expression of the location information is not limited here. For easier understanding, please refer to [further details omitted]. Figure 2 , Figure 2 The vertical rectangle filled in black after the encoder represents the position encoding of the obscured portion. Of course, Figure 2 The example shown is only one possible scenario in actual application. Other possible scenarios are not limited here, nor will they be listed one by one.

[0024] In a specific implementation scenario, after obtaining the spectral encoding representation of the unmasked portion (which may implicitly contain the location information of the unmasked portion) and the location encoding representation of the masked portion (containing the location information of the masked portion), decoding can be performed to obtain the first prediction time spectrum. Specifically, the spectral encoding representation and the location encoding representation can be decoded using a decoder to obtain the first prediction sub-spectrum of the masked portion and the first prediction sub-spectrum of the unmasked portion in the first sample time spectrum. Then, the first prediction time spectrum is obtained by combining the first prediction sub-spectrum of the masked portion and the first prediction sub-spectrum of the unmasked portion in the first sample time spectrum. For easier understanding, please refer to [reference needed]. Figure 2In the diagram, the green-filled vertical rectangles represent the spectral encoding of the unmasked portion, while the black-filled vertical rectangles represent the positional encoding of the masked portion. After the decoder performs the decoding operation, the corresponding first prediction sub-spectrum is obtained. For example, the spectrum located after the decoder and horizontally corresponding to the black-filled vertical rectangle represents the first prediction sub-spectrum of the masked portion, while the spectrum located after the decoder and horizontally corresponding to the green-filled vertical rectangle represents the first prediction sub-spectrum of the unmasked portion. Of course, Figure 2 The example shown is merely one possible scenario in practical applications; other possible scenarios are not limited here, nor will they be listed one by one. The above method, based on the decoder decoding the spectrogram encoding representation and the position encoding representation, obtains the first predicted sub-spectrum of the masked portion and the first predicted sub-spectrum of the unmasked portion of the first sample time-frequency spectrum. Then, based on the first predicted sub-spectrum of the masked portion and the first predicted sub-spectrum of the unmasked portion of the first sample time-frequency spectrum, the first predicted time-frequency spectrum is obtained by combining them. This forces the speech enhancement model to recover the signal from the contextual connections established during the decoding process.

[0025] In a specific implementation scenario, after obtaining the first predicted time-frequency spectrum, the parameters of the speech enhancement model can be adjusted by combining it with the first sample time-frequency spectrum. Specifically, a first pre-training loss can be obtained based on the difference in the real part spectrum between the first predicted time-frequency spectrum and the first sample time-frequency spectrum at the same time-frequency point, and a second pre-training loss can be obtained based on the difference in the imaginary part spectrum between the first predicted time-frequency spectrum and the first sample time-frequency spectrum at the same time-frequency point. Based on this, the network parameters of the speech enhancement model can be adjusted using the first and second pre-training losses. For ease of understanding, the total loss of the first and second pre-training losses is... It can be represented as:

[0026] In the above formula, These represent the spectrum during the first sample and the spectrum during the first prediction, respectively. Let these represent the real and imaginary parts of the spectrum for the first sample, respectively. Let these represent the real and imaginary parts of the spectrum during the first prediction, respectively. To represent a time-frequency point, This represents the total number of time-frequency points. The above method derives a first pre-training loss based on the difference in the real part spectrum between the first predicted time-frequency spectrum and the first sample time-frequency spectrum at the same time-frequency point. It then derives a second pre-training loss based on the difference in the imaginary part spectrum between the first predicted time-frequency spectrum and the first sample time-frequency spectrum at the same time-frequency point. Finally, based on the first and second pre-training losses, the network parameters of the speech enhancement model are adjusted. This allows the speech enhancement model to be forced to recover the speech as accurately as possible even when the time-frequency spectrum is partially occluded.

[0027] In one implementation scenario, the speech enhancement model can be post-trained after pre-training. Specifically, the encoder in the speech enhancement model encodes the spectral and positional information of the second sample sub-spectrum in the second sample time-frequency spectrum to obtain the second sub-coded representation of the second sample sub-spectrum. The decoder in the speech enhancement model then decodes this second sub-coded representation to obtain the second predicted sub-spectrum after speech enhancement. Based on the second predicted sub-spectrums of each frame of the second sample sub-spectrum in the second sample time-frequency spectrum, the second predicted time-frequency spectrum can be combined to obtain the second predicted time-frequency spectrum. Furthermore, the network parameters of the speech enhancement model can be adjusted based on the difference between the second predicted time-frequency spectrum and the third sample time-frequency spectrum. This approach, by utilizing the difference between the second predicted time-frequency spectrum recovered from the second sample time-frequency spectrum and the desired output third sample time-frequency spectrum during post-training, allows for parameter adjustment of the speech enhancement model, transferring the modeling capabilities learned during pre-training to the restoration of enhanced speech.

[0028] In a specific implementation scenario, the specific method for encoding the spectral and positional information of the second sample sub-map can be found in the aforementioned description of encoding the spectral and positional information of the first sample sub-map, and will not be repeated here. Furthermore, the specific method for decoding the second sub-encoded representation can be found in the aforementioned description of decoding the map encoding representation and position encoding representation, and will also not be repeated here.

[0029] In a specific implementation scenario, please refer to the relevant documents. Figure 3 , Figure 3 This is a schematic diagram illustrating the process of training an embodiment of this application. For example... Figure 3 As shown, after the second sample sub-graphs in each frame of the second sample time spectrum are encoded by the encoder, the corresponding second sub-coded representations can be obtained (e.g., Figure 3 The vertical rectangle filled in green is then decoded by the decoder to obtain the corresponding second prediction sub-spectrum. Based on this, the second prediction time spectrum can be obtained.

[0030] In a specific implementation scenario, after decoding to obtain the second predicted time-frequency spectrum, the difference can be measured by combining it with the third sample time-frequency spectrum, so as to adjust the parameters of the speech enhancement model accordingly. Specifically, the first post-training loss can be obtained based on the difference in the real part spectrum between the second predicted time-frequency spectrum and the third sample time-frequency spectrum at the same time-frequency point, and the second post-training loss can be obtained based on the difference in the imaginary part spectrum between the second predicted time-frequency spectrum and the third sample time-frequency spectrum at the same time-frequency point. Based on this, the network parameters of the speech enhancement model can be adjusted according to the first and second post-training losses. For ease of understanding, the total loss of the first and second post-training losses... It can be represented as:

[0031] In the above formula, These represent the spectrum during the third sample and the spectrum during the second prediction, respectively. Let these represent the real and imaginary parts of the spectrum for the third sample, respectively. Let these represent the real and imaginary parts of the spectrum during the second prediction, respectively. To represent a time-frequency point, This represents the total number of time-frequency points. The above method derives a first post-training loss based on the difference in the real part spectrum between the second predicted time-frequency spectrum and the third sample time-frequency spectrum at the same time-frequency point, and a second post-training loss based on the difference in the imaginary part spectrum between the second predicted time-frequency spectrum and the third sample time-frequency spectrum at the same time-frequency point. Then, based on the first and second post-training losses, the network parameters of the speech enhancement model are adjusted. This allows the speech enhancement model to be forced to transfer its pre-trained modeling capabilities to the restoration of enhanced speech through the first and second post-training losses.

[0032] In one implementation scenario, after the above pre-training and post-training, the speech enhancement model can be used to enhance the target speech in order to eliminate abnormal signals such as reverberation, echo, interference, and noise in the target speech as much as possible, so as to obtain clean speech, which can be used as the enhanced speech after speech enhancement of the target speech. Taking the encoder-decoder architecture of the speech enhancement model as an example, the target speech can be converted to obtain the target time spectrum. Then, the encoder in the speech enhancement model encodes the information to be encoded in the first sub-spectrum of the target time spectrum to obtain the encoded representation of the first sub-spectrum. The information to be encoded in the first sub-spectrum includes the spectral information and positional information of the first sub-spectrum. The decoder in the speech enhancement model can then decode the encoded representation of the first sub-spectrum to obtain the second sub-spectrum after speech enhancement. Furthermore, waveform recovery can be performed based on the second sub-spectrum after speech enhancement of each frame of the first sub-spectrum in the target time spectrum to obtain the enhanced speech. For example, the enhanced time spectrum can be obtained by combining the second sub-spectrum after speech enhancement of each frame of the first sub-spectrum in the target time spectrum, and then waveform recovery can be performed using the enhanced time spectrum to obtain the enhanced speech. For instance, when using the short-time Fourier transform to convert the target speech to obtain the target time spectrum, the inverse short-time Fourier transform can be used to perform waveform recovery on the enhanced time spectrum to obtain the enhanced speech. For specific processing details, please refer to the technical details of the Short-Time Fourier Transform (SFT) and its inverse, which will not be elaborated here. Of course, the above example is merely one possible illustration of speech enhancement when the speech enhancement model uses an encoder-decoder architecture. The specific process of speech enhancement using other architectures is not limited here, nor will it be listed in detail. The above method converts the target speech to obtain the target time spectrum. The encoder in the speech enhancement model encodes the information to be encoded in the first sub-spectrum of the target time spectrum to obtain the encoded representation of the first sub-spectrum. The information to be encoded in the first sub-spectrum includes its spectral and positional information. The decoder in the speech enhancement model then decodes the encoded representation of the first sub-spectrum to obtain the second sub-spectrum after speech enhancement. Waveform recovery is then performed based on the second sub-spectrum after speech enhancement of each frame of the first sub-spectrum in the target time spectrum to obtain the enhanced speech. This method can fully utilize the connections established from the context to recover a clean speech signal during the speech enhancement process.

[0033] The above scheme acquires the target speech and processes it based on a speech enhancement model to obtain enhanced speech. The speech enhancement model is pre-trained using the partially obscured first sample's time-spectrum as training samples and recovering the first sample's time-spectrum as the training objective. It is then post-trained using the unobscured second sample's time-spectrum as training samples and recovering the third sample's time-spectrum as the training objective. The second sample's time-spectrum is obtained by converting the noisy sample speech, which is then labeled with the enhanced sample speech. The third sample's time-spectrum is also obtained by converting the enhanced sample speech. Therefore, the speech enhancement model is first trained using unsupervised data... Pre-training, using the partially obscured first sample's time-spectrum as training samples and aiming to recover it, forces the speech enhancement model to extensively learn to recover signals from contextual connections during pre-training. Consequently, subsequent training requires only a small amount of supervised data (i.e., noisy sample speech and its labeled enhanced sample speech) to transfer the pre-trained modeling capabilities to speech enhancement. This eliminates the need for massive amounts of supervised data for supervised training; the speech enhancement model's capabilities can be improved through a pre-training followed by post-training approach. Therefore, it enhances speech enhancement performance, especially when supervised data is limited.

[0034] Please see Figure 4 , Figure 4 This is a schematic diagram of the framework of an embodiment of the speech enhancement device of this application. The speech enhancement device 40 includes: a speech acquisition module 41 and a speech processing module 42. The speech acquisition module 41 is used to acquire target speech; the speech processing module 42 is used to process the target speech based on a speech enhancement model to obtain enhanced speech after speech enhancement; wherein, the speech enhancement model first uses the partially occluded first sample time spectrum as training sample and recovers the first sample time spectrum as training target for pre-training, and then uses the unoccluded second sample time spectrum as training sample and recovers the third sample time spectrum as training target for post-training. The second sample time spectrum is obtained by converting the sample noisy speech, and the sample noisy speech is labeled with sample enhanced speech. The third sample time spectrum is obtained by converting the sample enhanced speech.

[0035] In the above scheme, the speech enhancement device 40 acquires the target speech and processes it based on the speech enhancement model to obtain the enhanced speech. The speech enhancement model first uses the partially obscured first sample time spectrum as training samples and reconstructs the first sample time spectrum as the training objective for pre-training. Then, it uses the unobscured second sample time spectrum as training samples and reconstructs the third sample time spectrum as the training objective for post-training. The second sample time spectrum is obtained by converting the noisy sample speech, which is labeled with the enhanced sample speech. The third sample time spectrum is obtained by converting the enhanced sample speech. Therefore, the speech enhancement model first uses the partially obscured first sample time spectrum as training samples and reconstructs the third sample time spectrum as the training objective for post-training. Unsupervised pre-training, using partially masked first-sample time-spectrum data as training samples and aiming to recover the first-sample time-spectrum data during pre-training, forces the speech enhancement model to extensively learn to recover signals from contextual connections during pre-training. Consequently, in subsequent training, only a small amount of supervised data (i.e., noisy sample speech and its labeled enhanced sample speech) is needed to transfer the modeling capabilities learned during pre-training to the recovery of enhanced speech. This means that the speech enhancement model's speech enhancement capabilities can be improved through a pre-training followed by post-training process without requiring massive amounts of supervised data for supervised training. Therefore, it can improve speech enhancement performance, especially when supervised data is limited.

[0036] In some disclosed embodiments, the speech processing module 42 includes a speech conversion submodule for converting based on the target speech to obtain a target time spectrum; the speech processing module 42 includes a spectrogram encoding submodule for encoding the information to be encoded in the first sub-spectrum of the target time spectrum based on the encoder in the speech enhancement model to obtain an encoded representation of the first sub-spectrum; wherein the information to be encoded in the first sub-spectrum includes spectral information and position information of the first sub-spectrum; the speech processing module 42 includes a feature decoding submodule for decoding the encoded representation of the first sub-spectrum based on the decoder in the speech enhancement model to obtain a second sub-spectrum after speech enhancement of the first sub-spectrum; the speech processing module 42 includes a waveform recovery submodule for performing waveform recovery based on the second sub-spectrum after speech enhancement of each frame of the first sub-spectrum in the target time spectrum to obtain enhanced speech.

[0037] In some disclosed embodiments, the speech enhancement device 40 includes a partial masking module for partially masking based on the first sample time spectrum to obtain a sample masked time spectrum; the speech enhancement device 40 includes a first encoding module for encoding the sample masked time spectrum based on the encoder in the speech enhancement model to obtain a spectrogram encoding representation, and encoding based on the position information of the masked portion in the first sample time spectrum to obtain a position encoding representation; the speech enhancement device 40 includes a first decoding module for decoding the spectrogram encoding representation and the position encoding representation based on the decoder in the speech enhancement model to obtain a first predicted time spectrum; and the speech enhancement device 40 includes a first adjustment module for adjusting the network parameters of the speech enhancement model based on the difference between the first predicted time spectrum and the first sample time spectrum.

[0038] In some disclosed embodiments, the partial masking module includes a spectrum selection submodule, used to select a first sample sub-spectrum with a frequency not less than a first proportional threshold in the spectrum of the first sample, and use them as sample target sub-spectrums respectively; the partial masking module includes a spectrum masking submodule, used to mask the frequency range of each sample target sub-spectrum with a frequency not less than a second proportional threshold respectively, to obtain the spectrum when the sample is masked.

[0039] In some disclosed embodiments, the first encoding module includes a spectrum detection submodule, used to detect whether the first sample sub-spectrum is partially occluded for each frame of the first sample time spectrum based on the sample occlusion time spectrum; the first encoding module includes a first response submodule, used to encode the spectral information and position information of the unoccluded portion of the first sample sub-spectrum based on the encoder in response to the partial occlusion of the first sample sub-spectrum, to obtain a first sub-encoded representation of the unoccluded portion of the first sample sub-spectrum; the first encoding module includes a second response submodule, used to encode the spectral information and position information of the first sample sub-spectrum based on the encoder in response to the unoccluded portion of the first sample sub-spectrum, to obtain a first sub-encoded representation of the first sample sub-spectrum; wherein, the spectrum encoding representation includes the first sub-encoded representation of each unoccluded portion in the first sample time spectrum.

[0040] In some disclosed embodiments, the first decoding module includes a spectrum decoding submodule, used to decode the spectrum encoding representation and the position encoding representation based on the decoder to obtain a first predicted sub-spectrum of the occluded portion of the spectrum at the first sample time and a first predicted sub-spectrum of the unoccluded portion of the spectrum at the first sample time; the first decoding module includes a spectrum combining submodule, used to combine the first predicted sub-spectrum of the occluded portion of the spectrum at the first sample time and the first predicted sub-spectrum of the unoccluded portion of the spectrum at the first sample time to obtain a first predicted time spectrum.

[0041] In some disclosed embodiments, the first adjustment module includes a first metric submodule, used to obtain a first pre-training loss based on the difference in the real part spectrum between the first predicted time spectrum and the first sample time spectrum at the same time-frequency point, and to obtain a second pre-training loss based on the difference in the imaginary part spectrum between the first predicted time spectrum and the first sample time spectrum at the same time-frequency point; the first adjustment module includes a first parameter tuning submodule, used to adjust the network parameters of the speech enhancement model based on the first pre-training loss and the second pre-training loss.

[0042] In some disclosed embodiments, the speech enhancement device 40 includes a second encoding module for encoding the spectral information and positional information of the second sample sub-graph in the second sample time-spectrum based on the encoder in the speech enhancement model, to obtain a second sub-coded representation of the second sample sub-graph; the speech enhancement device 40 includes a second decoding module for decoding the second sub-coded representation of the second sample sub-graph based on the decoder in the speech enhancement model, to obtain a second predicted sub-graph after speech enhancement; the speech enhancement device 40 includes a graph combination module for combining the second predicted sub-graphs of each frame of the second sample sub-graph in the second sample time-spectrum to obtain a second predicted time-spectrum; and the speech enhancement device 40 includes a second adjustment module for adjusting the network parameters of the speech enhancement model based on the difference between the second predicted time-spectrum and the third sample time-spectrum.

[0043] In some disclosed embodiments, the second adjustment module includes a second metric submodule, used to obtain a first post-training loss based on the difference in the real part spectrum between the second predicted time spectrum and the third sample time spectrum at the same time frequency point, and to obtain a second post-training loss based on the difference in the imaginary part spectrum between the second predicted time spectrum and the third sample time spectrum at the same time frequency point; the second adjustment module includes a second parameter tuning submodule, used to adjust the network parameters of the speech enhancement model based on the first post-training loss and the second post-training loss.

[0044] In some disclosed embodiments, the spectral information of some frequency points in a portion of the first sample sub-spectrum in the first sample time spectrum is masked; and / or, the first sample time spectrum is obtained by converting sample speech data, which is noisy or noiseless unsupervised data; and / or, the speech enhancement model adopts an encoder-decoder model architecture.

[0045] Please see Figure 5 , Figure 5This is a schematic diagram of a framework of an embodiment of the electronic device of this application. The electronic device 50 includes at least a memory 51 and a processor 52 coupled to each other. The memory 51 stores at least program instructions, and the processor 52 is used to execute the program instructions to implement the steps in any of the above-described embodiments of the voice enhancement method. For details, please refer to the foregoing disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 50 may include, but is not limited to, mobile phones, tablet computers, learning machines, translators, smart screens, servers, etc., and the specific type of the electronic device 50 is not limited here.

[0046] Specifically, processor 52 controls itself and memory 51 to implement the steps in any of the above-described speech enhancement method embodiments. Processor 52 can also be referred to as a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 52 can be implemented using integrated circuit chips.

[0047] In the above scheme, electronic device 50 acquires the target speech and processes it based on a speech enhancement model to obtain enhanced speech. The speech enhancement model first uses the partially occluded first sample time spectrum as training samples and reconstructs the first sample time spectrum as the training objective for pre-training. Then, it uses the unoccluded second sample time spectrum as training samples and reconstructs the third sample time spectrum as the training objective for post-training. The second sample time spectrum is obtained by converting the noisy sample speech, which is labeled with the enhanced sample speech. The third sample time spectrum is obtained by converting the enhanced sample speech. Therefore, the speech enhancement model first uses the unoccluded sample time spectrum as training samples and reconstructs the third sample time spectrum as the training objective during training. Supervised pre-training, using partially masked first-sample time-spectrum data as training samples and aiming to recover the first-sample time-spectrum, forces the speech enhancement model to extensively learn to recover signals from contextual connections during pre-training. Consequently, in subsequent training, only a small amount of supervised data (i.e., noisy sample speech and its labeled enhanced sample speech) is needed to transfer the pre-trained modeling capabilities to enhanced speech recovery. This eliminates the need for massive amounts of supervised data for supervised training of the speech enhancement model; the pre-training-then-training process can improve the model's speech enhancement capabilities. Therefore, it can improve speech enhancement performance, especially when supervised data is limited.

[0048] Please see Figure 6 , Figure 6 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 60 stores program instructions 61 that can be executed by a processor. The program instructions 61 are used to implement the steps in any of the above-described embodiments of the speech enhancement method.

[0049] In the above scheme, the computer-readable storage medium 60 acquires the target speech and processes it based on a speech enhancement model to obtain enhanced speech. The speech enhancement model first uses the partially obscured first sample time spectrum as training samples and reconstructs the first sample time spectrum as the training objective for pre-training. Then, it uses the unobscured second sample time spectrum as training samples and reconstructs the third sample time spectrum as the training objective for post-training. The second sample time spectrum is obtained by converting the noisy sample speech, which is labeled with the enhanced sample speech. The third sample time spectrum is obtained by converting the enhanced sample speech. Therefore, the speech enhancement model first... Pre-training with unsupervised data, and using the partially masked first sample's time-spectrum as training samples with the goal of recovering the first sample's time-spectrum, forces the speech enhancement model to extensively learn to recover signals from contextual connections during pre-training. Consequently, in subsequent training, only a small amount of supervised data (i.e., noisy sample speech and its labeled enhanced sample speech) is needed to transfer the modeling capabilities learned during pre-training to the recovery of enhanced speech. This means that the speech enhancement model's speech enhancement capabilities can be improved through a pre-training followed by post-training process without requiring massive amounts of supervised data for supervised training. Therefore, it can improve speech enhancement performance, especially when supervised data is limited.

[0050] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0051] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0052] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0053] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0054] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0055] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0056] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. A speech enhancement method, characterized in that, include: Acquire the target speech; The target speech is processed based on a speech enhancement model to obtain enhanced speech. The speech enhancement model is first pre-trained using the partially occluded first sample time spectrum as training sample and recovering the first sample time spectrum as training objective. Then, it is post-trained using the unoccluded second sample time spectrum as training sample and recovering the third sample time spectrum as training objective. The second sample time spectrum is obtained by converting the sample noisy speech, and the sample noisy speech is labeled with sample enhanced speech. The third sample time spectrum is obtained by converting the sample enhanced speech.

2. The method according to claim 1, characterized in that, The process of processing the target speech based on the speech enhancement model to obtain enhanced speech after speech enhancement includes: Based on the target speech, the target time spectrum is obtained by conversion; The encoder in the speech enhancement model encodes the information to be encoded in the first sub-spectrum of the target time spectrum to obtain the encoded representation of the first sub-spectrum; wherein, the information to be encoded in the first sub-spectrum includes the spectral information and positional information of the first sub-spectrum; Based on the decoder in the speech enhancement model, the encoded representation of the first sub-graph is decoded to obtain the second sub-graph after speech enhancement. The enhanced speech is obtained by reconstructing the waveform of the second sub-graph after speech enhancement of the first sub-graph in each frame of the target time spectrum.

3. The method according to claim 1, characterized in that, The pre-training steps include: Partial masking is performed based on the first sample's time spectrum to obtain the sample's masked time spectrum; The encoder in the speech enhancement model encodes the spectrum of the sample during occlusion to obtain a spectral encoding representation, and the encoder encodes the position information of the occluded part in the spectrum of the first sample to obtain a position encoding representation. Based on the decoding of the spectrogram encoding representation and the position encoding representation in the speech enhancement model, the first prediction time spectrum is obtained; The network parameters of the speech enhancement model are adjusted based on the difference between the first predicted time spectrum and the first sample time spectrum.

4. The method according to claim 3, characterized in that, The partial masking based on the first sample time spectrum to obtain the sample masked time spectrum includes: Select the first sample sub-spectrum, which contains at least a first proportional threshold value in the spectrum of the first sample, and use them as the sample target sub-spectrums respectively. Each sample target sub-spectrum is masked for a frequency range not lower than the second proportional threshold to obtain the sample masked spectrum.

5. The method according to claim 3, characterized in that, The encoding of the spectrum of the sample during masking based on the encoder in the speech enhancement model to obtain a spectrogram encoded representation includes: For each frame of the first sample sub-graph in the first sample time spectrum, detect whether the first sample sub-graph is partially occluded based on the sample occlusion time spectrum; In response to the first sample sub-map being partially obscured, the encoder encodes the spectral information and positional information of the unobscured portion of the first sample sub-map to obtain a first sub-encoded representation of the unobscured portion of the first sample sub-map. In response to the first sample sub-spectrum not being masked, the encoder encodes the spectral information and positional information of the first sample sub-spectrum to obtain a first sub-coded representation of the first sample sub-spectrum; wherein the spectral coded representation includes the first sub-coded representation of each unmasked portion of the spectrum of the first sample.

6. The method according to claim 3, characterized in that, The decoding of the spectrogram encoding representation and the position encoding representation based on the decoder in the speech enhancement model to obtain the first prediction time spectrum includes: Based on the decoder, the spectral encoding representation and the position encoding representation are decoded to obtain the first predicted sub-spectrum of the occluded portion of the first sample time spectrum and the first predicted sub-spectrum of the unoccluded portion of the first sample time spectrum; The first predicted time spectrum is obtained by combining the first predicted sub-spectrum of the occluded portion of the first sample time spectrum and the first predicted sub-spectrum of the unoccluded portion of the first sample time spectrum.

7. The method according to claim 3, characterized in that, The step of adjusting the network parameters of the speech enhancement model based on the difference between the first predicted time spectrum and the first sample time spectrum includes: Based on the difference in the real part spectrum between the first predicted time spectrum and the first sample time spectrum at the same time frequency point, a first pre-training loss is obtained, and based on the difference in the imaginary part spectrum between the first predicted time spectrum and the first sample time spectrum at the same time frequency point, a second pre-training loss is obtained. The network parameters of the speech enhancement model are adjusted based on the first pre-training loss and the second pre-training loss.

8. The method according to claim 1, characterized in that, The post-training steps include: Based on the encoder in the speech enhancement model, the spectral information and positional information of the second sample sub-graph in the second sample time spectrum are encoded to obtain the second sub-encoded representation of the second sample sub-graph; Based on the decoder in the speech enhancement model, the second sub-coded representation of the second sample sub-graph is decoded to obtain the second predicted sub-graph after speech enhancement of the second sample sub-graph. The second prediction time spectrum is obtained by combining the second sample sub-graphs of each frame in the second sample time spectrum after speech enhancement. The network parameters of the speech enhancement model are adjusted based on the difference between the second predicted time spectrum and the third sample time spectrum.

9. The method according to claim 8, characterized in that, The step of adjusting the network parameters of the speech enhancement model based on the difference between the second predicted time spectrum and the third sample time spectrum includes: Based on the difference in the real part spectrum between the second predicted time spectrum and the third sample time spectrum at the same time frequency point, a first post-training loss is obtained, and based on the difference in the imaginary part spectrum between the second predicted time spectrum and the third sample time spectrum at the same time frequency point, a second post-training loss is obtained. The network parameters of the speech enhancement model are adjusted based on the first post-training loss and the second post-training loss.

10. The method according to any one of claims 1 to 9, characterized in that, In the first sample time spectrum, the spectral information of some frequency points in a portion of the first sample sub-spectrum is obscured; And / or, the first sample time spectrum is obtained by converting sample speech data, wherein the sample speech data is noisy or noise-free unsupervised data; And / or, the speech enhancement model adopts an encoder-decoder model architecture.

11. A voice enhancement device, characterized in that, include: The voice acquisition module is used to acquire the target voice. The speech processing module is used to process the target speech based on a speech enhancement model to obtain enhanced speech after speech enhancement. The speech enhancement model is first pre-trained using the partially occluded first sample time spectrum as training samples and recovering the first sample time spectrum as the training objective. Then, it is post-trained using the unoccluded second sample time spectrum as training samples and recovering the third sample time spectrum as the training objective. The second sample time spectrum is obtained by converting the sample noisy speech, and the sample noisy speech is labeled with sample enhanced speech. The third sample time spectrum is obtained by converting the sample enhanced speech.

12. An electronic device, characterized in that, It includes at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor executes the program instructions to implement the speech enhancement method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The device stores program instructions that can be executed by a processor, the program instructions being used to implement the speech enhancement method according to any one of claims 1 to 10.

Images