Voice processing method, device, electronic device and storage medium

By extracting the band component characteristics of the speech signal and correcting it using the attention model, the clarity problem of the speech signal under reverberation and noise interference is solved, and more efficient noise suppression and reverberation cancellation is achieved, improving the quality of the speech signal Experience with audio calls.

CN114360562BActive Publication Date: 2024-11-05BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111555187.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-17
Publication Date
2024-11-05
Estimated Expiration
2041-12-17

Smart Images

  • Figure CN114360562B_ABST
    Figure CN114360562B_ABST
Patent Text Reader

Abstract

The present disclosure proposes a speech processing method, device, electronic device and storage medium, which relate to the fields of artificial intelligence technology, especially speech enhancement technology and deep learning technology. The specific implementation scheme is: obtaining the amplitude component features and phase component features corresponding to multiple frequency band components corresponding to the speech signal, increasing the amount of signal that the model can obtain, and then capturing the amplitude component features and phase component features corresponding to each frequency band component through the attention model. The correlation of each frequency band component in time and frequency improves the processing effect of the model, and obtains the amplitude correction factor and phase correction factor corresponding to each frequency band component to correct the amplitude component and phase component corresponding to the frequency band component, and then obtains the target speech signal through sub-band synthesis, which improves the effect of noise suppression and reverberation elimination.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of speech enhancement technology and deep learning technology, and specifically to a speech processing method, device, electronic device and storage medium. Background Art

[0002] In everyday acoustic environments, speech is often disturbed by reverberation and background noise, which can reduce speech clarity and quality, affect the user experience during audio calls, and also affect the performance of speech synthesis systems. Therefore, noise suppression and reverberation elimination have become hot issues in speech signal enhancement. Summary of the invention

[0003] The present disclosure provides a speech processing method, device, electronic device and storage medium.

[0004] According to one aspect of the present disclosure, there is provided a speech processing method, comprising:

[0005] Acquire multiple frequency band components corresponding to the speech signal;

[0006] Determine amplitude components and phase components corresponding to the multiple frequency band components;

[0007] Extracting features of amplitude components and phase components corresponding to the multiple frequency band components to obtain amplitude component features and phase component features corresponding to the multiple frequency band components;

[0008] Input the amplitude component features and the phase component features corresponding to the multiple frequency band components into the attention model, and output the amplitude correction factors and the phase correction factors corresponding to the multiple frequency band components;

[0009] Correcting the amplitude components and phase components corresponding to the multiple frequency band components according to the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components;

[0010] Sub-band synthesis is performed according to the corrected amplitude components and phase components corresponding to the multiple frequency band components to obtain a target speech signal.

[0011] According to another aspect of the present disclosure, there is provided a speech processing device, comprising:

[0012] An acquisition module, used for acquiring multiple frequency band components corresponding to the speech signal;

[0013] A determination module, used to determine the amplitude components and phase components corresponding to the multiple frequency band components;

[0014] A feature extraction module, used to extract features of amplitude components and phase components corresponding to the multiple frequency band components, to obtain amplitude component features and phase component features corresponding to the multiple frequency band components;

[0015] A processing module, used to input the amplitude component features and the phase component features corresponding to the multiple frequency band components into the attention model, and output the amplitude correction factors and the phase correction factors corresponding to the multiple frequency band components;

[0016] A correction module, used to correct the amplitude components and phase components corresponding to the multiple frequency band components according to the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components;

[0017] The synthesis module is used to perform sub-band synthesis according to the corrected amplitude components and phase components corresponding to the multiple frequency band components to obtain the target speech signal.

[0018] According to another aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the aforementioned aspect is implemented.

[0019] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in the aforementioned aspect is implemented.

[0020] According to another aspect of the present disclosure, a computer program product is provided, on which a computer program is stored. When the program is executed by a processor, the method described in the aforementioned aspect is implemented.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 A flowchart of a speech processing method provided by an embodiment of the present disclosure;

[0024] Figure 2 A flowchart of another speech processing method provided by an embodiment of the present disclosure;

[0025] Figure 3 A schematic diagram of a speech processing effect provided by an embodiment of the present disclosure;

[0026] Figure 4 A flowchart of another speech processing method provided by an embodiment of the present disclosure;

[0027] Figure 5 A schematic diagram of the structure of an attention model provided in an embodiment of the present disclosure;

[0028] Figure 6 A schematic diagram of the structure of an attention network provided by an embodiment of the present disclosure;

[0029] Figure 7 A flowchart of another image processing method provided by an embodiment of the present disclosure;

[0030] Figure 8 A schematic diagram of the structure of a speech processing device provided by an embodiment of the present disclosure;

[0031] Fig. 9 A schematic block diagram of an electronic device provided in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] The speech processing method, apparatus, electronic device, and storage medium of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0034] In daily acoustic environments, speech is usually disturbed by room reverberation and background noise, which can reduce the clarity and quality of speech and affect the user experience in audio calls. However, the single-channel algorithm in the related art is difficult to achieve ideal results in noise suppression and reverberation elimination due to limited available information. Therefore, the present disclosure proposes a speech processing method to obtain amplitude component features and phase component features corresponding to multiple frequency band components corresponding to the speech signal, increase the amount of signal that can be obtained by the model, and then capture the correlation of each frequency band component in time and frequency through the attention model. The processing effect of the model is improved, and the amplitude correction factor and phase correction factor corresponding to each frequency band component are obtained to correct the amplitude component and phase component corresponding to the frequency band component, and then the target speech signal is obtained through sub-band synthesis, which improves the effect of noise suppression and reverberation elimination.

[0035] Figure 1A flowchart of a speech processing method provided by an embodiment of the present disclosure.

[0036] The execution subject of the speech processing method of the embodiment of the present disclosure is a speech processing device, which can be set in an electronic device. The electronic device can be a terminal or a server, which is not limited in this embodiment.

[0037] like Figure 1 As shown, the method may include the following steps:

[0038] Step 101: Acquire multiple frequency band components corresponding to a speech signal.

[0039] In the implementation example of the present disclosure, by acquiring a voice signal collected by a voice collection device, that is, an unprocessed original voice signal, for example, a voice signal collected by a microphone, the collected voice signal is subjected to frame division and windowing processing to obtain each frame of voice signal, and then, the frequency band component corresponding to each frame of voice signal is determined, which can be specifically implemented by the following implementation method:

[0040] In one implementation of the embodiment of the present disclosure, sub-band decomposition is performed on each frame of speech signal to obtain multiple frequency bands, thereby obtaining multiple frequency band components corresponding to each frame of speech signal. Sub-band decomposition can better prevent frequency band information leakage, make information between different frequency bands more independent, and facilitate improving the efficiency of subsequent speech processing.

[0041] In another implementation of the embodiment of the present disclosure, short-time Fourier transform is used for each frame of speech signal to obtain the spectrum of each frame of speech signal, and the spectrum is divided into frequency bands to obtain multiple frequency band components corresponding to the speech signal, so as to convert the time domain signal into a frequency domain signal.

[0042] It should be noted that the granularity of frequency band division can be set according to demand and is not limited in this embodiment.

[0043] Among them, in the technical solution of the present disclosure, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0044] Step 102: determine amplitude components and phase components corresponding to a plurality of frequency band components.

[0045] In the embodiment of the present disclosure, for each frequency band component corresponding to the speech signal, an amplitude component and a phase component corresponding to each frequency band component are determined, thereby increasing the amount of information obtained from the speech signal.

[0046] Step 103: extract features of amplitude components and phase components corresponding to the multiple frequency band components to obtain amplitude component features and phase component features corresponding to the multiple frequency band components.

[0047] In the disclosed embodiment, for each frequency band component, feature extraction is performed on the amplitude component and phase component corresponding to the frequency band component to obtain amplitude component features and phase component features, thereby increasing the amount of information that can be obtained by subsequent attention models, allowing subsequent attention models to obtain more sufficient information, thereby improving the performance of the attention model.

[0048] Step 104, input the amplitude component features and phase component features corresponding to the multiple frequency band components into the attention model, and output the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components.

[0049] In the disclosed embodiment, the attention model analyzes the correlation between the amplitude component features and the phase component features corresponding to each frequency band component of the input, so as to capture the time and frequency correlation corresponding to each frequency band component, so as to improve the processing capability of the attention model, and further obtain the amplitude component correction factor for correcting the amplitude component corresponding to each frequency band component, and the phase component correction factor for correcting the corresponding phase component.

[0050] Step 105: correct the amplitude components and phase components corresponding to the multiple frequency band components according to the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components.

[0051] In the disclosed embodiment, for each frequency band component, the amplitude component corresponding to each frequency band component after correction is obtained according to the corresponding amplitude component and the amplitude correction factor, and the phase component corresponding to each frequency band component after correction is obtained according to the phase component and the phase correction factor corresponding to each frequency band component. As an implementation method, for each frequency band component, the amplitude component and the corresponding amplitude correction factor are multiplied to obtain the amplitude component corresponding to the frequency band component after correction; the phase component and the corresponding amplitude correction factor are multiplied to obtain the phase component corresponding to the frequency band component after correction.

[0052] Step 106: Perform sub-band synthesis according to the corrected amplitude components and phase components corresponding to the multiple frequency band components to obtain the target speech signal.

[0053] In the disclosed embodiment, for each frequency band component, the corresponding amplitude component and phase component are corrected according to the corresponding amplitude correction factor and phase correction factor to obtain the amplitude components and phase components corresponding to the corrected multiple frequency band components, and sub-band synthesis is performed to obtain the target speech signal, wherein the target speech signal is a time domain signal obtained after enhancing the acquired speech signal.

[0054] In the speech processing method of the embodiment of the present disclosure, the amplitude component features and phase component features corresponding to multiple frequency band components corresponding to the speech signal are obtained, which increases the amount of signal that can be obtained by the model. The amplitude component features and phase component features corresponding to each frequency band component are captured through an attention model to capture the feature correlation on each frequency band component, so as to strengthen the features of important areas, improve the processing effect of the model, and obtain the amplitude correction factor and phase correction factor corresponding to each frequency band component to correct the amplitude component and phase component corresponding to the frequency band component, and then obtain the target speech signal through sub-band synthesis, thereby improving the effect of speech enhancement.

[0055] Based on the above embodiments, Figure 2 A flowchart of another speech processing method provided by an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the method comprises the following steps:

[0056] Step 201: Acquire multiple frequency band components corresponding to a speech signal.

[0057] Step 202: determine amplitude components and phase components corresponding to multiple frequency band components.

[0058] Step 203: extract features of amplitude components and phase components corresponding to the multiple frequency band components to obtain amplitude component features and phase component features corresponding to the multiple frequency band components.

[0059] Specifically, step 201 to step 203 can refer to the explanation in the aforementioned embodiment, the principles are the same, and will not be repeated in this embodiment.

[0060] Step 204, input the amplitude component features and phase component features corresponding to the multiple frequency band components into the encoding network of the attention model to obtain the encoding features corresponding to the multiple frequency band components.

[0061] In the disclosed embodiment, for each frequency band component, the corresponding amplitude component feature and phase component feature are input into the coding network of the attention model, which increases the amount of information of the speech signal input to the coding network. The coding network encodes and fuses the amplitude feature and the phase component feature to obtain a finer granularity coding feature that carries more speech signal information.

[0062] Step 205, inputting the encoding features corresponding to the multiple frequency band components into the attention network of the attention model to obtain the fusion features corresponding to the multiple frequency band components.

[0063] Among them, the fusion feature contains the feature correlation of the corresponding frequency band components in the time dimension and frequency dimension.

[0064] Step 206, input the fusion features corresponding to the multiple frequency band components into the decoding network of the attention model to obtain the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components.

[0065] In the disclosed embodiment, the coded features are processed by an attention network to extract the feature correlation of each frequency band component in the time dimension and the frequency dimension to obtain a fused feature containing a global correlation, and the fused feature and the decoded feature are input into a decoding network. The decoding network is composed of three layers of two-dimensional deconvolution to reconstruct the partially compressed spectral features of the coding network and the attention network, and obtain the amplitude correction factor and phase correction factor corresponding to each frequency band component to improve the effect of the model.

[0066] Step 207: correct the amplitude components and phase components corresponding to the multiple frequency band components according to the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components.

[0067] Step 208: Perform sub-band synthesis according to the corrected amplitude components and phase components corresponding to the multiple frequency band components to obtain the target speech signal.

[0068] Among them, step 207 and step 208 can refer to the explanation in the above-mentioned embodiment, and the principles are the same, so they will not be repeated in this embodiment.

[0069] In everyday acoustic environments, speech is often disturbed by reverberation and background noise. As an example, Figure 3 A schematic diagram of a speech processing effect provided by an embodiment of the present disclosure is provided, and the speech signal to be processed is a speech signal with reverberation and background noise as an example for explanation. Figure 3 As shown in the figure above, the time domain waveform and spectrum of the speech signal to be processed are shown in the spectrum of the figure above. It can be seen that the speech signal to be processed has a long tail, the speech reverberation is large, and there is a certain amount of background noise. The figure below is the target speech signal obtained after processing. From the spectrum and time domain waveform of the figure below, it can be seen that the background noise in the target speech signal is suppressed, the reverberation is eliminated, and the speech signal is almost not damaged, which improves the effect of speech enhancement.

[0070] In the speech processing method of the embodiment of the present disclosure, the amplitude component features and phase component features corresponding to multiple frequency band components corresponding to the speech signal are obtained, which increases the amount of signal that can be obtained by the model. The amplitude component features and phase component features corresponding to each frequency band component are then captured through an attention model with respect to the feature correlation of each frequency band component in time and frequency, so as to strengthen the corresponding important areas in the spectrogram, thereby improving the processing effect of the model. The amplitude correction factor and phase correction factor corresponding to each frequency band component are obtained to correct the amplitude component and phase component corresponding to the frequency band component, and then the target speech signal is obtained through sub-band synthesis, thereby improving the effect of speech enhancement.

[0071] Based on the above embodiments, Figure 4 A flowchart of another speech processing method provided by an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, the method comprises the following steps:

[0072] Step 401: Acquire multiple frequency band components corresponding to a speech signal.

[0073] Step 402: determine amplitude components and phase components corresponding to multiple frequency band components.

[0074] Step 403: extract features of amplitude components and phase components corresponding to the multiple frequency band components to obtain amplitude component features and phase component features corresponding to the multiple frequency band components.

[0075] Step 404, input the amplitude component features and phase component features corresponding to the multiple frequency band components into the encoding network of the attention model to obtain the encoding features corresponding to the multiple frequency band components.

[0076] Among them, the encoding network consists of three layers of two-dimensional convolution, with the number of channels being 16, 32, and 64 respectively.

[0077] As an implementation method, before the amplitude component features corresponding to each frequency band component are input into the attention model, the amplitude component can be compressed, for example, by performing a square root operation, to compress the dynamic range of the amplitude component features so that the model can achieve better performance.

[0078] Specifically, step 401 to step 404 can refer to the explanation in the aforementioned embodiment, the principles are the same, and will not be repeated in this embodiment.

[0079] As an example, Figure 5 A schematic diagram of the structure of an attention model provided in an embodiment of the present disclosure is shown in FIG. Figure 5As shown, the attention model includes an encoding network, an attention network and a decoding network, wherein the attention network can include multiple ones. The output of the previous attention network is restored to the original 2-channel feature data and then used as the input of the next attention network. The accuracy of the fused features is improved through the multi-layer attention network.

[0080] Step 405: For the coding features corresponding to each frequency band component, the coding features are input into the residual module of the attention network to obtain the intermediate coding features.

[0081] In one implementation of the embodiment of the present disclosure, the attention network includes a residual module, a frequency attention module and a frequency transformation module. Figure 6 A schematic diagram of the structure of an attention network provided by an embodiment of the present disclosure, Figure 6 In the figure, a residual block is taken as an example to illustrate.

[0082] In the disclosed embodiment, for the coding features corresponding to each frequency band component in the multiple frequency band components, the coding features are input into the residual module of the attention network. During the process of processing the coding features, the residual model will add the input coding features to supplement some information in the original features that may be lost in the processing process, that is, avoid the loss of intermediate coding feature information and improve the accuracy of the intermediate coding features.

[0083] It should be noted that the residual module in the embodiment of the present disclosure may also be multiple residual modules to improve the accuracy of the intermediate coding features.

[0084] Step 406, input the intermediate coding feature into the frequency attention module of the attention network to obtain a first weighting coefficient, and weight the intermediate coding feature according to the first weighting coefficient to obtain a weighted first coding feature in the time dimension.

[0085] The first weighting coefficient indicates the frequency correlation of the corresponding frequency band component in the time dimension.

[0086] In the disclosed embodiment, for each frequency band component, the frequency attention module calculates the frequency correlation of the corresponding frequency band component in the time dimension based on the intermediate coding feature to obtain a first weighting coefficient for weighting the intermediate coding feature in the time dimension, and multiplies the first weighting coefficient by the intermediate coding feature to output the coding feature in which the important area is strengthened. The frequency attention module captures the correlation between the time dimension and the frequency dimension, and focuses on utilizing the feature information in the time dimension, thereby achieving the capture of the global correlation of each frequency band component and improving the modeling capability of the model.

[0087] Step 407, input the intermediate coding feature into the frequency conversion module of the attention network to obtain a second weighting coefficient, and weight the intermediate coding feature according to the first weighting coefficient to obtain a second coding feature weighted in the frequency dimension.

[0088] The second weighting coefficient indicates the time correlation of the corresponding frequency band component in the frequency dimension.

[0089] In the disclosed embodiment, for each frequency band component, the frequency transformation module calculates the time correlation of the corresponding frequency band component in the frequency dimension based on the intermediate coding feature to obtain a second weighting coefficient for weighting the intermediate coding feature in the frequency dimension, and multiplies the second weighting coefficient by the intermediate coding feature to output the coding feature in which the important area is strengthened. The frequency transformation module captures the correlation between the time dimension and the frequency dimension, and focuses on utilizing the feature information in the frequency dimension, thereby achieving the capture of the global correlation of each frequency band component and improving the modeling capability of the model.

[0090] Step 408, obtaining a fusion feature according to the first coding feature, the second coding feature and the intermediate coding feature.

[0091] In the embodiment of the present disclosure, the first coding feature, the second coding feature and the intermediate coding feature are input into the output module for fusion to obtain the fusion feature, which is specifically achieved in the following manner:

[0092] As an implementation method, the first coding feature, the second coding feature and the intermediate coding feature are concatenated to obtain a fused feature with increased feature dimension, thereby increasing the amount of information contained in the fused feature.

[0093] As another implementation, the first coding feature, the second coding feature and the intermediate coding feature are added to obtain a fusion feature without increasing the feature dimension, which reduces the amount of calculation and improves the processing efficiency while ensuring the amount of feature information.

[0094] Step 409, input the fusion features corresponding to the multiple frequency band components into the decoding network of the attention model to obtain the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components.

[0095] Step 410: correct the amplitude components and phase components corresponding to the multiple frequency band components according to the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components.

[0096] Step 411, performing sub-band synthesis according to the corrected amplitude components and phase components corresponding to the multiple frequency band components to obtain a target speech signal.

[0097] Among them, step 409 to step 411 can refer to the explanation in the above-mentioned embodiment, and the principle is the same, so it will not be repeated in this embodiment.

[0098] In the disclosed embodiment, the amplitude component and phase component of the noisy signal are simultaneously input into the model, which allows the model to obtain more sufficient information compared to the previous single amplitude feature, thereby improving the model performance, performing a square root operation on the amplitude component, compressing the dynamic range of the amplitude, and enabling the model to obtain better performance. The frequency domain attention module and the frequency transformation module are introduced into the set attention network to capture the global correlation of the spectrum, greatly improving the modeling ability of the model and improving the effect of speech enhancement.

[0099] Based on the above embodiments, Figure 7 A flowchart of another image processing method provided in an embodiment of the present disclosure illustrates how to train an attention model, such as Figure 7 As shown, before the above step 104, the following steps are included:

[0100] Step 701, obtaining training samples.

[0101] The training samples include the speech signal to be processed and the standard speech signal.

[0102] A standard speech signal refers to a pure speech signal without interference signals such as noise and reverberation.

[0103] Step 702: Acquire multiple frequency band components corresponding to the speech signal to be processed.

[0104] Step 703: Input multiple frequency band components corresponding to the speech signal to be processed into the attention model to obtain amplitude correction factors and phase correction factors corresponding to the multiple frequency band components.

[0105] Step 704: correct the amplitude components and phase components corresponding to the multiple frequency band components according to the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components.

[0106] Step 705: Perform sub-band synthesis according to the corrected amplitude components and phase components corresponding to the multiple frequency band components to obtain a target speech signal.

[0107] Among them, steps 701-702 can refer to the explanations in the previous embodiment, and the principles are the same, so they will not be repeated in this embodiment.

[0108] Step 706, determine a loss function according to the target speech signal and the standard speech signal, and train the attention model according to the loss function.

[0109] Among them, the loss function includes complex loss function and amplitude loss function.

[0110] The loss function of the disclosed embodiment includes an amplitude loss function and a complex loss function, wherein the complex loss function is determined based on the difference between the real and imaginary parts of the spectrum of the target speech signal and the real and imaginary parts of the spectrum of the standard speech signal, so as to more accurately indicate the difference between the spectrum of the target speech signal and the spectrum of the standard speech signal, thereby achieving stricter constraints on the speech spectrum, so that the attention model can achieve better performance when trained according to the loss function.

[0111] Among them, the loss function can be expressed by the following formula:

[0112]

[0113] Among them, L represents the loss function, where the complex loss function is and The sum of, where Sr is the real part of the spectrum of the target speech signal, and is the real part of the spectrum of the standard speech signal, Si is the imaginary part of the spectrum of the target speech signal, is the imaginary part of the spectrum of the standard speech signal. is the amplitude loss function. Among them, the function MSE() is used to calculate the mean square error.

[0114] In the speech processing method of the embodiment of the present disclosure, the attention model is trained based on the amplitude loss function and the complex loss function, wherein the complex loss function is determined according to the difference between the real and imaginary parts of the spectrum of the target speech signal and the real and imaginary parts of the spectrum of the standard speech signal. The loss function is determined according to the amplitude loss function and the complex loss function to more accurately indicate the difference between the spectrum of the target speech signal and the spectrum of the standard speech signal, and to impose stricter constraints on the spectrum of the speech, so that the attention model can achieve better performance through training.

[0115] In order to implement the above embodiment, the embodiment of the present disclosure also proposes a speech processing device.

[0116] Figure 8 A schematic diagram of the structure of a speech processing device provided in an embodiment of the present disclosure.

[0117] like Figure 8 As shown, the speech processing device 80 may include:

[0118] An acquisition module 81, used to acquire multiple frequency band components corresponding to the speech signal;

[0119] The determination module 82 is used to determine the amplitude components and phase components corresponding to the multiple frequency band components.

[0120] The feature extraction module 83 is used to extract features of the amplitude components and phase components corresponding to the multiple frequency band components to obtain amplitude component features and phase component features corresponding to the multiple frequency band components.

[0121] The processing module 84 is used to input the amplitude component features and the phase component features corresponding to the multiple frequency band components into the attention model, and output the amplitude correction factors and the phase correction factors corresponding to the multiple frequency band components.

[0122] The correction module 85 is used to correct the amplitude components and phase components corresponding to the multiple frequency band components according to the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components.

[0123] The synthesis module 86 is used to perform sub-band synthesis according to the corrected amplitude components and phase components corresponding to the multiple frequency band components to obtain the target speech signal.

[0124] Further, in an implementation of the embodiment of the present disclosure, the processing module 84 is specifically configured to:

[0125] Inputting the amplitude component features and the phase component features corresponding to the multiple frequency band components into the encoding network of the attention model to obtain encoding features corresponding to the multiple frequency band components;

[0126] Inputting the encoding features corresponding to the multiple frequency band components into the attention network of the attention model to obtain fused features corresponding to the multiple frequency band components; wherein the fused features include feature correlations of the corresponding frequency band components in the time dimension and the frequency dimension;

[0127] The fused features corresponding to the multiple frequency band components are input into the decoding network of the attention model to obtain the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components.

[0128] In one implementation of the embodiment of the present disclosure, the processing module 84 is further configured to:

[0129] Inputting the encoding features corresponding to the multiple frequency band components into the residual module of the attention network to obtain intermediate encoding features corresponding to the multiple frequency band components;

[0130] Inputting the intermediate coding features corresponding to the multiple frequency band components into the frequency attention module of the attention network, obtaining first weighting coefficients corresponding to the multiple frequency band components, and weighting the intermediate coding features corresponding to the multiple frequency band components according to the first weighting coefficients corresponding to the multiple frequency band components to obtain first coding features corresponding to the multiple frequency band components weighted in the time dimension; wherein the first weighting coefficient indicates the frequency correlation of the corresponding frequency band components in the time dimension;

[0131] Inputting the intermediate coding features corresponding to the multiple frequency band components into the frequency conversion module of the attention network to obtain second weighting coefficients corresponding to the multiple frequency band components, and weighting the intermediate coding features corresponding to the multiple frequency band components according to the first weighting coefficients corresponding to the multiple frequency band components to obtain second coding features corresponding to the multiple frequency band components weighted in the frequency dimension; wherein the second weighting coefficient indicates the time correlation of the corresponding frequency band components in the frequency dimension;

[0132] According to the first coding features, the second coding features and the intermediate coding features corresponding to the multiple frequency band components, fusion features corresponding to the multiple frequency band components are obtained.

[0133] In one implementation of the embodiment of the present disclosure, the acquisition module 81 is used to:

[0134] Acquire a speech signal; and perform sub-band decomposition on the speech signal to obtain multiple frequency band components corresponding to the speech signal.

[0135] In an implementation of the embodiment of the present disclosure, the device further includes a training module, and the training module is specifically used to:

[0136] Acquire a training sample; the training sample includes a speech signal to be processed and a standard speech signal;

[0137] Acquire multiple frequency band components corresponding to the speech signal to be processed;

[0138] Inputting multiple frequency band components corresponding to the speech signal to be processed into an attention model to obtain amplitude correction factors and phase correction factors corresponding to the multiple frequency band components;

[0139] Correcting the amplitude components and phase components corresponding to the multiple frequency band components according to the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components;

[0140] Perform sub-band synthesis according to the amplitude components and phase components corresponding to the corrected multiple frequency band components to obtain a target speech signal;

[0141] A loss function is determined according to the target speech signal and the standard speech signal, and the attention model is trained according to the loss function; wherein the loss function includes a complex loss function and an amplitude loss function.

[0142] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment, and will not be repeated here.

[0143] In the image processing device of the embodiment of the present disclosure, the amplitude component features and phase component features corresponding to multiple frequency band components corresponding to the speech signal are obtained, which increases the amount of signal that can be obtained by the model. The amplitude component features and phase component features corresponding to each frequency band component are then captured through an attention model with respect to the feature correlation of each frequency band component in time and frequency, so as to strengthen the corresponding important areas in the spectrogram, thereby improving the processing effect of the model. The amplitude correction factor and phase correction factor corresponding to each frequency band component are obtained to correct the amplitude component and phase component corresponding to the frequency band component, and then the target speech signal is obtained through sub-band synthesis, thereby improving the effect of speech enhancement.

[0144] In order to implement the above embodiments, the present disclosure also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the above method embodiments is implemented.

[0145] In order to implement the above embodiments, the present disclosure further proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in the above method embodiments is implemented.

[0146] In order to implement the above embodiments, the present disclosure further proposes a computer program product on which a computer program is stored. When the computer program is executed by a processor, the method described in the above method embodiments is implemented.

[0147] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0148] Fig. 9 Schematic block diagram of an electronic device provided for an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0149] like Fig. 9As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 902 or a computer program loaded from a storage unit 908 to a RAM (Random Access Memory) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An I / O (Input / Output) interface 905 is also connected to the bus 904.

[0150] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0151] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as a speech processing method. For example, in some embodiments, the speech processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the speech processing method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the speech processing method in any other appropriate manner (eg, by means of firmware).

[0152] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0154] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0155] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0156] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0157] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0158] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0159] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0160] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A speech processing method, comprising: Acquire multiple frequency band components corresponding to the speech signal; Determine amplitude components and phase components corresponding to the multiple frequency band components; Extracting features of amplitude components and phase components corresponding to the multiple frequency band components to obtain amplitude component features and phase component features corresponding to the multiple frequency band components; Input the amplitude component features and the phase component features corresponding to the multiple frequency band components into the attention model, and output the amplitude correction factors and the phase correction factors corresponding to the multiple frequency band components; Correcting the amplitude components and phase components corresponding to the multiple frequency band components according to the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components; Sub-band synthesis is performed according to the corrected amplitude components and phase components corresponding to the multiple frequency band components to obtain a target speech signal.

2. The method of claim 1, wherein: The step of inputting the amplitude component features and the phase component features corresponding to the multiple frequency band components into the attention model and outputting the amplitude correction factors and the phase correction factors corresponding to the multiple frequency band components comprises: Inputting the amplitude component features and the phase component features corresponding to the multiple frequency band components into the encoding network of the attention model to obtain encoding features corresponding to the multiple frequency band components; Inputting the encoding features corresponding to the multiple frequency band components into the attention network of the attention model to obtain fused features corresponding to the multiple frequency band components; wherein the fused features include feature correlations of the corresponding frequency band components in the time dimension and the frequency dimension; The fused features corresponding to the multiple frequency band components are input into the decoding network of the attention model to obtain the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components.

3. The method of claim 2, wherein: The step of inputting the encoded features corresponding to the multiple frequency band components into the attention network of the attention model to obtain fused features corresponding to the multiple frequency band components includes: Inputting the encoding features corresponding to the multiple frequency band components into the residual module of the attention network to obtain intermediate encoding features corresponding to the multiple frequency band components; Inputting the intermediate coding features corresponding to the multiple frequency band components into the frequency attention module of the attention network, obtaining first weighting coefficients corresponding to the multiple frequency band components, and weighting the intermediate coding features corresponding to the multiple frequency band components according to the first weighting coefficients corresponding to the multiple frequency band components to obtain first coding features corresponding to the multiple frequency band components weighted in the time dimension; wherein the first weighting coefficient indicates the frequency correlation of the corresponding frequency band components in the time dimension; Inputting the intermediate coding features corresponding to the multiple frequency band components into the frequency conversion module of the attention network to obtain second weighting coefficients corresponding to the multiple frequency band components, and weighting the intermediate coding features corresponding to the multiple frequency band components according to the first weighting coefficients corresponding to the multiple frequency band components to obtain second coding features corresponding to the multiple frequency band components weighted in the frequency dimension; wherein the second weighting coefficient indicates the time correlation of the corresponding frequency band components in the frequency dimension; According to the first coding features, the second coding features and the intermediate coding features corresponding to the multiple frequency band components, fusion features corresponding to the multiple frequency band components are obtained.

4. The method of claim 1, wherein: The step of obtaining a plurality of frequency band components corresponding to the speech signal comprises: Acquire speech signals; The speech signal is decomposed into sub-bands to obtain a plurality of frequency band components corresponding to the speech signal.

5. The method according to any one of claims 1 to 4, wherein: Before inputting the amplitude component features and the phase component features corresponding to the multiple frequency band components into the attention model and outputting the amplitude correction factors and the phase correction factors corresponding to the multiple frequency band components, the method further includes: Acquire a training sample; the training sample includes a speech signal to be processed and a standard speech signal; Acquire multiple frequency band components corresponding to the speech signal to be processed; Inputting multiple frequency band components corresponding to the speech signal to be processed into an attention model to obtain amplitude correction factors and phase correction factors corresponding to the multiple frequency band components; Correcting the amplitude components and phase components corresponding to the multiple frequency band components according to the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components; Perform sub-band synthesis according to the amplitude components and phase components corresponding to the corrected multiple frequency band components to obtain a target speech signal; A loss function is determined according to the target speech signal and the standard speech signal, and the attention model is trained according to the loss function; wherein the loss function includes a complex loss function and an amplitude loss function.

6. A speech processing device, comprising: An acquisition module, used for acquiring multiple frequency band components corresponding to the speech signal; A determination module, used to determine the amplitude components and phase components corresponding to the multiple frequency band components; A feature extraction module, used to extract features of amplitude components and phase components corresponding to the multiple frequency band components, to obtain amplitude component features and phase component features corresponding to the multiple frequency band components; A processing module, used to input the amplitude component features and the phase component features corresponding to the multiple frequency band components into the attention model, and output the amplitude correction factors and the phase correction factors corresponding to the multiple frequency band components; A correction module, used to correct the amplitude components and phase components corresponding to the multiple frequency band components according to the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components; The synthesis module is used to perform sub-band synthesis according to the corrected amplitude components and phase components corresponding to the multiple frequency band components to obtain the target speech signal.

7. The device according to claim 6, wherein: The processing module is specifically used for: Inputting the amplitude component features and the phase component features corresponding to the multiple frequency band components into the encoding network of the attention model to obtain encoding features corresponding to the multiple frequency band components; Inputting the encoding features corresponding to the multiple frequency band components into the attention network of the attention model to obtain fused features corresponding to the multiple frequency band components; wherein the fused features include feature correlations of the corresponding frequency band components in the time dimension and the frequency dimension; The fused features corresponding to the multiple frequency band components are input into the decoding network of the attention model to obtain the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components.

8. The device according to claim 7, wherein: The processing module is further specifically used for: Inputting the encoding features corresponding to the multiple frequency band components into the residual module of the attention network to obtain intermediate encoding features corresponding to the multiple frequency band components; Inputting the intermediate coding features corresponding to the multiple frequency band components into the frequency attention module of the attention network, obtaining first weighting coefficients corresponding to the multiple frequency band components, and weighting the intermediate coding features corresponding to the multiple frequency band components according to the first weighting coefficients corresponding to the multiple frequency band components to obtain first coding features corresponding to the multiple frequency band components weighted in the time dimension; wherein the first weighting coefficient indicates the frequency correlation of the corresponding frequency band components in the time dimension; Inputting the intermediate coding features corresponding to the multiple frequency band components into the frequency conversion module of the attention network to obtain second weighting coefficients corresponding to the multiple frequency band components, and weighting the intermediate coding features corresponding to the multiple frequency band components according to the first weighting coefficients corresponding to the multiple frequency band components to obtain second coding features corresponding to the multiple frequency band components weighted in the frequency dimension; wherein the second weighting coefficient indicates the time correlation of the corresponding frequency band components in the frequency dimension; According to the first coding features, the second coding features and the intermediate coding features corresponding to the multiple frequency band components, fusion features corresponding to the multiple frequency band components are obtained.

9. The device according to claim 6, wherein: The acquisition module is specifically used for: Acquire speech signals; The speech signal is decomposed into sub-bands to obtain a plurality of frequency band components corresponding to the speech signal.

10. The device according to any one of claims 6 to 9, wherein: The device further comprises: a training module; The training module is used to obtain training samples; the training samples include a speech signal to be processed and a standard speech signal; obtain multiple frequency band components corresponding to the speech signal to be processed; input the multiple frequency band components corresponding to the speech signal to be processed into an attention model to obtain amplitude correction factors and phase correction factors corresponding to the multiple frequency band components; correct the amplitude components and phase components corresponding to the multiple frequency band components according to the amplitude correction factors and phase correction factors corresponding to the multiple frequency band components; perform sub-band synthesis according to the amplitude components and phase components corresponding to the corrected multiple frequency band components to obtain a target speech signal; determine a loss function according to the target speech signal and the standard speech signal, and train the attention model according to the loss function; wherein the loss function includes a complex loss function and an amplitude loss function.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-5.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.