Acoustic processing model training method, voice processing method and related products

Abstract

The invention relates to a training method of an acoustic processing model, a voice processing method and a related product. The method comprises the steps of determining a first frequency spectrum of a sample object based on voices of the sample object in different preset time periods; performing fusion processing on the sample phoneme of the sample object in the current time period and the first frequency spectrum through an acoustic processing model to obtain fusion feature information; and performing acoustic processing on the fused feature information to obtain a second frequency spectrum, and training an acoustic processing model based on the difference between the second frequency spectrum and the label frequency spectrum of the sample phonemes. By adopting the method, the synthesized voice timbre is stable and more real.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method for an acoustic processing model, a speech processing method, and related products. Background Art

[0002] Speech processing technology is receiving more and more attention from all walks of life. Common speech processing technologies include speech recognition, semantic analysis, speech synthesis, etc.

[0003] Take speech synthesis as an example. Speech synthesis can convert text or pinyin information into speech output, providing a more convenient way of human-computer interaction. The goal is usually to use acoustic processing models for spectrum generation. The accuracy of spectrum generation by acoustic processing models affects the accuracy of speech synthesis. Therefore, how to train acoustic processing models with higher accuracy has become a hot topic of research. Summary of the invention

[0004] Based on this, it is necessary to provide an acoustic processing model training method, a speech processing method and related products to address the above-mentioned technical problems, which can improve the accuracy of spectrum generation by the acoustic processing model, thereby improving the accuracy of speech synthesis.

[0005] In a first aspect, the present application provides a method for training an acoustic processing model, comprising:

[0006] Determining a first frequency spectrum of the sample object based on the speech of the sample object in different preset time periods;

[0007] The sample phonemes of the sample object in the current time period are fused with the first spectrum through the acoustic processing model to obtain fused feature information; and the fused feature information is acoustically processed to obtain a second spectrum, and the acoustic processing model is trained based on the difference between the second spectrum and the label spectrum of the sample phonemes.

[0008] In a second aspect, the present application also provides a training device for an acoustic processing model, comprising:

[0009] A determination module, configured to determine a first frequency spectrum of the sample object based on the speech of the sample object in different preset time periods;

[0010] The training module is used to fuse the sample phonemes of the sample object in the current time period with the first spectrum through the acoustic processing model to obtain fused feature information; and to acoustically process the fused feature information to obtain a second spectrum, and train the acoustic processing model based on the difference between the second spectrum and the label spectrum of the sample phonemes.

[0011] In a third aspect, the present application provides a speech processing method, comprising:

[0012] Acquire the to-be-processed phonemes and the first spectrum of the to-be-processed object in the current time period, where the first spectrum is determined based on the speech of the to-be-processed object in different preset time periods;

[0013] Performing spectrum conversion processing on the phoneme to be processed and the first spectrum through the acoustic processing model to obtain a target spectrum of the phoneme to be processed; wherein the acoustic processing model is obtained based on a training method of the acoustic processing model;

[0014] Determine the speech corresponding to the target spectrum.

[0015] In a fourth aspect, the present application further provides a speech processing device, comprising:

[0016] An acquisition module, used to acquire the to-be-processed phonemes and a first spectrum of the to-be-processed object in a current time period, where the first spectrum is determined based on the spectrum of the speech of the to-be-processed object in different preset time periods;

[0017] A processing module, used for performing spectrum conversion processing on the phoneme to be processed and the first spectrum through an acoustic processing model to obtain a target spectrum of the phoneme to be processed; wherein the acoustic processing model is obtained based on a training method of the acoustic processing model;

[0018] The determination module is used to determine the speech corresponding to the target spectrum.

[0019] In a fifth aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in any of the above embodiments when executing the computer program.

[0020] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method in any of the above embodiments when the computer program is executed by a processor.

[0021] In a seventh aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the method in any of the above embodiments when executed by a processor.

[0022] In the above-mentioned training process of the acoustic processing model, the first spectrum of the sample object is determined by obtaining the speech of the sample object in different preset time periods, and when training the acoustic processing model, the first spectrum is fused with the sample phonemes in the current time period to obtain fused feature information, and the fused feature information is acoustically processed to obtain the second spectrum, and then the acoustic processing model is trained based on the difference between the second spectrum and the label spectrum of the sample object. It can be seen that in the training process of the acoustic processing model, the first spectrum determined based on the speech of the sample object in different preset time periods is referenced, which is equivalent to training the acoustic processing model by combining the sample phonemes in the current time period with the first spectrum of each speech of the sample object in other different preset time periods, so that the trained acoustic processing model can learn the spectrum dependency between different speech at different times, so as to generate a more accurate spectrum, and then make the speech timbre synthesized by the spectrum generated by the acoustic processing model more stable and more realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] FIG1 is a diagram of an application environment of a training method for an acoustic processing model and a speech processing method in one embodiment;

[0025] FIG2 is a schematic diagram of a flow chart of a method for training an acoustic processing model in one embodiment;

[0026] Figure 3 A schematic diagram of the specific structure of the attention layer in a specific application example;

[0027] Figure 4 A schematic diagram of the specific structure of the scaled dot product attention layer in a specific application example;

[0028] Figure 5 A schematic diagram of a convolution structure in a specific application example;

[0029] Figure 6 is a flow chart of a speech processing method in one embodiment;

[0030] Figure 7 A schematic diagram of the specific process of acoustic processing model training in a specific application example;

[0031] Figure 8is a flowchart of a speech processing method in a specific application example;

[0032] Fig. 9 is a structural block diagram of a training device for an acoustic processing model in an embodiment;

[0033] Fig.10 is a structural block diagram of a speech processing device in one embodiment;

[0034] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0036] Among the technologies related to speech synthesis, the non-autoregressive speech synthesis TTS (Text To Speech) model has attracted more and more attention from the industry and academia. Among the many TTS models, the most popular one is the FastSpeech2 model, which has excellent speech synthesis effect and fast response speed. Compared with the FastSpeech1 model, the FastSpeech2 model abandons the Teacher-Student knowledge distillation framework to reduce the training complexity on the basis of the FastSpeech1 model, and directly uses the Mel spectrum of real speech as the training target to avoid information loss. At the same time, it introduces more accurate duration information and other variable information in speech, such as pitch, volume energy, etc. to improve the quality of synthesized speech. In the application process of the trained FastSpeech2 model, the input is the phoneme to be processed, and the output is the Mel spectrum of the phoneme to be processed. The Mel spectrum is a spectrogram that converts the Mel frequency into the mel scale. The Mel frequency is proposed based on the auditory characteristics of the human ear, and it has a nonlinear corresponding relationship with the Hz frequency.

[0037] Specifically, the FastSpeech2 model includes an embedding layer, an encoding layer, a transform adaptation layer, and a decoding layer. The following is a detailed introduction to the various network structure levels in the FastSpeech2 model in the related art:

[0038] 1. Phoneme Embedding: The embedding layer is used to map the input discrete representation (such as pinyin, etc.) into a continuous space (i.e., real number vector) identifier, that is, to convert the input phoneme into a vector form. In this layer, the text information between the input phonemes can be obtained. The text information here refers to the distance information between the input phonemes, which can be obtained through the phoneme vector.

[0039] 2. Encoder layer: The encoder layer consists of N transformer models, which are used to encode the input vector. The transformer model is a time series model based on the self-attention mechanism. The encoder part of the transformer model can effectively encode the time series information and convert it into a digital signal. Its ability to process time series information is much better than LSTM (Long Short-Term Memory) and is faster. Transformer models are widely used in natural language processing, computer vision, machine translation, and speech recognition. At the input of the encoding layer, since the data is input into the encoding layer at the same time for processing, the position information in the input of the encoding layer is relatively small. Positional encoding processing is required to add the code that can represent the position information to the current input so that the encoding layer knows the position and order of the current input. The encoding layer learns the text feature information between higher-level phonemes, such as syntax, through the attention mechanism of the transformer.

[0040] 3. Variance Adaptor: The Variance Adaptor contains many predictors that can be used to predict the duration, pitch, and energy of speech, so as to add various acoustic features to the input of the Variance Adaptor based on duration, pitch, and energy, thereby better controlling the synthesized speech. In addition, the Variance Adaptor also has a length regular structure that can map the output of the Variance Adaptor to the same length as the Mel spectrum, solving the problem of mismatch between the length of the original phoneme sequence and the spectrum sequence.

[0041] 4. Decoder layer: The decoder layer consists of N transformer models. At the input end of the decoder layer, positional encoding is used to preprocess the input of the decoder layer. Through the decoder layer, more abundant information can be learned to predict the Mel spectrum, and the target Mel spectrum and the predicted Mel spectrum are input into the Loss function to calculate the Loss, so as to perform back propagation of the model and optimize the model. At this time, the difference between the target Mel spectrum and the predicted Mel spectrum will become smaller and smaller.

[0042] In addition, the vocoder model is generally used in conjunction with the FastSpeech2 model. The vocoder model is used to convert the Mel spectrum of the to-be-processed phoneme output by the FastSpeech2 model into speech and output the speech. Currently, the most popular vocoder model is the HiFiGAN (HiFi-Generative Adversarial Network) model, whose synthesized sound quality is far ahead of existing vocoder models. Specifically, it adopts a training strategy based on GAN (Generative Adversarial Network), which includes the Generator model and the Discriminator model during training. Only the Generator model is used during prediction, and the Mel spectrum is input, and the output is the speech corresponding to the Mel spectrum.

[0043] However, the FastSpeech2 model in the related art is trained based on the phonemes input in the current time period and the Mel-spectra corresponding to the phonemes. By using the trained FastSpeech2 model to perform spectrum conversion on the phonemes to be processed, the Mel-spectra corresponding to the phonemes to be processed can be generated. In this process, since the training data of the FastSpeech2 model only includes the spectrum corresponding to the phonemes input in the current time period, the FastSpeech2 model can only learn the spectrum of the phonemes in the current time period during training, and cannot learn the spectrum dependency between different voices of the same object at different times, which in turn causes the Mel-spectra timbre of the phonemes to be processed output by the model to be unstable, and the generated voice is not realistic enough.

[0044] Therefore, the present application proposes a training method for an acoustic processing model and a speech processing method.

[0045] In the whole process, the first spectrum of the sample object is determined by obtaining the speech of the sample object in different preset time periods, and when training the acoustic processing model, the first spectrum is fused with the sample phonemes in the current time period to obtain fused feature information, and the fused feature information is acoustically processed to obtain the second spectrum, and then the acoustic processing model is trained based on the difference between the second spectrum and the label spectrum of the sample object. It can be seen that in the process of training the acoustic processing model, the first spectrum determined based on the speech of the sample object in different preset time periods is referenced, which is equivalent to training the acoustic processing model by combining the sample phonemes in the current time period with the first spectrum of each speech of the sample object in other different preset time periods, so that the trained acoustic processing model can learn the spectrum dependency between different speech at different times, so as to generate a more accurate spectrum, and then make the speech timbre synthesized by the spectrum generated by the acoustic processing model more stable and more realistic.

[0046] The training method of the acoustic processing model and the speech processing method provided in the embodiments of the present application can be performed by a computer device, which may include a terminal and / or a server. The terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented as an independent server or a server cluster consisting of multiple servers, or a cloud server capable of cloud computing.

[0047] The training method of the acoustic processing model and the speech processing method provided in the embodiment of the present application can be applied in the application environment shown in FIG1. ​​In which, the terminal 102 communicates with the server 104 through a network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers.

[0048] During the training of the acoustic processing model, the user selects a sample object identifier on the speech processing platform on the terminal 102, and based on the sample object identifier, the terminal 102 queries the sample phoneme of the sample object corresponding to the sample object identifier and the label spectrum of the sample phoneme from the local, and sends the query information to the server 104. The server 104 obtains the sample phoneme of the sample object and the label spectrum of the sample phoneme; determines the first spectrum of the sample object based on the speech of the sample object in different preset time periods; fuses the sample phoneme with the first spectrum through the acoustic processing model to obtain fused feature information; and acoustically processes the fused feature information to obtain a second spectrum, and trains the acoustic processing model based on the difference between the second spectrum and the label spectrum. The server 104 stores the trained acoustic processing model in a database.

[0049] When performing voice simulation, for example, when the customer service needs to introduce the product to the customer, it will be more time-consuming and laborious to introduce it to multiple customers in sequence. Therefore, an acoustic processing model can be used to simulate the real speaking voice of the customer service to introduce the product. Specifically, the customer service can input the to-be-processed phonemes related to the introduction of the product in the voice processing platform on the terminal 102 in the current time period. When the customer service inputs the to-be-processed phonemes, the customer service identification can also be uploaded to the terminal 102 along with the input action. The terminal 102 generates a voice processing request based on the to-be-processed phonemes and the customer service identification, and forwards the voice processing request to the server 104. The server 104 extracts the to-be-processed phonemes and the customer service identification in the voice processing request, determines the customer service corresponding to the customer service identification, and queries the spectrum of the customer service's voice in different preset time periods, and then determines the first spectrum based on the spectrum of the customer service's voice in different preset time periods. The server 104 obtains the acoustic processing model trained previously from the database, performs spectrum conversion processing on the to-be-processed phonemes and the first spectrum through the acoustic processing model, obtains the target spectrum of the to-be-processed phonemes; and determines the voice corresponding to the target spectrum. The server 104 can push the voice to the terminal 102, and the terminal 102 plays the voice corresponding to the phoneme to be processed.

[0050] In other embodiments, the training method and speech processing method of the acoustic processing model provided in the present application can also be applied separately in a terminal environment. For example, in the training process of the acoustic processing model, the user selects a sample object identifier on the speech processing platform on the terminal, and based on the sample object identifier, the terminal queries the sample phoneme of the sample object corresponding to the sample object identifier and the label spectrum of the sample phoneme from the local to complete the training process of the above-mentioned acoustic processing model, which will not be repeated here. The terminal 102 stores the trained acoustic processing model locally. When performing voice simulation, for example, when it is necessary to simulate the real speaking voice of the customer service and introduce the product, the customer service enters the phoneme to be processed in the voice processing platform on the terminal 102. When the customer service enters the phoneme to be processed, the customer service identifier can also be uploaded to the terminal 102 along with the input action. The terminal 102 obtains the customer service identifier and the phoneme to be processed, determines the object to be processed corresponding to the customer service identifier, and queries the first spectrum determined by the customer service corresponding to the voice in different preset time periods, and processes the queried information and the phoneme to be processed based on the local acoustic processing model to complete the voice processing process, which will not be repeated here.

[0051] In some embodiments, as shown in FIG2 , a method for training an acoustic processing model is provided, and the method is applied to the server 104 in FIG1 as an example for explanation. In which:

[0052] S100, determining a first frequency spectrum of a sample object based on speech of the sample object in different preset time periods.

[0053] Here, the sample is a part of individuals observed or investigated, that is, the sample object in this application is selected from a plurality of predetermined objects and is used to train one or more objects of the acoustic processing model. For example, 10,000 objects are determined in advance from a certain area, and 1,000 objects are randomly selected from the 10,000 objects as sample objects for training the acoustic processing model; for another example, 10,000 objects speaking in a certain time period are determined in advance, and 1,000 objects are randomly selected from the 10,000 objects as sample objects for training the acoustic processing model.

[0054] Spectrum is the abbreviation of frequency spectrum density, which is the distribution curve of frequency. Spectrum in this application includes but is not limited to Mel spectrum or other types of spectrum. The first spectrum is a spectrum generated by splicing the spectra of several voices of the sample object in different preset time periods. The length of the first spectrum can be a fixed length or a variable. In practical applications, different preset time periods are different historical time periods. The first spectrum is often a fixed-length spectrum during model training and application.

[0055] Optionally, the user selects a sample object identifier on the voice processing platform of the terminal, and the terminal queries a number of voices corresponding to the sample object in different preset time periods from the local based on the sample object identifier, and sends the queried number of voices of the sample object in different preset time periods to the server. After obtaining the number of voices of the sample object, the frequency spectra of the number of voices of the sample object in different preset time periods are spliced ​​to generate a first frequency spectrum of the sample object, and the first frequency spectrum is used for training the acoustic processing model.

[0056] Furthermore, the spectra of several voices of the sample object in different preset time periods are spliced ​​together, and the specific steps of generating a first spectrum of the sample object include: extracting the spectra of several voices of the sample object in different preset time periods through a spectrum extraction tool in digital signal processing, and randomly sampling the several spectra, and splicing the spectra obtained after random sampling to form the first spectrum of the sample object.

[0057] S200, fusing the sample phonemes of the sample object in the current time period with the first spectrum through the acoustic processing model to obtain fused feature information; and acoustically processing the fused feature information to obtain a second spectrum, and training the acoustic processing model based on the difference between the second spectrum and the label spectrum of the sample phonemes.

[0058] Among them, the phonemes include pinyin and rhythm. For example, in the case of the phonemes: wo 7 wo he 7 pang, wo, he, and pang here are pinyin, 7 is an identifier representing the rhythm, and the rhythm is the pause time between each pinyin. In practical applications, the rhythm can be represented by each identifier to represent the length of the pause time. For example, 7 can be used to represent a short pause time, 9 can be used to represent a long pause time, and 8 represents a medium pause time. Generally, the phonemes can also include tones. For example, the phonemes to be processed include: wo3 7 wo3 he2 7 pang4, wherein 2, 3, and 4 are tones. In some embodiments, the first tone is 1, the second tone is 2, the third tone is 3, the fourth tone is 4, the light tone is 5, and the modified tone is 6.

[0059] The second spectrum is the training spectrum of the acoustic processing model during the training process. In the field of deep learning, labels are usually used to label target variables or category label spectra in a dataset, that is, the label spectrum is a label used to label the category of sample phonemes during the training process of the acoustic processing model.

[0060] The acoustic processing model in the embodiment of the present application can be a model that is an optimized version of the fastspeech2 acoustic model. The difference from the fastspeech2 acoustic model is that the input of the fastspeech2 acoustic model is the phoneme of the sample object, and the output is the frequency spectrum corresponding to the phoneme of the sample object; while the acoustic processing model in the present application, the input is the first frequency spectrum corresponding to the phoneme of the sample object and the speech of the same object in different preset time periods, and the output is the second frequency spectrum corresponding to the phoneme of the sample object.

[0061] That is to say, the training data of the fastspeech2 acoustic model are the sample phonemes of the sample object and the frequency spectra corresponding to the sample phonemes, while the training data of the acoustic processing model in the present application not only include the sample phonemes of the sample object and the frequency spectra corresponding to the sample phonemes, but also include the first frequency spectra of the sample object determined based on the speech of the sample object in different preset time periods.

[0062] Specifically, the acoustic processing model based on the embodiment of the present application is adopted in that: the first spectrum is the spectrum determined by the speech of the sample object in different preset time periods. By referring to the first spectrum, it is equivalent to training the acoustic processing model by combining the sample phonemes in the current time period with the first spectrum of each speech of the sample object in other different preset time periods. In this way, the trained acoustic processing model can learn the spectral dependency between different speech at different times, so as to generate a more accurate spectrum, and thus make the speech timbre synthesized by the spectrum generated by the acoustic processing model more stable and more realistic.

[0063] In more detail, the user selects a sample object identifier on the voice processing platform of the terminal, and based on the sample object identifier, the terminal queries the sample phoneme of the sample object corresponding to the sample object identifier from the local, determines the label spectrum of the sample phoneme based on the sample phoneme, and sends the query information to the server. The server obtains the sample phoneme of the sample object and the label spectrum of the sample phoneme.

[0064] The sample phoneme and the first spectrum are fused through the acoustic processing model to obtain fused feature information, and the learning process of the sample phoneme for the first spectrum determined by the speech of the sample object in different preset time periods is realized, and the fused feature information is used as the actual input of the acoustic processing. After acoustic processing at multiple network levels, the second spectrum is obtained. Furthermore, based on the loss in the acoustic processing model training process, that is, the difference between the second spectrum and the label spectrum, the acoustic processing model is trained to realize back propagation in the model training process, and when the loss converges, a trained acoustic processing model is obtained.

[0065] In the above-mentioned training process of the acoustic processing model, the first spectrum of the sample object is determined by obtaining the speech of the sample object in different preset time periods, and when training the acoustic processing model, the first spectrum is fused with the sample phonemes in the current time period to obtain fused feature information, and the fused feature information is acoustically processed to obtain the second spectrum, and then the acoustic processing model is trained based on the difference between the second spectrum and the label spectrum of the sample object. It can be seen that in the training process of the acoustic processing model, the first spectrum determined based on the speech of the sample object in different preset time periods is referenced, which is equivalent to training the acoustic processing model by combining the sample phonemes in the current time period with the first spectrum of each speech of the sample object in other different preset time periods, so that the trained acoustic processing model can learn the spectrum dependency between different speech at different times, so as to generate a more accurate spectrum, and then make the speech timbre synthesized by the spectrum generated by the acoustic processing model more stable and more realistic.

[0066] In some embodiments, the acoustic processing model includes a coding layer and an attention layer; the acoustic processing model is used to fuse the sample phonemes of the sample object in the current time period with the first spectrum to obtain fusion feature information, including:

[0067] The sample phonemes of the sample object in the current time period are encoded through the encoding layer to obtain text feature information; the text feature information is fused with the first spectrum through the attention layer to obtain fused feature information.

[0068] Specifically, during the training of the acoustic processing model, first, the sample phonemes of the sample object in the current time period are encoded by the encoding layer in the acoustic processing model to obtain text feature information. Then, the text feature information and the first spectrum are used as inputs of the attention layer, and the attention layer fuses the text feature information with the first spectrum to obtain fused feature information.

[0069] In one embodiment, the present application optimizes the acoustic processing model by providing an attention layer, and using the attention layer to perform a fusion process on the sample phoneme and the first spectrum based on a multi-head attention mechanism. Among them, the multi-head attention is to establish different projection information in multiple different projection spaces, for example, to perform different projections on the input matrix, obtain many output matrices, and then splice them together.

[0070] In the multi-head attention mechanism, q, k, and v represent three different matrices: q matrix (Query), k matrix (Key), and v matrix (Value). The input of the multi-head attention mechanism is the q matrix, k matrix, and v matrix. Through the multi-head attention mechanism, the q matrix and the k matrix are matrix-multiplied to obtain a new attention matrix, and then the fused matrix is ​​matrix-multiplied with the v matrix to obtain feature information with attention information added.

[0071] More specifically, when the text feature information is used as the input of the attention layer, it means that the text feature information is used as the input q of the attention layer, and the first spectrum is used as the input k and v of the attention layer. In this application, k and v have the same features, that is, the first spectrum can be used as the input k of the attention layer, and the first spectrum can also be used as the input v of the attention layer. Through the multi-head attention mechanism of the attention layer, the q matrix (query), k matrix (key) and v matrix (value) are processed to obtain fused feature information that fuses the text feature information and the first spectrum.

[0072] In one embodiment, the specific structure of the attention layer is as follows: Figure 3 As shown in FIG. 1 , it includes a first linear transformation Linear layer, a scaled dot-product attention layer, a concat layer, and a second linear transformation Linear layer. There are three first linear transformation Linear layers, each of which is used for input of the k matrix, the q matrix, and the v matrix. Figure 3 The network structure shown can perform matrix multiplication of the q matrix and the k matrix to obtain an attention matrix, and then perform matrix multiplication of the attention matrix and the v matrix to obtain a feature matrix with attention information added. Among them, the first linear transformation Linear layer and the second linear transformation Linear layer are used to map the input vector to a higher-dimensional feature space through the learning of the weight matrix, that is, to perform a linear transformation on the input. The concat layer is used to concatenate multiple groups of inputs input to the concat layer.

[0073] Furthermore, through the linear transformation of the first linear transformation Linear layer, h groups of different k matrices, q matrices, and v matrices can be obtained, where h>1 and h is a positive integer. Then, the h groups of different k matrices, q matrices, and v matrices are sent to the scaled dot-product attention layer in parallel. Finally, the h groups of aggregated outputs are concatenated together through the concat layer, and linear projection transformation is performed through another learnable second linear transformation Linear layer to produce the final output.

[0074] In one embodiment, the specific structure of the scaled dot product attention layer is as follows: Figure 4 As shown, it includes: a first matrix product Matmul layer, a scale transformation Scale layer, a mask Mask layer, a SoftMax activation function layer, and a second matrix product Matmul layer.

[0075] The specific functions of each layer in the scaled dot product attention layer include: the first matrix product Matmul layer is used to calculate the correlation between the two q and k input vectors, the scale transformation Scale layer is used to scale the correlation, the SoftMax activation function layer is used to normalize the input data, and the Mask layer is a mask layer that can filter certain specific elements. The second matrix product Matmul layer is used to calculate the correlation between the two k and v input vectors, and the linear transformation Linear layer can perform linear transformation on the input features to obtain new feature representations.

[0076] When the attention layer outputs the fused feature information, due to the mechanism of the attention layer, that is, the output of the attention layer is consistent with the dimension of q, the dimension of the output of the attention layer is consistent with the dimension of the output in the encoding layer. Therefore, the fused feature information can be added to the text feature information output by the previous encoding layer as the input of the next layer.

[0077] In the above embodiment, by adding a new attention layer in the acoustic processing model, the multi-head attention mechanism can be used to fuse the text feature information output by the encoding layer with the first spectrum to obtain fused feature information that fuses the first spectrum and the text feature information. Moreover, since the first spectrum is determined based on the speech of the sample object in different preset time periods, the trained acoustic processing model can learn the spectral dependency between different speech at different times, and thus generate a more accurate spectrum with a stronger dependency.

[0078] In some embodiments, the acoustic processing model further includes a convolution layer; the text feature information is fused with the first spectrum through an attention layer to obtain fused feature information, including:

[0079] The first spectrum is subjected to feature extraction processing through the convolution layer to obtain spectrum feature information; the text feature information is fused with the spectrum feature information through the attention layer to obtain fused feature information.

[0080] The convolution layer is used to perform feature extraction processing on the first spectrum. It can efficiently extract local features of the input through convolution operations and parameter sharing.

[0081] Specifically, since the spectrum information cannot be directly fused with the text feature information, the first spectrum needs to be subjected to feature extraction before being input into the attention layer, so as to fuse the text feature information with the spectrum feature information of the first spectrum and obtain fused feature information. Furthermore, the feature extraction of the first spectrum is achieved through a convolutional layer.

[0082] In this embodiment, by setting a convolutional layer in the acoustic processing model, the spectral feature information of the first spectrum can be efficiently extracted and unnecessary information can be filtered out, so that the text feature information of the sample phonemes in the attention layer can be accurately fused with the spectral feature information of the first spectrum, thereby training the acoustic processing model.

[0083] In some embodiments, the convolution layer includes M convolution structures, where M is an integer greater than 1; each convolution structure includes a feature extraction layer; the convolution kernel sizes of the feature extraction layers in different convolution structures are different, and the feature extraction layers with different convolution kernel sizes are used to extract spectrum feature information of different granularities; performing feature extraction processing on the first spectrum through the convolution layer to obtain spectrum feature information includes:

[0084] For each convolution structure, feature extraction processing is performed on the first spectrum through a feature extraction layer in the convolution structure to obtain candidate spectrum feature information; and M pieces of candidate spectrum feature information are concatenated to obtain spectrum feature information.

[0085] Among them, granularity refers to the degree of refinement of a piece of data.

[0086] Specifically, the convolution layer includes at least one convolution structure, and each convolution structure includes a feature extraction layer; the convolution kernel sizes of the feature extraction layers in different convolution structures are different, and the first spectrum is extracted through the feature extraction layers in each convolution structure, that is, the convolution kernels of different sizes, so that candidate spectrum feature information of different granularities can be obtained.

[0087] Furthermore, since only one piece of spectrum feature information needs to be output in the end, the outputs of the M convolution structures can be spliced ​​together, that is, M pieces of candidate spectrum feature information of different granularities can be spliced ​​together to obtain the final spectrum feature information.

[0088] In this embodiment, by using multiple convolution structures to process the first spectrum, and the convolution kernel size in each convolution structure is different, candidate spectrum feature information of different granularities can be obtained, thereby improving the accuracy of feature extraction of the first spectrum, and further obtaining accurate spectrum feature information.

[0089] In some embodiments, the feature extraction layer in each convolution structure includes a first feature extraction layer and a second feature extraction layer having the same convolution kernel; performing feature extraction processing on the first spectrum through the feature extraction layer in the convolution structure to obtain candidate spectrum feature information includes:

[0090] The first spectrum is subjected to feature extraction through the first feature extraction layer of the convolution structure to obtain initial spectrum feature information; the initial spectrum feature information is subjected to activation function processing, and the initial spectrum feature information after the activation function processing is subjected to normalization processing and overfitting processing to obtain spectrum feature processing information; the spectrum feature processing information is subjected to feature extraction through the second feature extraction layer of the convolution structure to obtain spectrum feature update information; the first spectrum is concatenated with the spectrum feature update information to obtain candidate spectrum feature information of the convolution structure.

[0091] In one embodiment, the convolution structure in this application is as follows: Figure 5 As shown, it includes: 2 layers of feature extraction layer Conv1D, activation function layer ReLU, normalization layer BatchNorm, and removal layer Dropout, wherein the 2 layers of feature extraction layer Conv1D include the first feature extraction layer and the second feature extraction layer. In detail, the feature extraction layer Conv1D is used to perform one-dimensional convolution on the input composed of multiple input planes to extract the input features; the activation function in the activation function layer ReLU is the ReLU function, and the derivative of the ReLU function is always 1 when the input is greater than 0, but when the input is less than or equal to 0, the derivative of the ReLU function is 0, that is, the derivative of the ReLU function has only two values ​​of 0 and 1, so it can solve the problem of gradient disappearance during training; the normalization layer BatchNorm can be used to normalize the input features to speed up the convergence of training; the removal layer Dropout can solve the problem of overfitting during training. And in the same convolution structure, the convolution kernels of the two one-dimensional convolution layers have the same size.

[0092] Specifically, feature extraction is performed on the first spectrum through the first feature extraction layer to obtain initial spectrum feature information; activation function processing is performed on the initial spectrum feature information through the activation function layer, and the initial spectrum feature information after activation function processing is normalized and overfitted respectively through the normalization layer and the removal layer to obtain spectrum feature processing information; finally, feature extraction is performed on the spectrum feature processing information through the second feature extraction layer to obtain spectrum feature update information.

[0093] Furthermore, in the process of training the convolution structure, the input of the first feature extraction layer in the convolution structure and the output of the second feature extraction layer in the convolution structure are added to update the output of the second feature extraction layer in the convolution structure. In other words, the spectrum feature update information is concatenated with the spectrum feature to obtain candidate spectrum feature information, which has both coarse-grained features and fine-grained features.

[0094] In this embodiment, the feature extraction of the first spectrum is performed through the internal network structure of the convolution structure, which can accurately and efficiently extract the feature information of the first spectrum, and the final obtained candidate spectrum feature information has both coarse-grained features and fine-grained features, thereby improving the feature richness of the candidate spectrum feature information to better train the attention layer.

[0095] In some embodiments, performing acoustic processing on the fused feature information to obtain a second spectrum includes:

[0096] Text feature extraction is performed on sample phonemes to obtain text feature information; acoustic feature prediction is performed on the fused feature information and the text feature information to obtain speech feature information; and acoustic processing is performed based on the speech feature information to obtain a second spectrum.

[0097] The acoustic processing model also includes an embedding layer and a transform adaptation layer. The output of the embedding layer can be used as the input of the encoding layer, and the output of the encoding layer can be used as the input of the transform adaptation layer. The embedding layer is used to map the discretized representation of the input into a continuous spatial identifier. The transform adaptation layer can introduce the pitch, energy, and duration of the phoneme, so that the final output spectrum is not a simple text feature information, but a spectrum that introduces multiple acoustic feature information.

[0098] Specifically, through the combined action of the embedding layer and the encoding layer, the sample phonemes are subjected to text feature extraction processing to obtain text feature information. That is, the sample phonemes are processed by the embedding layer to obtain text information in the form of vectors corresponding to the sample phonemes, where the text information includes but is not limited to the vector distance information corresponding to the text. The output of the embedding layer is then used as the input of the encoding layer, which encodes the text information in the form of vectors to obtain deeper text feature information, such as syntactic information.

[0099] After the text feature information is fused with the first spectrum, the fused feature information is obtained, and then the fused feature information is spliced ​​with the text feature information output by the encoding layer and input into the transformation adaptation layer. There are several predictors in the transformation adaptation layer, which can be used to predict the pitch, energy, and duration of the sample phoneme to obtain the speech feature information of the sample phoneme. The predictors in the transformation adaptation layer are also trained by deep learning. Furthermore, in the subsequent network structure, the obtained speech feature information is acoustically processed to obtain the second spectrum.

[0100] In this embodiment, text feature extraction is performed on sample phonemes through the embedding layer and the encoding layer, and the extracted text feature information is combined with the fusion feature information. The transform adaptation layer performs acoustic feature prediction, and by introducing information such as the pitch, energy, and duration of the sample phonemes, the subsequently produced second spectrum contains various speech feature information, thereby improving the accuracy of speech synthesis, rather than only the existence of text features.

[0101] In some embodiments, the second spectrum is the second spectrum of the acoustic processing model in the nth round; n is an integer greater than 1; performing acoustic processing based on speech feature information to obtain the second spectrum includes:

[0102] Obtain the second spectrum cutoff ratio updated by the acoustic processing model after the end of the n-1th round of training; if the second spectrum cutoff ratio does not reach a preset cutoff threshold, cut the second spectrum obtained in the n-1th round based on the second spectrum cutoff ratio to obtain the cutoff spectrum of the nth round; splice the cutoff spectrum of the nth round with the speech feature information, and perform acoustic processing on the splicing result to obtain the second spectrum.

[0103] The second spectrum interception ratio refers to the ratio of a portion of the spectrum intercepted from a complete second spectrum. The preset interception threshold can be pre-defined. In practical applications, the preset interception threshold is often 0. As the number of training rounds increases, the preset interception ratio gradually decreases, that is, the preset interception ratio is gradually reduced to the preset interception threshold.

[0104] Specifically, in the related art, when the FastSpeech2 acoustic model is used to generate a spectrum, not only is it impossible to learn the dependency between the spectra of the same object in different time periods, resulting in the model outputting the unstable timbre of the Mel spectrum of the to-be-processed phoneme, but the generated speech is not realistic enough. Therefore, the fusion mechanism of the attention layer can be used to solve this problem, that is, the sample phonemes in the current time period can be combined with the first spectrum of each speech of the sample object in other different preset time periods to train the acoustic processing model, so that the trained acoustic processing model can learn the spectral dependency between different speech at different times, thereby generating a more accurate spectrum; further, in the process of generating the spectrum, the related art generates it at one time, and cannot enhance the dependency between frames in the spectrum, resulting in the lack of dependency between frames in the Mel spectrum of the to-be-processed phoneme output by the model. Therefore, when training the acoustic processing model, the present application will also add a strategy similar to feature reflux to the training process.

[0105] A strategy similar to feature reflow means that in the nth round of training, if the second spectrum interception ratio updated in the n-1th round of training does not reach the preset interception threshold, a fixed percentage of frames in the second spectrum output by the coding layer in the n-1th round of training is randomly extracted according to the second spectrum interception ratio updated in the n-1th round of training to obtain the intercepted spectrum of the nth round. At this time, the second spectrum interception ratio will be reduced by a certain percentage, and the second spectrum interception ratio after the reduction by a certain percentage will be used as the second spectrum interception ratio updated in the nth round of training.

[0106] The intercepted spectrum of the nth round is used as the new input of the encoding layer of the nth round of training process and is spliced ​​with the original input, that is, the intercepted spectrum is spliced ​​with the speech feature information, and the splicing result is acoustically processed to obtain the second spectrum in the nth round process.

[0107] According to the second spectrum clipping ratio updated during the nth round of training, a fixed percentage of frames in the second spectrum output by the coding layer during the nth round of training is randomly extracted to obtain the clipped spectrum of the (n+1)th round, and the above steps are repeated until the second spectrum clipping ratio is reduced to a preset clipping threshold.

[0108] In this embodiment, a strategy similar to feature reflow is added, that is, some frames are extracted from the second spectrum obtained in the previous round of training and fed back to the coding layer, so that the original input of the coding layer can learn the feature information in the second spectrum in the previous round, that is, gradually learn to reconstruct the intrinsic dependency of the spectrum output in this round of training, increase the intrinsic dependency between frames in the generated second spectrum, reduce the impact of the second spectrum being too smooth on the subsequent generated speech, and make the subsequent synthesized speech more realistic.

[0109] In some embodiments, Figure 6 As shown, a speech processing method is provided, and the method is applied to the server 104 in FIG. 1 as an example for explanation. In which:

[0110] S300, obtaining the to-be-processed phonemes and the first spectrum of the to-be-processed object in the current time period.

[0111] The first spectrum is determined based on the spectrum of the speech of the object to be processed in different preset time periods.

[0112] Specifically, the voice processing platform on the terminal obtains the to-be-processed phonemes and the to-be-processed object identifier of the to-be-processed object, generates a voice processing request based on the to-be-processed phonemes and the to-be-processed object identifier, and sends the voice processing request to the server. The server extracts the to-be-processed phonemes and the to-be-processed object identifier in the voice processing request, obtains a number of voices of the to-be-processed object corresponding to the to-be-processed object identifier in different preset time periods, and generates a first spectrum based on the spectrum of the several voices of the to-be-processed object in different preset time periods.

[0113] In practical applications, the tool for generating the first spectrum based on the spectra of several voices of the object to be processed within different preset time periods is generally a spectrum extraction tool in digital signal processing. The spectrum extraction tool in digital signal processing is used to extract the spectra of several voices of the object to be processed within different preset time periods, and randomly sample the several spectra. The spectra obtained after random sampling are spliced ​​to form the first spectrum of the object to be processed.

[0114] S400, performing spectrum conversion processing on the phoneme to be processed and the first spectrum through an acoustic processing model to obtain a target spectrum corresponding to the phoneme to be processed.

[0115] The acoustic processing model is obtained based on the training method of the acoustic processing model. After the acoustic processing model training is completed, the trained acoustic processing model can be stored in a database for subsequent use.

[0116] Specifically, the acoustic processing model is a model obtained through training. It refers to the first spectrum determined based on the speech of the sample object in different preset time periods, which is equivalent to training the acoustic processing model by combining the sample phonemes in the current time period with the first spectrum of each speech of the sample object in other different preset time periods. In this way, the trained acoustic processing model can learn the spectral dependency between different speech at different times, so as to generate a more accurate spectrum, and then make the speech timbre synthesized by the spectrum generated by the acoustic processing model more stable and more realistic. Therefore, by performing spectrum conversion processing on the phoneme to be processed and the first spectrum through the acoustic processing model, the target spectrum corresponding to the phoneme to be processed can be obtained, and the target spectrum has a strong dependency and high accuracy.

[0117] S500, determining the speech corresponding to the target spectrum.

[0118] Specifically, after the acoustic processing model outputs the target spectrum corresponding to the phoneme to be processed, the target spectrum is processed to determine the speech corresponding to the target spectrum to achieve the final speech output. When the speech corresponding to the target spectrum is obtained, since the target spectrum is a spectrum that integrates the feature information of the phoneme to be processed and the feature information of the first spectrum, the output speech can integrate the spectral dependency between different speeches of the same object at different times, and the output speech has a stable timbre and a sense of reality.

[0119] In the above-mentioned speech processing method, during the training of the acoustic processing model, reference is made to the first spectrum determined based on the speech of the sample object in different preset time periods, which is equivalent to training the acoustic processing model by combining the sample phonemes in the current time period with the first spectrum of each speech of the sample object in other different preset time periods. In this way, the trained acoustic processing model can learn the spectral dependency between different speech at different times, thereby generating a more accurate spectrum, and thus making the speech timbre synthesized by the spectrum generated by the acoustic processing model more stable and realistic.

[0120] In some embodiments, determining the speech corresponding to the target spectrum includes:

[0121] The target spectrum is processed by speech generation model to obtain the speech corresponding to the target spectrum.

[0122] The speech generation model is trained based on clean speech within a preset time period, a clean speech spectrum of clean speech, and a noisy speech spectrum of clean speech.

[0123] Specifically, the speech corresponding to the target spectrum is determined by a speech generation model. The speech generation model performs speech generation processing on the target spectrum to obtain the speech corresponding to the target spectrum. A common speech generation model is a vocoder.

[0124] The speech generation model in this application is different from the existing speech generation model, and some optimization processing has been performed on it, as follows:

[0125] In the related art, when training a speech generation model, clean speech and the spectrum corresponding to the clean speech are often used as training data, so that the trained speech generation model supports outputting the clean speech corresponding to the spectrum when the spectrum is input. However, once the spectrum corresponding to the noise is input, the sound quality of the generated speech will deteriorate and lack a certain degree of robustness.

[0126] Therefore, in order to solve the above problems in the related art, the present application uses training data including: clean speech within a preset time period, clean speech spectrum of the clean speech, and noisy speech spectrum corresponding to the clean speech when training the speech generation model. Further, obtaining the noisy speech spectrum corresponding to the clean speech includes: adding a certain amount of noise to the clean speech to obtain the noisy speech, and obtaining the noisy speech spectrum corresponding to the noisy speech.

[0127] The clean speech is used as the label of the noisy speech spectrum and the clean speech spectrum, and the speech generation model is trained based on the clean speech, the noisy speech spectrum, and the clean speech spectrum. The trained speech generation model supports the output of clean speech regardless of whether the noisy speech spectrum or the clean speech spectrum is input. In other words, when the noisy speech spectrum is processed by the speech generation model, the model outputs clean speech; when the clean speech spectrum is processed by the speech generation model, the model also outputs clean speech.

[0128] Furthermore, when training a speech generation model based on clean speech, clean speech spectrum of clean speech, and noisy speech spectrum of clean speech within a preset time period, it is necessary to mix the clean speech spectrum with the noisy speech spectrum in a certain ratio. The mixing ratio needs to be debugged while training the speech generation model, so that the obtained speech generation model has anti-noise performance, and when the amount of training data is different, the mixing ratio of the clean speech spectrum and the noisy speech spectrum in the training data is also different.

[0129] When debugging the ratio, it is also possible to pre-configure training data of multiple mixing ratios, and train the speech generation models separately based on the training data of multiple mixing ratios, and compare the model effects of the trained multiple speech generation models to obtain the mixing ratio corresponding to the model with the best training effect.

[0130] In this embodiment, the speech generation model obtained by training based on clean speech within a preset time period, the clean speech spectrum of clean speech, and the noisy speech spectrum of clean speech can make the trained speech generation model have noise resistance performance, so that no matter whether the noisy speech spectrum or the clean speech spectrum is input, the output is clean speech, which improves the noise resistance and stability of speech synthesis.

[0131] In one embodiment, the speech processing method further includes:

[0132] Noise detection is performed on a speech set to obtain an initial clean speech; the speech in the initial clean speech is combined in pairs to obtain a clean speech pair; a target clean speech pair is determined based on the clean speech pair; and clean speech within a preset time period is determined based on the target clean speech pair.

[0133] The speech collection includes open source datasets and proprietary datasets. The open source datasets may contain noise. The speech similarity between the two voices in the target clean speech pair is greater than the preset similarity threshold.

[0134] Specifically, since there may be noise in the open source dataset, it is necessary to use noise detection methods to detect the noise content of the data in the open source dataset, extract as much speech as possible with the total duration and number of speakers within a specific time period, avoid extracting speech with much noise, and obtain the initial clean speech.

[0135] Furthermore, the initial clean speech is evenly divided to obtain clean speech pairs. The means for evenly dividing the initial clean speech includes: combining the speech in the initial clean speech in pairs to obtain clean speech pairs, and obtaining a speaker recognition model, which is used to calculate the similarity of the two speech in the input speech pair. The clean speech pairs can be filtered through the speaker recognition model.

[0136] The specific process of calculating the similarity of two voices in an input voice pair through a speaker recognition model includes: inputting a clean voice pair into the speaker recognition model, the speaker recognition model obtains the vectors corresponding to the two clean voices in the clean voice pair, and calculates the distance between the vectors corresponding to the two clean voices in the clean voice pair, and obtains the voice similarity between the two clean voices in the clean voice pair through the distance, if the voice similarity is greater than a preset similarity threshold, the clean voice pair is determined as a target clean voice pair, and any one vector in the target clean voice pair is filtered, and then the clean voice is obtained based on the filtered target clean voice pair.

[0137] In this embodiment, on the one hand, by performing noise detection on the speech set, clean speech can be obtained to avoid interference caused by noisy speech; on the other hand, by filtering the clean speech through speech similarity, the timbre similarity of each speaker in the training data can be made as low as possible, thereby improving robustness and allowing the trained model to be directly used in the timbre cloning scenario without repeated training.

[0138] In some embodiments, the input of the acoustic processing model in this application can be not only phonemes and the first spectrum, but also text and the first spectrum. However, when inputting text and the first spectrum, the text needs to be converted into phonemes or modeled by Chinese characters. And the optimization points proposed in this application are not limited to optimization on the fastspeech2 acoustic model, but can also be extended to models such as Vits and Tacotron 1 / 2.

[0139] In some embodiments, the speech processing of the present application can be regarded as a certain optimization of the fastspeech2 acoustic model and the speech generation model to make the generated speech more realistic. The fastspeech2 acoustic model includes an embedding layer, a coding layer, a transform adaptation layer, and a decoding layer. The optimized acoustic processing model includes an embedding layer, a coding layer, a convolution layer, an attention layer, a transform adaptation layer, and a decoding layer. Before the coding layer and the decoding layer, a position coding process is also set. The position coding adds the code that can represent the position information to the current input, so that the subsequent layers know the position and order of the current input.

[0140] Taking the spectrum as Mel spectrum and the speech generation model as vocoder as an example, from the perspective of the specific acoustic processing model training method and speech processing method, the optimization of this application is reflected in the following aspects:

[0141] 1. Optimization of acoustic processing model:

[0142] like Figure 7 The specific training process of the optimized acoustic processing model is shown. When the acoustic processing model is trained, the label spectrum is used as the label of the sample phoneme, and the acoustic processing model is trained based on the sample phoneme of the sample object, the first spectrum of the sample object, and the label spectrum of the sample phoneme to obtain the trained acoustic processing model. Among them, the first spectrum and the label spectrum of the phoneme to be trained are both Mel spectra corresponding to the same speaker, but the first spectrum is a Mel spectrum determined based on the voice of the sample object in different preset time periods. In other words, the present application will add the first spectrum as training data, so that the acoustic processing model can refer to the first spectrum determined based on the voice of the sample object in different preset time periods, which is equivalent to training the acoustic processing model by combining the sample phonemes in the current time period with the first spectrum of each voice of the sample object in other different preset time periods. In this way, the trained acoustic processing model can learn the spectrum dependency between different voices at different times, so that a more accurate spectrum can be generated, and the voice timbre synthesized by the spectrum generated by the acoustic processing model is more stable and more realistic.

[0143] In order to achieve this technical effect, the present application adds a convolution layer and an attention layer to the acoustic processing model, uses the output of the encoding layer in the acoustic processing model as a q matrix input of the attention layer, uses the convolution layer to extract features from the Mel spectrum of the first spectrum, and uses the output of the convolution layer as the other two k matrices and v matrix inputs of the attention layer, so that the attention layer realizes attention fusion processing of the q matrix, the k matrix and the v matrix, thereby outputting fused feature information that fuses the spectral feature information of the additional Mel spectrum and the text feature information of the sample phonemes. Since the output dimension of the attention layer is consistent with the output dimension of the encoding layer, the fused feature information can be combined with the text feature information originally output by the encoding layer and input into the transform adaptation layer for subsequent processing.

[0144] Furthermore, the acoustic processing model of the present application can not only learn the spectral dependency between different voices at different times, but also add a feature reflux-like strategy between the output and input ends of the decoding layer of the acoustic processing model, guiding the decoding layer to perform iterative optimization learning, which helps the acoustic processing model learn the intrinsic dependency between frames in the Mel spectrum. Specifically, in the nth round of training, if the second spectrum interception ratio updated in the n-1th round of training does not reach the preset interception threshold, according to the second spectrum interception ratio updated in the n-1th round of training, a fixed percentage of frames in the second spectrum output by the encoding layer in the n-1th round of training are randomly extracted to obtain the intercepted spectrum of the nth round. At this time, the second spectrum interception ratio will be reduced by a certain proportion, and the second spectrum interception ratio after the reduction by a certain proportion will be used as the second spectrum interception ratio updated in the nth round of training. The intercepted spectrum of the nth round is used as the new input of the encoding layer of the nth round of training, and is spliced ​​with the original input, that is, the intercepted spectrum is spliced ​​with the speech feature information, and the splicing result is acoustically processed to obtain the second spectrum in the nth round. Repeat the above steps, and gradually reduce the percentage of the second spectrum interception ratio as the number of training steps increases until it reaches 0. Based on the loss calculation of the updated training Mel spectrum and the label spectrum, the trained acoustic processing model is obtained. Through this measure, the intrinsic dependencies in the reconstructed spectrum can be gradually learned, the intrinsic dependencies between frames in the Mel spectrum can be increased, and the impact of oversmoothing on the subsequent generated speech can be reduced, making the subsequent synthesized speech more realistic.

[0145] 2. Vocoder optimization:

[0146] First, the training data is filtered, that is, the noisy speech is filtered and the speech with high timbre similarity is filtered out from the clean speech. Specifically, the noisy speech is filtered out by means of noise detection, and the speech with high timbre similarity is filtered out by means of a deep learning model, such as a speaker recognition model. By filtering the training data, the robustness of the model can be improved, so that the timbre similarity of each object in the training data is as low as possible.

[0147] Secondly, during the training process, the method of speech enhancement can also be used. The Mel spectrum of clean speech and the Mel spectrum of noisy speech are mixed in proportion, where the noisy speech is the noisy speech corresponding to the clean speech after adding noise. The Mel spectrum of clean speech and the Mel spectrum of noisy speech mixed in proportion, as well as the clean speech, are used as training data to train the vocoder to obtain a trained vocoder. The trained vocoder supports the output of clean speech regardless of whether the input is the Mel spectrum of noisy speech or the Mel spectrum of clean speech, thereby improving the anti-noise performance of the vocoder.

[0148] In some embodiments, the steps of the speech processing method in the present application are as follows: Figure 8 As shown, specifically including:

[0149] The phoneme to be processed and the first spectrum are input into the trained acoustic processing model, the phoneme to be processed and the first spectrum are processed by the acoustic processing model, and the acoustic processing model outputs the target spectrum of the phoneme to be processed; the target spectrum is input into the vocoder, the target spectrum is processed by the vocoder, and the speech corresponding to the target spectrum is directly output.

[0150] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0151] Based on the same inventive concept, the embodiment of the present application also provides a training device for an acoustic processing model for implementing the training method of the acoustic processing model involved above, and a speech processing device for implementing the speech processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more acoustic processing models provided below can refer to the limitations of the training method for the acoustic processing model above, and the specific limitations in the embodiments of one or more speech processing devices can refer to the limitations of the speech processing method above, which will not be repeated here.

[0152] In some embodiments, Fig. 9 As shown, a training device for an acoustic processing model is provided, comprising: a determination module 100 and a training module 200, wherein:

[0153] A determination module 200, configured to determine a first frequency spectrum of a sample object based on the speech of the sample object in different preset time periods;

[0154] The training module 200 is used to fuse the sample phonemes of the sample object in the current time period with the first spectrum through the acoustic processing model to obtain fused feature information; and to acoustically process the fused feature information to obtain a second spectrum, and train the acoustic processing model based on the difference between the second spectrum and the label spectrum of the sample phonemes.

[0155] In one embodiment, the acoustic processing model includes an encoding layer and an attention layer; the training module 200 is also used to encode the sample phonemes of the sample object in the current time period through the encoding layer to obtain text feature information; and fuse the text feature information with the first spectrum through the attention layer to obtain fused feature information.

[0156] In one embodiment, the acoustic processing model also includes a convolution layer; the training module 200 is also used to perform feature extraction processing on the first spectrum through the convolution layer to obtain spectrum feature information; and fuse the text feature information with the spectrum feature information through the attention layer to obtain fused feature information.

[0157] In one embodiment, the convolution layer includes M convolution structures, where M is an integer greater than 1; each convolution structure includes a feature extraction layer; the convolution kernel sizes of the feature extraction layers in different convolution structures are different, and the feature extraction layers with different convolution kernel sizes are used to extract spectrum feature information of different granularities; the training module 200 is also used to perform feature extraction processing on the first spectrum through the feature extraction layer in the convolution structure for each convolution structure to obtain candidate spectrum feature information; the M candidate spectrum feature information is spliced ​​to obtain spectrum feature information.

[0158] In one embodiment, the feature extraction layer in each convolution structure includes a first feature extraction layer and a second feature extraction layer having the same convolution kernel; the training module 200 is also used to perform feature extraction on the first spectrum through the first feature extraction layer of the convolution structure to obtain initial spectrum feature information; perform activation function processing on the initial spectrum feature information, and perform normalization and overfitting processing on the initial spectrum feature information after the activation function processing to obtain spectrum feature processing information; perform feature extraction on the spectrum feature processing information through the second feature extraction layer of the convolution structure to obtain spectrum feature update information; and concatenate the first spectrum with the spectrum feature update information to obtain candidate spectrum feature information of the convolution structure.

[0159] In one embodiment, the training module 200 is also used to extract text features from sample phonemes to obtain text feature information; perform acoustic feature prediction on the fused feature information and the text feature information to obtain speech feature information; and perform acoustic processing based on the speech feature information to obtain a second spectrum.

[0160] In one embodiment, the second spectrum is the second spectrum of the acoustic processing model in the nth round; n is an integer greater than 1; the training module 200 is also used to obtain the second spectrum cutoff ratio updated after the acoustic processing model is trained in the n-1th round; when the second spectrum cutoff ratio does not reach a preset cutoff threshold, based on the second spectrum cutoff ratio, the second spectrum of the n-1th round is cut off to obtain the cutoff spectrum of the nth round; the cutoff spectrum of the nth round is spliced ​​with the speech feature information, and the splicing result is acoustically processed to obtain the second spectrum.

[0161] Each module in the training device of the acoustic processing model can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0162] In some embodiments, Fig.10 As shown, a speech processing device is provided, comprising: an acquisition module 300, a processing module 400 and a determination module 500, wherein:

[0163] An acquisition module 300 is used to acquire a to-be-processed phoneme and a first spectrum of the to-be-processed object in a current time period, where the first spectrum is determined based on the spectrum of the speech of the to-be-processed object in different preset time periods;

[0164] The processing module 400 is used to perform spectrum conversion processing on the to-be-processed phoneme and the first spectrum through an acoustic processing model to obtain a target spectrum corresponding to the to-be-processed phoneme; wherein the acoustic processing model is obtained based on a training method of the acoustic processing model;

[0165] The determination module 500 is used to determine the speech corresponding to the target spectrum.

[0166] In one embodiment, the determination module 500 is also used to perform speech generation processing on the target spectrum through a speech generation model to obtain speech corresponding to the target spectrum; the speech generation model is trained based on clean speech within a preset time period, a clean speech spectrum of clean speech, and a noisy speech spectrum of clean speech.

[0167] In one embodiment, the speech processing device also includes a filtering module, which is used to perform noise detection on a speech set to obtain an initial clean speech; combine the speech in the initial clean speech in pairs to obtain a clean speech pair; determine a target clean speech pair based on the clean speech pair, wherein the speech similarity between the two speech in the target clean speech pair is greater than a preset similarity threshold; and determine the clean speech based on the target clean speech pair.

[0168] Each module in the above-mentioned speech processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0169] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as label spectrum. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a speech processing method is implemented.

[0170] Those skilled in the art will understand that Fig.11The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0171] In some embodiments, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0172] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0173] In some embodiments, a computer program product is provided, including a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0174] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0175] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for an acoustic processing model, It is characterized in that Said include: Determining a first frequency spectrum of the sample object based on the speech of the sample object in different preset time periods; fusing the sample phonemes of the sample object in the current time period with the first spectrum through an acoustic processing model to obtain fusion feature information; And, acoustic processing is performed on the fused feature information to obtain a second spectrum, and the acoustic processing model is trained based on the difference between the second spectrum and the label spectrum of the sample phoneme.

2. The method according to claim 1, It is characterized in that The acoustic processing model includes a coding layer and an attention layer; the fusing process of the sample phonemes of the sample object in the current time period with the first spectrum by the acoustic processing model to obtain fusion feature information includes: Encoding the sample phonemes of the sample object in the current time period through the encoding layer to obtain text feature information; The text feature information is fused with the first spectrum through the attention layer to obtain the fused feature information.

3. The method according to claim 2, It is characterized in that The acoustic processing model further includes a convolution layer; fusing the text feature information with the first spectrum through the attention layer to obtain the fused feature information, including: Performing feature extraction processing on the first spectrum through the convolution layer to obtain spectrum feature information; The text feature information and the spectrum feature information are fused through the attention layer to obtain the fused feature information.

4. The method according to claim 3, It is characterized in that The convolution layer includes M convolution structures, where M is an integer greater than 1; each convolution structure includes a feature extraction layer; the convolution kernels of the feature extraction layers in different convolution structures have different sizes, and the feature extraction layers with different convolution kernel sizes are used to extract spectrum feature information of different granularities; the feature extraction processing of the first spectrum through the convolution layer to obtain the spectrum feature information includes: For each convolution structure, performing feature extraction processing on the first spectrum through a feature extraction layer in the convolution structure to obtain candidate spectrum feature information; The M candidate spectrum feature information are concatenated to obtain the spectrum feature information.

5. The method according to claim 4, It is characterized in that The feature extraction layers in each convolution structure include a first feature extraction layer and a second feature extraction layer having the same convolution kernel; the feature extraction layer in the convolution structure performs feature extraction processing on the first spectrum to obtain candidate spectrum feature information, including: Extracting features from the first spectrum through the first feature extraction layer of the convolution structure to obtain initial spectrum feature information; Performing activation function processing on the initial spectrum feature information, and performing normalization processing and overfitting processing on the initial spectrum feature information after the activation function processing, to obtain spectrum feature processing information; Performing feature extraction on the spectrum feature processing information through the second feature extraction layer of the convolution structure to obtain spectrum feature update information; The first spectrum is concatenated with the spectrum feature update information to obtain candidate spectrum feature information of the convolution structure.

6. The method according to any one of claims 1 to 5, It is characterized in that The step of performing acoustic processing on the fused feature information to obtain a second spectrum includes: Performing text feature extraction on the sample phonemes to obtain text feature information; Performing acoustic feature prediction on the fused feature information and the text feature information to obtain speech feature information; Acoustic processing is performed based on the speech feature information to obtain a second spectrum.

7. The method according to claim 6, It is characterized in that The second spectrum is the second spectrum of the acoustic processing model in the nth round; n is an integer greater than 1; and the acoustic processing based on the speech feature information to obtain the second spectrum includes: Obtaining a second spectrum interception ratio updated by the acoustic processing model after the end of the n-1th round of training; When the second spectrum interception ratio does not reach a preset interception threshold, intercepting the second spectrum of the (n-1)th round based on the second spectrum interception ratio to obtain the intercepted spectrum of the nth round; The intercepted spectrum of the nth round is spliced ​​with the speech feature information, and the spliced ​​result is acoustically processed to obtain the second spectrum.

8. A speech processing method, It is characterized in that The method comprises: Acquire a to-be-processed phoneme and a first spectrum of the to-be-processed object in a current time period, where the first spectrum is determined based on the speech of the to-be-processed object in different preset time periods; Performing spectrum conversion processing on the to-be-processed phoneme and the first spectrum through an acoustic processing model to obtain a target spectrum of the to-be-processed phoneme; wherein the acoustic processing model is obtained based on the method according to any one of claims 1 to 7; Determine the speech corresponding to the target spectrum.

9. The method according to claim 8, It is characterized in that The determining the speech corresponding to the target spectrum includes: The target spectrum is processed for speech generation through a speech generation model to obtain speech corresponding to the target spectrum; the speech generation model is trained based on clean speech within a preset time period, a clean speech spectrum of the clean speech, and a noisy speech spectrum of the clean speech.

10. The method according to claim 9, It is characterized in that The method further comprises: Perform noise detection on the speech set to obtain initial clean speech; Combining the initial clean speech in pairs to obtain clean speech pairs; Determine a target clean speech pair according to the clean speech pair, wherein the speech similarity between the two speech in the target clean speech pair is greater than a preset similarity threshold; A clean speech is determined according to the target clean speech pair.

11. A training device for an acoustic processing model, It is characterized in that The device comprises: A determination module, configured to determine a first frequency spectrum of the sample object based on the speech of the sample object in different preset time periods; A training module is used to fuse the sample phonemes of the sample object in the current time period with the first spectrum through an acoustic processing model to obtain fused feature information; and to acoustically process the fused feature information to obtain a second spectrum, and train the acoustic processing model based on the difference between the second spectrum and the label spectrum of the sample phonemes.

12. A speech processing device, It is characterized in that The device comprises: An acquisition module, used to acquire a to-be-processed phoneme and a first spectrum of the to-be-processed object in a current time period, wherein the first spectrum is determined based on the spectrum of the speech of the to-be-processed object in different preset time periods; A processing module, used for performing spectrum conversion processing on the to-be-processed phoneme and the first spectrum through an acoustic processing model to obtain a target spectrum of the to-be-processed phoneme; wherein the acoustic processing model is obtained based on the method according to any one of claims 1 to 7; The determination module is used to determine the speech corresponding to the target spectrum.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

15. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images