Speech synthesis model generation method and its device, equipment, medium, and product

Abstract

The present application relates to a method for generating a speech synthesis model and its apparatus, equipment, medium, and product, the method comprising: calling a controller, the controller generating a structural code of a vocoder; constructing a vocoder according to the structural code, the vocoder comprising a conditional network and an autoregressive network generated according to the structural code; iteratively training the vocoder to a convergence state using a training set, and implementing a gradient update on the controller and iteratively generating a new vocoder according to the performance score obtained by the vocoder on a test set before the controller reaches convergence; after the controller reaches convergence, selecting the vocoder as a speech synthesis model according to the performance score. The present application uses a controller to realize the automatic generation and optimization of vocoders, so that the obtained speech synthesis model meets the requirements of mobile terminal device deployment, can obtain good performance after the mobile terminal device is deployed, and can meet the requirements for model miniaturization and high real-time performance in speech synthesis scenarios.

Description

Technical Field

[0001] The present application relates to the field of voice communication technology, and in particular to a method for generating a speech synthesis model and its apparatus, equipment, medium, and product. Background Art

[0002] As the number of online audio users continues to grow, users have higher and higher requirements for online audio content, entertainment attributes, transmission quality, etc. Deploying deep speech synthesis models in different application scenarios and network environments not only requires the model to have better inference effects, but also places higher requirements on the real-time performance of model operations.

[0003] The deep speech synthesis model has sufficient representation capabilities, but there are many problems if it is to be deployed on a mobile terminal. For example, on the one hand, the deep speech synthesis model generally has more parameters. For example, the WaveNet model has 4.6M parameters, the WaveGlow model has 87.9M parameters, and the FloWaveNet model has 182.6M parameters, which far exceeds the computing power of the mobile terminal. On the other hand, in some application scenarios based on speech synthesis, such as packet loss compensation, the model is required to reach or exceed the speed of real-time synthesis.

[0004] Traditional mobile speech synthesis methods generally use concatenated synthesis, parameter synthesis, or shallow learning to achieve real-time effects on mobile devices. These methods are usually limited by the number of model parameters and the number of floating point operations per second (FLOPS). In practice, they often fail to meet the requirements of miniaturization and real-time performance, and often require manual compression and pruning, or large-scale computing power deployment in the background. Summary of the invention

[0005] The purpose of the present application is to solve the above-mentioned problems and to provide a method for generating a speech synthesis model and its corresponding apparatus, device, non-volatile readable storage medium, and computer program product.

[0006] According to one aspect of the present application, a method for generating a speech synthesis model is provided, comprising the following steps:

[0007] calling a controller, and generating a structural code of the vocoder by the controller;

[0008] Constructing a vocoder according to the structural coding, the vocoder comprising a conditional network and an autoregressive network generated according to the structural coding;

[0009] Iteratively training the vocoder to a converged state using a training set, and performing a gradient update on the controller and iteratively generating a new vocoder before the controller reaches convergence according to a performance score of the vocoder on a test set;

[0010] After the controller reaches convergence, the vocoder is selected as the speech synthesis model based on the performance score.

[0011] According to another aspect of the present application, there is provided a speech synthesis model generating device, comprising:

[0012] A code generation module is configured to call the controller, and the controller generates a structural code of the vocoder;

[0013] A vocoder construction module, configured to construct a vocoder according to the structural coding, the vocoder comprising a conditional network and an autoregressive network generated according to the structural coding;

[0014] an iterative decision module, configured to iteratively train the vocoder to a converged state using a training set, and to perform a gradient update on the controller and iteratively generate a new vocoder before the controller reaches convergence according to a performance score obtained by the vocoder on a test set;

[0015] The model output module is configured to select the vocoder as the speech synthesis model based on the performance score after the controller reaches convergence.

[0016] According to another aspect of the present application, a speech synthesis model generation device is provided, comprising a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the speech synthesis model generation method described in the present application.

[0017] According to another aspect of the present application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the speech synthesis model generation method in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.

[0018] According to another aspect of the present application, a computer program product is provided, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the steps of the method described in any one of the embodiments of the present application are implemented.

[0019] Compared with the prior art, the present application adopts a controller to generate structural codes, constructs a vocoder according to the structural codes, trains and tests the vocoder to obtain corresponding performance scores, controls the iterative process of the controller according to the performance scores, generates multiple vocoders during the training process of the controller, and finally selects the vocoder with the best measured performance as the speech synthesis model. With the help of the controller, the automatic generation and selection of vocoders are realized, so that the obtained speech synthesis model meets the requirements of mobile terminal device deployment, can obtain good performance after being deployed on the mobile terminal device, and can meet the requirements of model miniaturization and high real-time performance in speech synthesis scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A schematic diagram of the network architecture corresponding to the voice call service used in this application;

[0022] Figure 2 It is a principle block diagram of the topological structure of the vocoder of the present application;

[0023] Figure 3 A flowchart of an embodiment of a method for generating a speech synthesis model of the present application;

[0024] Figure 4 A flowchart of an exemplary training process of an encoder generated by the controller of the present application;

[0025] Figure 5 A schematic diagram of the processing process before deploying the speech synthesis model in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of a process for implementing multi-stage training on a speech synthesis model in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of a process for implementing the third stage of training for a speech synthesis model in an embodiment of the present application;

[0028] Figure 8 This is a principle block diagram of the speech synthesis model generation device of the present application;

[0029] Fig. 9 A schematic diagram of the structure of a speech synthesis model generation device used in this application. DETAILED DESCRIPTION

[0030] The models referenced or may be referenced in this application, including traditional machine learning models or deep learning models, can be deployed on a remote server and remotely called on the client, or deployed on a client with sufficient device capabilities and directly called, unless explicitly specified. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0031] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0032] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0033] See also Figure 1 The network architecture adopted in the exemplary application scenario of the present application can be used to deploy a voice call service. The voice call service supports real-time voice communication. During the encoding and decoding process of the voice stream of the voice call service, packet loss compensation can be achieved by running the speech synthesis model generated by any one of the embodiments of the present application. Figure 1 The application server 81 shown can be used to support the operation of the voice call service, and the media server 82 can be used to process the decoding process of the voice stream pushed by each user to achieve relay. The terminal devices such as the computer 83 and the mobile phone 84 are generally provided to the terminal user as clients and can be used to send or receive voice streams. In addition, when it is necessary to encode and decode the voice stream on the terminal device, the speech synthesis model obtained by each embodiment of the present application can also be deployed in the terminal device to compensate for packet loss of the received or sent voice stream. The application scenarios of the speech synthesis model disclosed above are for illustrative purposes only. For example, it can also be used in a live broadcast service scenario to compensate for packet loss of the voice stream in the live broadcast stream.

[0034] Figure 2 The principle block diagram of the vocoder generated by the present application is shown, which provides a fixed topology structure. The vocoder of the present application can be constructed according to the topology structure, and at least one vocoder can be selected as a speech synthesis model.

[0035] according to Figure 2 As shown in the topology, the vocoder includes a conditional network and an autoregressive network. The conditional network is mainly used to represent the characteristics of the voice data in the voice stream, and the autoregressive network is mainly used to generate subsequent voice frames required for packet loss compensation for the voice data.

[0036] The conditional network mainly consists of a residual network responsible for extracting the global feature information of speech data, and an upsampling network responsible for extracting the local feature information corresponding to multiple scales of speech data with multiple different scaling factors. The global feature information and the local feature information are then spliced ​​into comprehensive feature information through a splicing layer to extract the deep semantic information of speech data, so that the comprehensive feature information can represent the characteristics of slowly changing information such as phonemes and rhythms in acoustic features. The global feature information obtained by the residual network is further divided into multiple outputs to provide reference information for the process of the autoregressive network processing the comprehensive feature information.

[0037] The autoregressive network is implemented using a recurrent neural network (RNN, Recurrent Convolutional Network), which uses two unidirectional gated recurrent units (GRU, Gate Recurrent Unit) internally. The number of gated recurrent units can be flexibly set as needed. Each gated recurrent unit is responsible for processing comprehensive feature information, obtaining corresponding deep semantic information and splicing it with the global feature information output by the residual network and then inputting it into the next node for processing. The autoregressive network is provided with a classification network at its end, which is used to implement classification mapping based on the combined information of the output processed by the gated recurrent unit and the global feature information, so as to restore the subsequent speech frames.

[0038] In one embodiment, the vocoder can be modified based on WaveRNN (Wave Recurrent Neural Network) or its variant model such as SC-WaveRNN (Speaker Conditional WaveRNN) as a basic model. WaveRNN is an autoregressive network model suitable for processing audio data in sequence form. The original design goal of WaveRNN is to maintain high-speed sequence generation. The author uses simplified models, sparsification, parallel sequence generation and other technologies to significantly improve the sequence generation speed. Its good performance can even achieve real-time speech synthesis on the CPU.

[0039] The speech synthesis model determined by processing any embodiment of the present application can be trained to a convergence state in advance for use. The network structure of the speech synthesis model has reduced overall complexity, fewer parameters, and uses global feature information to provide temporary context, making it easier to train to convergence and making more accurate predictions.

[0040] See also Figure 3 According to one aspect of the present application, a method for generating a speech synthesis model is provided, in one embodiment of which, the method comprises the following steps:

[0041] Step S1100, calling a controller, and having the controller generate a structural code of a vocoder;

[0042] This application uses a controller suitable for implementing network architecture search (NAS) to iteratively generate the structural encoding of a vocoder, and uses the measured performance of the vocoder generated by the controller to perform policy gradient optimization on the controller, and trains the controller to a convergence state, so that the controller gradually generates increasingly excellent performance of the vocoder during the iterative training process.

[0043] The controller is constructed by using a recurrent convolutional network (RNN), whose input is a random number, and outputs a structural code represented by a high-dimensional vector, and the structural code represents the coding information corresponding to the conditional network and the autoregressive network of the vocoder in the form of a sequence. When constructing the vocoder, according to the structural code that complies with the predetermined specification of the topological structure of the vocoder, the coding information corresponding to the conditional network and the autoregressive network is selected from it, which can be used to construct the conditional network and the autoregressive network respectively.

[0044] In one embodiment, the structural coding includes coding information corresponding to the upsampling network in the conditional network of the vocoder. In the exemplary specific structure, the upsampling network includes three convolutional layers that match different scaling factors for sampling at different scales. For this purpose, the coding information corresponding to the conditional network in the structural coding can provide corresponding design parameters for the three convolutional layers. The design parameters can be the layer type and number of channels of the convolutional layer.

[0045] In one embodiment, the layer type of the convolutional layer of the upsampling network can be any of the following:

[0046] The first type: the convolution kernel is 1*1 convolution, and the layer type is 0;

[0047] The second type: the convolution kernel is 3*3 convolution, and the layer type is 1;

[0048] The third type: the convolution kernel is 5*5, and the layer type is 2;

[0049] The fourth type: convolution with a convolution kernel of 1*1, deconvolution with a stride of 2, and the layer type is 3;

[0050] The fifth type: the convolution kernel is 3*3, the deconvolution is performed with a step size of 2, and the layer type is 4;

[0051] The sixth type: convolution with a convolution kernel of 5*5, deconvolution with a stride of 2, and the layer type is represented as 5.

[0052] In one embodiment, the number of channels corresponding to the convolutional layer can be any one of the following channel numbers: 32, 64, 96, 128, etc. Correspondingly, the type code of each channel number can be represented by 0 to 3.

[0053] According to the above example, the encoding information corresponding to the upsampling network of the conditional network in the structural encoding can be a subsequence consisting of the layer type and the number of channels provided for each convolutional layer thereof, in the form of:

[0054] {First convolutional layer type; Number of channels in the first convolutional layer; Second convolutional layer type; Number of channels in the second convolutional layer; Third convolutional layer type; Number of channels in the third convolutional layer}

[0055] According to the example here, it can be seen that each convolutional layer in the upsampling network occupies two dimensions in the structural encoding, corresponding to its layer type and number of channels, which is very concise.

[0056] In one embodiment, the structural code includes coding information corresponding to the gated recurrent unit in the autoregressive network of the vocoder. Assume that the input feature of the gated recurrent unit at step t is x t , the hidden state is h t , the neuron structure of the gated recurrent unit can be viewed as a functional relationship:

[0057] h t =f(h t-1 ,x t )

[0058] In particular, the neuron structure of the gated recurrent unit of the vocoder generally has an architecture expressed by the following description formula:

[0059] h t =(1-Z t )*h t-1 +Z t *C t

[0060] Where Z t is the update gate, which controls the speed of updating the hidden state information. t is a candidate hidden state. According to the description formula of the above neuron structure, the focus of the architecture search of the gated recurrent unit is to search for Z t and C t architecture.

[0061] Therefore, taking four execution units as an example, the composition of the encoding information of the gated recurrent unit can be expressed as follows:

[0062] {first binary operation type; first binary operation activation function; second binary operation type; second binary operation activation function; third binary operation type; third binary operation activation function; fourth binary operation type; fourth binary operation activation function}

[0063] According to the above example, the gated recurrent unit consists of 4 units, each of which includes a binary operation and an activation function. The first unit determines the update gate Z. t The calculation method of the last three units determines the candidate hidden state C t .

[0064] Here, each binary operation can take one of the following types:

[0065] Addition, the type code can be represented as 0

[0066] Element-wise multiplication, type encoding can be represented as 1

[0067] · Splicing, the type code can be represented as 2

[0068] The type of activation function can be selected from the following types:

[0069] No activation function, the type code can be represented as 0

[0070] ReLU, type encoding can be represented as 1

[0071] ·Sigmoid, the type encoding can be represented as 2

[0072] tanh, the type code can be represented as 3

[0073] It can be seen that according to the above example, the encoding information of the conditional network will occupy 6 dimensions in the structural encoding, and the encoding information of the autoregressive network will occupy 8 dimensions in the structural encoding. The structural encoding generated by the controller is a vector of 14 dimensions. Of course, the dimensions required to be occupied by the conditional network and the autoregressive network can be flexibly set according to the preset topological structure of the vocoder to be constructed, and their respective occupation of the dimensions of the structural encoding is also correspondingly variable, and is not limited to the above example.

[0074] Step S1200, constructing a vocoder according to the structural coding, the vocoder comprising a conditional network and an autoregressive network generated according to the structural coding;

[0075] It is not difficult to understand that as long as the structural coding is utilized according to the above principles, a corresponding vocoder can be constructed. In one embodiment, the vocoder can be constructed according to the following process:

[0076] First, constructing an upsampling network in a conditional network of a vocoder according to first encoding information in the structural encoding, wherein the first encoding information includes layer types corresponding to a plurality of convolutional layers of the upsampling network and the corresponding number of channels;

[0077] Following the example of the previous step, the structure is encoded to obtain a substring corresponding to the conditional network, that is, the feature values ​​corresponding to the first 6 dimensions, which are used to construct the conditional network, specifically, the upsampling network in the conditional network. Similarly, a substring corresponding to the autoregressive network, that is, the feature values ​​corresponding to the last 8 dimensions, can be obtained to construct the autoregressive network, specifically, the gated recurrent unit in the autoregressive network.

[0078] An exemplary structure is encoded as follows:

[0079] {0,0,1,1,5,3,0,2,0,2,1,0,0,3}

[0080] Among them, the first 6 bits in the exemplary structure code are taken as the first coding information corresponding to the upsampling network, that is:

[0081] {0,0,1,1,5,3}

[0082] The upsampling network constructed according to the first coding information has a network structure consisting of three convolutional layers. The structure and function of each convolutional layer are as follows: after the first convolutional layer obtains the acoustic features of the input, it repeats it twice, performs a convolution operation with a convolution kernel of 1*1, and processes it into data of 32 channels; the second convolutional layer repeats the output of the first convolutional layer twice, performs a convolution operation with a convolution kernel of 3*3, and processes it into data of 64 channels; the third convolutional layer performs a 5*5 convolution operation in the exemplary structural encoding, and processes it into data of 128 channels so as to access the autoregressive network.

[0083] Secondly, constructing a gated recurrent unit in the autoregressive network of the vocoder according to the second encoding information in the structural encoding, wherein the second encoding information includes an operation type corresponding to a structural node of the gated recurrent unit and its corresponding activation type;

[0084] The last 8 bits of the exemplary structure code are taken as the second coding information corresponding to the gated cyclic unit, that is:

[0085] {0,2,0,2,1,0,0,3}

[0086] The gated recurrent unit is constructed according to the second encoding information, and the following heuristic rules are followed when applying binary operations:

[0087] 1. The operands of the addition operation are multiplied by the learnable matrix before being added, and the output dimension of the learnable matrix is ​​the smaller of the two dimensions;

[0088] 2. Element-wise product: If the dimensions of the two operands are the same, they are directly multiplied without using the learnable matrix; if the dimensions are different, the operand with higher dimension is multiplied by the learning matrix to make it consistent with the operand with lower dimension before multiplying;

[0089] 3. The concatenation operation does not use a learnable matrix for direct concatenation.

[0090] According to the above rules, the second coded information obtained in the previous example is taken as an example, where the first two bits corresponding to the update gate are coded {0,2}, and its binary operation is specified as an addition operation, which is directly applied to h t-1 ,x t , specifically means that h t-1 and x t Matching can be learned by adding matrices, and then using sigmoid function for activation output. The structural description formula is expressed as:

[0091] Z t =σ(ω 1 *h t-1 +ω 2 *x t )

[0092] The last six bits corresponding to the candidate hidden states are encoded as {0,2,1,0,0,3} and are first applied to h t-1 ,x t , and then compare the result with h t-1 Combined, the result x t Combined, they correspond to the following functions: f 1 、f 2 、f 3 , the principle formula example is as follows:

[0093] i 1 =f 1 (h t-1 ,x t )

[0094] i 2 =f 2 (i 1 ,h t-1 )

[0095] C t =f 3 (i 2 ,x t )

[0096] Therefore, according to the above principle formula, the process formula corresponding to the second coding information is expressed as follows:

[0097] i 1 =σ(ω 3 *ht-1 +ω 4 *x t )

[0098] i 2 =i 1 ⊙h t-1

[0099] C t =tanh(ω 5 *i 2 +ω 6 *x t )

[0100] According to the architecture described above, the gated recurrent unit combines the above two parts and outputs them, and the formula is expressed as:

[0101] h t =(1-Z t )*h t-1 +Z t *C t

[0102] After the above process, the gated recurrent unit in the autoregressive network can be obtained. When the autoregressive network defines multiple gated recurrent units connected in series according to the preset topology structure of the vocoder to be constructed, each gated recurrent unit can adopt the same structure.

[0103] Finally, the conditional network and the autoregressive network are constructed as a vocoder according to the preset topology.

[0104] Since other structures of the vocoder have been defined by its preset topology structure, after constructing the upsampling network and gated recurrent unit of the vocoder according to the encoding information of the vocoder, the corresponding vocoder can be obtained by configuring them in the preset topology structure.

[0105] It is not difficult to see that by utilizing the prior knowledge provided by the prototype of the vocoder, its preset topological structure is known. On this basis, the controller produces the encoding information of the vocoder, generates the upsampling network and gated recurrent unit therein, and constructs the corresponding vocoder. The generation process does not need to rely on a directed acyclic graph like a general network search architecture. This is because the complex branch structure obtained by relying on the directed acyclic graph, even if the scale of weight parameters is saved, the fronthaul cost is still high when deployed to the terminal device. Moreover, when relying on a directed acyclic graph, if the data dimension changes during the upsampling process, the branch structure needs to deal with the problem of data dimension consistency, which often leads to performance degradation.

[0106] Correspondingly, the control structure vocoder according to the present application has the following advantages in terms of network architecture search:

[0107] 1. Using prior knowledge of the vocoder, we sacrificed the spatial complexity of the network topology search in exchange for a joint search of the key weight-dense modules of the vocoder, namely the upsampling network and the gated recurrent unit;

[0108] 2. Designing the search space with the goal of compressing the number of network FLOPS, using operators with fewer weights and applying some heuristic rules can improve efficiency;

[0109] 3. Combined with the subsequent soft constraint objective function, search for the speech synthesis model with the best resource allocation to achieve the goal of mobile terminal deployment that meets real-time requirements.

[0110] Step S1300, iteratively training the vocoder to a converged state using a training set, and performing a gradient update on the controller and iteratively generating a new vocoder before the controller reaches convergence according to the performance score of the vocoder on the test set;

[0111] The vocoder generated by the controller needs to obtain its actual measured performance. Therefore, a training set is prepared, and sufficient training samples are used to perform iterative training on the vocoder generated by the controller, and the vocoder is trained to a convergence state. The training samples can be audio data provided for the input of the corresponding vocoder, and each audio data includes multiple speech frames. A speech frame sequence formed continuously by timestamps is taken, and its acoustic features are extracted and input into the vocoder. The acoustic features are represented by the conditional network of the vocoder, and then the subsequent speech frames of the speech frame sequence are predicted by the autoregressive network of the vocoder based on the feature representation. The vocoder finally calculates the model loss value according to the subsequent speech frame and the subsequent speech frame continuous with the speech frame sequence of the training sample, and the gradient update and iteration process of the vocoder are controlled according to the model loss value. Through continuous iterative training, the vocoder is finally brought to a convergence state, thereby completing the training of the vocoder.

[0112] After the vocoder is trained to convergence, the controller can be updated as follows:

[0113] First, the vocoder is tested using a test sample in a test set to obtain a performance score, wherein the performance score includes a quality score obtained after the test sample is processed by the vocoder;

[0114] A corresponding prepared test set is used to test the vocoder that has reached a converged state using the test samples therein, and a performance score corresponding to the entire test set is obtained according to a preset statistical method, so as to analyze the performance of the vocoder on the test set.

[0115] In one embodiment, the performance score may be a summary statistic of the quality score obtained for each test sample, and the quality score may be a subjective quality score (MOS, Mean Opinion Score) or an objective quality score, such as a peak signal-to-noise ratio or a structural similarity score. The algorithm for performing statistics on the quality scores of all test samples generally uses an average value. According to this embodiment, the controller may apply the following hard-constrained multi-objective function to calculate its model loss value:

[0116]

[0117] stlat(a,h)≤B

[0118] Among them, A is the architecture search space corresponding to the vocoder, MOS(a) is the average subjective opinion score of the architecture a of the vocoder on the test set, lat(a,h) is the inference delay of architecture a under hardware condition h, and B is the maximum inference delay obtained by the vocoder on the entire test set, which will limit the controller to sample only in the search space where the inference delay does not exceed B.

[0119] In another embodiment, the performance score can be obtained by summarizing the statistical value obtained based on the quality score statistics corresponding to each test sample, and then superimposing the inference delay generated by the vocoder testing the entire test set, so as to achieve a comprehensive evaluation by combining the voice quality with the machine execution performance. According to this embodiment, the controller can apply the following soft constraint objective function to calculate its model loss value:

[0120]

[0121] Where A is the architecture search space corresponding to the vocoder, MOS(a) is the average subjective opinion score of architecture a of the vocoder on the test set, lat(a,h) is the inference delay of architecture a on hardware condition h, and c is a constant that controls the compression of the architecture.

[0122] Then, policy gradient optimization is performed on the controller according to the performance score, and the controller is iteratively called to generate a new vocoder if the controller has not reached a convergence state.

[0123] It can be seen that after the vocoder obtains its corresponding performance score, it can be determined whether the controller has converged based on the performance score, thereby deciding whether to continue to iterate and generate a new vocoder. Specifically, if the controller has not converged, the controller can be subject to policy gradient optimization based on the performance score, and then the new vocoder can be generated again from step S1100, and steps S1100 to S1300 are executed in a loop; if the controller has converged, it can jump to the next step, and at the same time, the training of the controller is terminated.

[0124] In one embodiment of the controller using a soft constraint objective function, the controller is updated with a policy gradient, and the following policy update function may be applied:

[0125]

[0126] Among them, π θ is the current policy probability distribution (parameterized by the controller parameter θ), a t The architecture design selected for the controller gated recurrent unit at step t, R is a 1:T The reward value calculated for the chosen architecture under the soft constraint objective function.

[0127] In another embodiment, the controller is updated with a policy gradient, and a corresponding strategy of a deep reinforcement learning method may be applied, and the following policy update function is used:

[0128]

[0129] Among them, Q(a t |a 1:t-1 ) is the Q function, which is implemented by an RNN that shares parameters with the controller, α is the learning rate of Q learning, and γ is the discount factor.

[0130] It is not difficult to understand that implementing gradient optimization on the controller based on the performance score of the vocoder on the test set can continuously improve the controller's ability to produce vocoders with excellent performance.

[0131] Step S1400: After the controller reaches convergence, a vocoder is selected as a speech synthesis model according to the performance score.

[0132] During its training process, the controller generates multiple vocoders through continuous iteration. These vocoders are trained and tested to obtain their corresponding performance scores. The performance score reflects the performance of the corresponding vocoder. The higher the performance score of the vocoder, the better the real-time performance of the speech synthesis will be, and its network structure is usually the best. Therefore, based on the performance score, one of the large number of vocoders generated during the iterative training process of the controller can be selected as the speech synthesis model. Usually, one or more vocoders with the highest performance score can be used as the speech synthesis model. Of course, in other embodiments, the vocoder with the second highest performance score can be selected as the speech synthesis model to adapt to the actual hardware capabilities of the terminal equipment to be deployed, which is essentially the same.

[0133] The speech synthesis model selected in this step can be further trained as needed to optimize its reasoning ability so that it can be deployed in terminal devices for use.

[0134] According to the above embodiments, the present application adopts a controller to generate structural codes, constructs a vocoder according to the structural codes, trains and tests the vocoder to obtain corresponding performance scores, controls the iterative process of the controller according to the performance scores, generates multiple vocoders during the training process of the controller, and finally selects the vocoder with the best measured performance as the speech synthesis model. With the help of the controller, the automatic generation and selection of vocoders are realized, so that the obtained speech synthesis model meets the requirements of mobile terminal device deployment, can obtain good performance after being deployed on the mobile terminal device, and can meet the requirements of model miniaturization and high real-time performance in the speech synthesis scenario.

[0135] Based on any of the above embodiments, please refer to Figure 4 , iteratively training the vocoder to a converged state using a training set, comprising:

[0136] Step S1310: calling a single training sample in the training set, obtaining a plurality of continuous speech frames of a predetermined duration therein to construct a speech frame sequence, and extracting acoustic features corresponding to the speech sequence;

[0137] The training set can be a public data set or online user data. The public data set is a speech data set that contains multiple languages ​​and is a collection of audio data provided by tens of thousands of contributors, where each speech data can be used as a first-class training sample. The online user data can be collected by itself and may include audio data sampling segments of tens of thousands of online users. After the original sampling segments are subjected to background noise elimination, pure human voice segments are intercepted through Voice Activity Detection (VAD), and finally a 15-30s training sample is formed. During the vocoder training process, one training sample is called for each iterative training.

[0138] For each training sample, multiple time-sequentially continuous speech frames are obtained with a predetermined time length to form a speech frame sequence, and then its acoustic features are extracted. For example, the length of the speech frame sequence can be 20ms as the time unit of each speech frame, and a total of 300ms corresponding speech frames are selected, and then the acoustic features of the speech frame sequence are extracted accordingly.

[0139] The acoustic features serve to describe relevant information of relatively stable style features in a speech frame, such as phonemes, rhythms, etc., and may be any one of logarithmic Mel spectrum information, time-frequency spectrum information, and CQT filter information.

[0140] Those skilled in the art understand that the above various acoustic features can be encoded using corresponding algorithms. In the encoding process, the speech signal is first subjected to conventional processing such as pre-emphasis, framing and windowing, and then analyzed in the time domain or frequency domain, that is, speech signal analysis is achieved. The purpose of pre-emphasis is to enhance the high-frequency part of the speech signal and smooth the spectrum; generally, pre-emphasis is achieved through a first-order high-pass filter. Before analyzing the speech signal, it is also necessary to frame it. Usually, the length of each frame of the speech signal is set to 20ms. Considering the frame shift factor, there can be 10ms overlap between adjacent frames. In order to achieve framing, it can be achieved by windowing the speech signal. Different window selections will affect the results of speech signal analysis. More commonly, the window function corresponding to the Hamming window is used to implement the windowing operation.

[0141] In one embodiment, for the time-frequency spectrum information, the speech data of each speech information in the time domain is pre-emphasized, framed, windowed, and short-time Fourier transformed (STFT) to the frequency domain to obtain data corresponding to the spectrogram, thereby forming the time-frequency spectrum information.

[0142] In another embodiment, the logarithmic Mel spectrum may be obtained by filtering the time-frequency spectrum information with a Mel-scale filter bank and then taking the logarithm.

[0143] In another embodiment, for the CQT filtering information, CQT (Constant Q Transform), i.e., constant Q transform, refers to a filter group whose center frequencies are distributed according to an exponential law, whose filtering bandwidths are different, but whose center frequency to bandwidth ratio is a constant Q. It is different from Fourier transform in that the horizontal axis frequency of its spectrum is not linear, but based on log2, and the filter window length can be changed according to the different spectral line frequencies to obtain better performance.

[0144] Any of the above specific acoustic features can be used as the input of the vocoder of the present application. In order to facilitate the processing of the vocoder, in one embodiment, the acoustic features can be constructed according to a certain preset format. For example, the acoustic features corresponding to each speech frame are organized into a row vector, and for the entire speech frame sequence to be encoded, the row vectors of each speech frame are spliced ​​together longitudinally in time sequence to obtain a two-dimensional matrix as the acoustic feature of the entire speech frame sequence.

[0145] Step S1320: input the acoustic feature into the conditional network of the vocoder, obtain global feature information of the acoustic feature through the residual network therein, obtain local feature information of the acoustic feature at multiple scales through the upsampling network therein, and obtain comprehensive feature information composed of the global feature information and the local feature information;

[0146] According to the principle of the vocoder of the present application, the acoustic features are input into the conditional network of the vocoder, and the output is obtained according to the following process:

[0147] Firstly, the acoustic features are extracted based on the residual network in the conditional network to obtain its global feature information;

[0148] In this embodiment, taking the specific architecture of the vocoder of the present application obtained by using SC-WaveRNN as the prototype topology structure as an example, compared with the prototype network provided by the original author, the Speaker Encoder in the prototype network can be omitted. Of course, in another embodiment, the encoder can also be used.

[0149] From the perspective of implementing speech synthesis in this application, the speaker encoder in the prototype network of SC-WaveRNN is not necessary. The speaker encoder is an important contribution of the SC-WaveRNN paper. The author uses PESQ (Perceptual evaluation of speech quality, objective speech quality evaluation) to measure that the speaker encoder has positive benefits in all situations; this application uses the same indicator to measure, and the contribution of the speaker encoder is not obvious for the task of performing packet loss compensation in this application. The reason is that SC-WaveRNN is based on TTS (Text to Speech, from text to speech), the model input contains a complete Mel spectrum, and it is more important for the speaker encoder to map the Mel spectrum to speaker features; for the application scenario of packet loss hiding in this application, the speaker features can only affect the first frame of the compensated speech, and the added speaker encoder includes LSTM (Long Short-Term Memory, long short-term memory network), which has a high computational complexity and the benefits are not obvious. Therefore, those skilled in the art can adopt or not adopt the speaker encoder according to the principles disclosed herein to realize the configuration of the vocoder.

[0150] The acoustic features of the speech frame sequence obtained above, which are used to generate subsequent speech frames, are input into the conditional network of the vocoder, one of which is input into the residual network of the conditional network. The residual network is responsible for performing residual convolution operations on the acoustic features, extracting deep semantic information from the speech frame sequence at the global scale, thereby obtaining corresponding global feature information and realizing global representation of the acoustic features of the speech frame sequence.

[0151] Then, the acoustic features are sampled at multiple scales based on the upsampling network in the conditional network to obtain local feature information thereof;

[0152] The acoustic features of the speech frame sequence are input from the second path to the upsampling network in the conditional network. Under the constraint of the first encoding information provided by the structural encoding, the upsampling network is constructed to correspond to multiple scaling scales, for example, three scaling scales. Deep semantic information is extracted from the acoustic features at these different scaling scales, and the information granularity is continuously refined to obtain the corresponding local feature information at different information granularities, thereby realizing the local representation of the acoustic features of the speech frame sequence.

[0153] Furthermore, the global feature information and the local feature information are feature spliced ​​based on the splicing layer in the conditional network to obtain comprehensive feature information.

[0154] The splicing layer set in the conditional network performs feature splicing on the global feature information and local feature information obtained by the residual network to construct comprehensive feature information. The comprehensive feature information includes both the global information of the acoustic feature and the local information of the acoustic feature at different finer scales. It can comprehensively and completely characterize the important features of the acoustic feature and help guide the autoregressive network to generate effective subsequent speech frames.

[0155] According to the above process, it can be understood that the vocoder realizes effective feature representation of the speech frame sequence by integrating the important features of acoustic features under global and local conditions, which is the basis for generating subsequent speech frames. In addition, by adopting a vocoder with an optimal network structure, the working efficiency of the vocoder can be improved and good benefits can be obtained.

[0156] Step S1330, extracting relatively stable style features in the speech frame sequence from the comprehensive feature information through the gated recurrent unit of the autoregressive network in the vocoder to obtain predicted feature information;

[0157] First, the comprehensive feature information output from the conditional network is fully connected through the first fully connected layer in the autoregressive network to further achieve feature integration.

[0158] Then, the fully connected comprehensive feature information is first input into the first gated recurrent unit generated according to the second encoding information in the structural encoding output by the controller for feature extraction, so as to select the important features therein and obtain the first gated feature information. After the first gated feature information is further spliced ​​with the global feature information obtained by the residual network, it is input into the second gated recurrent unit generated according to the second encoding information in the structural encoding output by the controller. The second gated recurrent unit similarly extracts the feature information input therein to obtain the second gated feature information, and the second gated feature information is similarly spliced ​​with the global feature information obtained by the residual network and then output.

[0159] In one embodiment, the output after the second gated feature information is spliced ​​with the global feature information can be used as the prediction feature information. In another embodiment, the feature information obtained by splicing the second gated feature information with the global feature information is further fully connected, and after the full connection, it is further spliced ​​with the global feature information obtained by the residual network to obtain the prediction feature information. In each of the above steps, the global feature information obtained by the residual network is continuously referenced to provide contextual references, which helps to accurately extract important features in the acoustic features and make the subsequent speech frames generated by the autoregressive network more effective.

[0160] Step S1340, generating subsequent speech frames of the speech frame sequence according to the predicted feature information through a classification network in the vocoder;

[0161] The predicted feature information is input into a classification network preset in each autoregressive network, and after classification mapping by the classification network, the probability of each bit position required for constructing a subsequent speech frame is determined, thereby constructing a subsequent speech frame.

[0162] In one embodiment, in the classification network, in the process of constructing subsequent speech frames according to the predicted feature information, the following temperature coefficient-based formula is applied to perform audio sampling:

[0163]

[0164] Where T is the sampling temperature, y i is the predicted label, P i is the probability of the i-th bit of the subsequent speech frame.

[0165] According to the above process, it can be understood that the autoregressive network can effectively generate subsequent speech frames of the current speech frame according to the output of the conditional network, under the guidance of the global and local features of the acoustic features, to achieve effective packet loss compensation for the speech stream.

[0166] Step S1350: Calculate the loss value of the subsequent speech frame using the speech frame that is consecutively following in time sequence in the speech frame sequence in the training sample, and control the iterative training of the vocoder according to the loss value.

[0167] After generating subsequent speech frames for a speech frame sequence in a training sample, the next speech frame of the last speech frame in the speech frame sequence is used as a supervisory label to calculate the loss value of the subsequent speech frame relative to the next speech frame, and then decide whether the vocoder has converged based on the loss value. When it has not converged, backpropagation is performed on the vocoder based on the loss value, and the weight parameters of its conditional network and autoregressive network are updated by gradients, and the next training sample is called from the training set to continue iterative training of the vocoder until the vocoder converges.

[0168] It can be seen that in the process of vocoder training, the last speech frame in the speech frame sequence sampled in the training sample and its next speech frame in sequence are used as the supervision label of the subsequent speech frame generated based on the speech frame sequence, and are used to calculate the loss value of the subsequent speech frame, thereby realizing effective supervision of the pre-training process of the vocoder.

[0169] The above embodiments show that the controller outputs the structural code, and the vocoder constructed according to the structural code will be trained on the basis of the same training set. Therefore, the performance of multiple vocoders produced by the controller can be compared later to produce the preferred vocoder as a speech synthesis model.

[0170] Based on any of the above embodiments, please refer to Figure 5 , after the controller reaches convergence, the step of selecting the vocoder as the speech synthesis model according to the performance score includes:

[0171] Step S1500, training the speech synthesis model to a convergence state;

[0172] After at least one vocoder is selected as a speech synthesis model suitable for deployment to a terminal device in step S1400, the speech synthesis model can be subjected to more in-depth training to comprehensively improve its speech synthesis capability. An exemplary training method is to use the conditional network of the speech synthesis model as a generator and its autoregressive network as a discriminator to construct a generative adversarial model for training. The following text will also reveal an embodiment corresponding to a training method recommended by the present application, which will not be listed for the time being. Through training, the selected speech synthesis model reaches a convergence state again and can be deployed in a terminal device for use.

[0173] Step S1600: configure the speech synthesis model to smoothly connect subsequent speech frames generated according to the speech stream to the speech stream.

[0174] When the speech synthesis model that has completed the final training is deployed to the terminal device, the speech synthesis model can be enabled to connect the subsequent speech frames it generates for the speech stream to the speech stream by configuring the transition method between the subsequent speech frames it generates and the corresponding speech stream. This can be achieved in the following process:

[0175] Firstly, the subsequent speech frames generated by the speech frame synthesis model are spliced ​​to obtain a compensation frame sequence;

[0176] For subsequent speech frames generated by the speech synthesis model, no matter whether a single subsequent speech frame is generated or multiple subsequent speech frames are generated continuously, they can be processed centrally, and these subsequent speech frames are sequentially spliced ​​according to the timestamps to construct a compensation frame sequence.

[0177] Then, adjusting the volume corresponding to the compensation frame sequence so that it does not exceed the volume of the speech frame sequence;

[0178] The volume of each subsequent voice frame obtained after packet loss compensation may be different. In order to unify the volume effect, a preset compressor, specifically a volume compressor, can be called to implement volume compression control on each subsequent voice frame in the compensation frame sequence. Taking the volume in the voice frame sequence as a reference, the excessive volume in the subsequent voice frames is reduced so that the volume of the subsequent voice frames in the compensation frame sequence does not exceed the volume of the voice frames in the voice frame sequence, thereby controlling the volume of the voice frames obtained by packet loss compensation to a reasonable level and maintaining the consistency of sound quality.

[0179] Finally, the compensation frame sequence is smoothly connected to the speech stream where the speech frame sequence is located.

[0180] By inserting the compensation frame sequence into the voice stream, packet loss compensation for the voice stream can be further realized. In order to maintain the auditory smoothness of the compensation frame sequence obtained by the speech synthesis model after inserting into the voice stream, the compensation frame sequence can be smoothly inserted into the voice stream by fading in and out.

[0181] In one embodiment, after the compensation frame sequence is connected to the voice stream, it is controlled to start fading out at 20ms, that is, it starts fading out from the second subsequent voice frame until it is completely silent 20ms after the packet loss ends or 120ms after the packet loss starts. In addition, the voice frames in the voice stream after the compensation frame sequence are controlled to fade in within a 20ms time window. The above time settings corresponding to fade-out and fade-in can be flexibly adjusted according to actual needs and are not limited to the above examples.

[0182] It can be understood from the above process that the subsequent speech frames restored by the speech synthesis model through packet loss compensation can be smoothly connected to the original speech stream, so that the original speech stream remains smooth in hearing and obtains good sound quality.

[0183] The above embodiments reveal that after the speech synthesis model selected from the vocoder generated by the controller is retrained to a convergence state, the speech synthesis model can be suitable for deployment in terminal devices such as mobile terminals, and by configuring it to have the function of smoothly connecting subsequent speech frames generated according to the voice stream to the voice stream, the speech synthesis model can continuously generate multiple subsequent speech frames for the voice stream to achieve continuous packet loss compensation. Since the selected speech synthesis model is effectively compressed by the controller, its weight parameters are greatly reduced and the operating efficiency is significantly enhanced, thereby meeting the high real-time requirements of the terminal device, so that after implementing packet loss compensation for the voice stream, the sound quality of the voice stream can be significantly improved.

[0184] Based on any of the above embodiments, please refer to Figure 6, the step of training the speech synthesis model to a convergence state comprises:

[0185] Step S1510, using the first type of training samples in the data set to perform the first stage training on the speech synthesis model, training the speech synthesis model to a convergence state, and performing the training according to a preset weight sparsification target in the first stage training;

[0186] Two types of training samples are prepared, namely, first type of training samples and second type of training samples. The two types of training samples can be stored in the same data set or in different data sets.

[0187] The first type of training samples are mainly used for pre-training of speech synthesis models, and the second type of training samples are mainly used for fine-tuning training of speech synthesis models. Therefore, the first type of training samples can use materials with appropriately relaxed environmental noise, and the second type of training samples can use materials with clearer foreground speech.

[0188] In one embodiment, the first type of training samples can use a public data set. As mentioned above, a public data set selected in the actual test training of this application is a speech data set containing multiple languages ​​and is composed of audio data provided by tens of thousands of contributors, where each speech data can be used as the first type of training samples.

[0189] In one embodiment, the second type of training samples can collect online user data by themselves. As mentioned above, the online user data used in the actual test training of this application includes audio data sampling segments of tens of thousands of online users. After the background noise of the original sampling segments is eliminated, pure human voice segments are intercepted through Voice Activity Detection (VAD), and finally a 15-30s training sample is formed.

[0190] Both the first type of training samples and the second type of training samples may be pre-processed in advance to determine each speech frame therein so as to extract the speech frame sequence and its corresponding acoustic features.

[0191] In the first stage of training, the speech synthesis model is trained for each first-class training sample, and the process is the same as the embodiment disclosed above, which will not be described in detail. Similarly, it can be seen that in the pre-training process of the speech synthesis model, the speech frame with the last time sequence in the speech frame sequence sampled in the first-class training sample and the speech frame with the subsequent time sequence are used as the supervision label of the subsequent speech frame generated based on the speech frame sequence, and are used to calculate the loss value of the subsequent speech frame, so as to realize the effective supervision of the pre-training process of the speech synthesis model.

[0192] In order to further reduce the weight of the speech synthesis model, during the first stage of training, for each training sample, the weight of the speech synthesis model is arranged by absolute value, and the n with the smallest absolute value is t The weights are set to 0, where:

[0193]

[0194] Where N is the total number of model weights, S∈[0,1] is the target sparsification rate, t is the training step, and t 0 is the step to start sparse deployment, T is the total number of sparse training steps, and α>1 is a constant that controls the sparsification rate.

[0195] Step S1520: using the second type of training samples in the data set to perform the second stage training on the speech synthesis model, and training the speech synthesis model to a convergence state;

[0196] In the second stage training process, compared with the first stage training, the basic training process is the same, but the training samples used in the second stage training are the second type of training samples, and when performing the second stage training, the speech synthesis model is trained at a smaller learning rate than the first stage training, so that the parameterization of the conditional network is more in line with the data distribution of online users. Since the second type of training samples provided by online users are pure human voice clips, using the second type of training samples to train the speech synthesis model at a smaller learning rate helps to further improve the ability of the speech synthesis model to generate subsequent speech frames.

[0197] Similarly, during the second stage of training, weight sparse training can be implemented on the speech synthesis model based on the same principles as the first stage of training.

[0198] Step S1530, solidify the weights of the conditional network of the speech synthesis model, use the second type of training samples in the data set to implement the third stage training of the speech synthesis model, train the speech synthesis model to a convergence state, and adjust the weights of the autoregressive network in the speech synthesis model.

[0199] The third stage of training for the speech synthesis model is actually to implement fine-tuning training on the speech synthesis model after the speech synthesis model has been pre-trained in the first two stages, so as to further adjust the weights of the autoregressive network therein to ensure that it can effectively produce subsequent speech frames of a given speech frame sequence.

[0200] Therefore, before executing the third stage of training, the conditional network is considered to have reached the desired requirements, and its weights are frozen so that the weights of the conditional network will not be updated by gradients during the third stage of training. As for the weights of the autoregressive network, they are still maintained as learnable weights and will be further modified during the third stage of training. Then, the third stage of training for the speech synthesis model can be started.

[0201] When implementing the third stage training of the speech synthesis model, the training samples used are the second type of training samples corresponding to the online user data. The basic process of training each training sample is the same as the first and second stage training. Similarly, weight sparse training can also be applied as needed. In the process of pre-training the speech synthesis model, the speech frame with the last time sequence in the speech frame sequence sampled in the second type of training samples and the speech frame that follows in time sequence are used as the supervision labels of the subsequent speech frames generated based on the speech frame sequence, and are used to calculate the loss values ​​of the subsequent speech frames, thereby realizing effective supervision of the fine-tuning training process of the speech synthesis model.

[0202] After the third stage of training, the weights of the conditional network remain unchanged, and the weights of the autoregressive network are continuously corrected, eventually reaching a convergence state, which terminates the entire training process of the speech synthesis model.

[0203] According to the above embodiments, it is not difficult to understand that the speech synthesis model of the present application undergoes multi-stage training, wherein the first type of training samples are used for pre-training in the first stage, and the second type of training samples are used in the second stage to improve the pre-training effect with a smaller learning rate, and in the third stage, the weights of the conditional network are solidified, and the second type of training samples are used to improve the ability of the autoregressive network to generate subsequent speech frames, and finally, comprehensive training is achieved, so that the obtained speech synthesis model has the ability to generate effective subsequent speech frames for a given speech frame sequence. The speech synthesis model obtained according to the above training process has a low number of parameters and a low number of floating-point operations per second, and can realize real-time inference on the mobile terminal, which is particularly suitable for deployment in mobile terminals such as mobile phones and computers.

[0204] Based on any of the above embodiments, please refer to Figure 7 , the second type of training samples in the data set are used to implement the third stage of training for the speech synthesis model, including:

[0205] Step S1531, replacing the acoustic features of a preset number of subsequent speech frames whose time sequence is continuous with the last speech frame in the speech frame sequence sampled from each second type of training sample with mask representation;

[0206] During the third stage of training, the speech synthesis model can be trained accordingly according to the number of subsequent speech frames that the speech synthesis model is expected to predict and generate for the speech frame sequence, that is, the maximum compensation number. To this end, in each iterative training, a speech frame sequence is obtained for the second type of training samples called, and for the multiple consecutive speech frames after the last speech frame of the speech frame sequence, the specific number is determined according to the preset maximum compensation number, and the acoustic features of these speech frames are replaced with mask representations. For example, the maximum compensation number is determined as 6 speech frames according to the duration of 120ms, and the acoustic features of the 6 consecutive subsequent speech frames after the speech frame sequence are replaced with mask representations. The mask representation method can be, for example, replacing all the feature values ​​of the corresponding acoustic features with values ​​1, 0.5, etc., which can be flexibly set.

[0207] Step S1532: the speech synthesis model iteratively generates subsequent speech frames corresponding to the plurality of subsequent speech frames based on the speech frame sequence of the training sample;

[0208] For the second type of training samples, the speech synthesis model first starts to generate subsequent speech frames based on the acoustic features of the first speech frame sequence of the second type of training samples. After generating a subsequent speech frame, the subsequent speech frame is queued as the last speech frame in the speech frame sequence to obtain a new speech frame sequence, and then continues to iteratively extract the acoustic features of the new speech frame sequence for generating the next subsequent speech frame. This iteration is performed until multiple subsequent speech frames corresponding to the maximum compensation number are generated.

[0209] Step S1533: Calculate the loss values ​​of the corresponding subsequent speech frames according to the multiple subsequent speech frames in the second type of training samples, and correct the weights of the autoregressive network in the speech synthesis model according to the loss values.

[0210] In the process of the speech synthesis model generating multiple subsequent speech frames for each second-category training sample, for each subsequent speech frame, the speech synthesis model uses the subsequent speech frame whose timestamp in the training sample corresponds to the subsequent speech frame as the supervisory label of the subsequent speech frame, calculates the loss value of the subsequent speech frame, and performs backpropagation on the speech synthesis model based on the loss value, and gradient updates the weights of the autoregressive network, while the conditional network does not participate in the gradient update because its weights have been frozen.

[0211] According to the above embodiments, it is not difficult to understand that by replacing the acoustic features of the subsequent speech frames in the second type of training samples with mask representations during the third stage of training, the autoregressive network can be guided to generate subsequent speech frames corresponding to the subsequent speech frames, and can continuously iterate and generate corresponding multiple subsequent speech frames according to the preset maximum compensation number, so that the speech synthesis model learns the ability to continuously compensate for multiple speech frames, thereby improving the efficiency of the speech synthesis model in performing packet loss compensation.

[0212] See also Figure 8 According to one aspect of the present application, a speech synthesis model generation device is provided, comprising a coding generation module 1100, a vocoder construction module 1200, an iterative decision module 1300, and a model generation module 1400, wherein: the coding generation module 1100 is configured to call a controller, and the controller generates a structural code of the vocoder; the vocoder construction module 1200 is configured to construct a vocoder according to the structural code, and the vocoder includes a conditional network and an autoregressive network generated according to the structural code; the iterative decision module 1300 is configured to use a training set to iteratively train the vocoder to a convergence state, and according to the performance score obtained by the vocoder on the test set, the controller is gradient updated and a new vocoder is iteratively generated before the controller reaches convergence; the model generation module 1400 is configured to select the vocoder as the speech synthesis model according to the performance score after the controller reaches convergence.

[0213] Based on any of the above embodiments, the vocoder construction module 1200 includes: a first construction unit, configured to construct an upsampling network in a conditional network of the vocoder according to first coding information in structural coding, wherein the first coding information includes layer types corresponding to multiple convolutional layers of the upsampling network and their corresponding number of channels; a second construction unit, configured to construct a gated recurrent unit in an autoregressive network of the vocoder according to second coding information in structural coding, wherein the second coding information includes an operation type corresponding to a structural node of the gated recurrent unit and its corresponding activation type; and an overall construction unit, configured to construct the conditional network and the autoregressive network into a vocoder according to a preset topological structure.

[0214] On the basis of any of the above embodiments, the iterative decision module 1300 includes: a sample processing unit, configured to call a single training sample in a training set, obtain a plurality of continuous speech frames of a predetermined duration therein to construct a speech frame sequence, and extract the acoustic features corresponding to the speech sequence; a feature representation unit, configured to input the acoustic features into a conditional network of a vocoder, obtain the global feature information of the acoustic features through the residual network therein, obtain the local feature information of the acoustic features at multiple scales through the upsampling network therein, and obtain comprehensive feature information composed of the global feature information and the local feature information; a feature prediction unit, configured to extract the relatively stable style features in the speech frame sequence from the comprehensive feature information through the gated recurrent unit of the autoregressive network in the vocoder, and obtain the predicted feature information; a speech frame generation unit, configured to generate subsequent speech frames of the speech frame sequence according to the predicted feature information through the classification network in the vocoder; and a loss calculation unit, configured to calculate the loss value of the subsequent speech frame using the speech frame that is sequentially consecutive in the speech frame sequence in the training sample, and control the iterative training of the vocoder according to the loss value.

[0215] Based on any of the above embodiments, the iterative decision module 1300 includes: a test scoring unit, configured to use test samples in a test set to test the vocoder to obtain a performance score, wherein the performance score includes a quality score obtained after the test sample is processed by the vocoder; a gradient optimization unit, configured to implement policy gradient optimization on the controller according to the performance score, and continue to iteratively call the controller to generate a new vocoder when the controller has not reached a convergence state.

[0216] Based on any of the above embodiments, after the model output module 1400, it includes: a model training module, which is configured to train the speech synthesis model to a convergence state; a packet loss compensation module, which is configured to configure the speech synthesis model to smoothly connect subsequent speech frames generated according to the voice stream to the voice stream.

[0217] On the basis of any of the above embodiments, the packet loss compensation module includes: a first training unit, configured to implement a first-stage training on the vocoder using the first type of training samples in the data set, and train the vocoder to a convergence state, and the training is implemented according to a preset weight sparsification target in the first stage training; a second training unit, configured to implement a second-stage training on the vocoder using the second type of training samples in the data set, and train the vocoder to a convergence state; a third training unit, configured to solidify the weight of the conditional network of the vocoder, and implement the third-stage training on the vocoder using the second type of training samples in the data set, and train the vocoder to a convergence state to adjust the weight of the autoregressive network in the vocoder; wherein the training sample is audio data, including a plurality of speech frames that are continuous in time sequence, and the speech frames that are continuous in time sequence are used to calculate the loss value of the subsequent speech frames generated by the vocoder corresponding to the speech frames that are continuous in time sequence, and the second type of training samples are audio data corresponding to pure human voice segments.

[0218] Based on any of the above embodiments, the third training unit includes: a mask processing subunit, which is configured to replace the acoustic features of a preset number of subsequent speech frames that are consecutive in time sequence from the last speech frame in the speech frame sequence sampled from each second-category training sample with a mask representation; an iterative generation subunit, which is configured to iteratively generate subsequent speech frames corresponding to the multiple subsequent speech frames based on the speech frame sequence of the training sample by the vocoder; and a weight correction subunit, which is configured to calculate the loss values ​​of the corresponding subsequent speech frames based on the multiple subsequent speech frames in the second-category training sample, and correct the weights of the autoregressive network in the vocoder according to the loss values.

[0219] Another embodiment of the present application also provides a speech synthesis model generation device. Fig. 9 As shown, a schematic diagram of the internal structure of a speech synthesis model generation device. The speech synthesis model generation device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable non-volatile readable storage medium of the speech synthesis model generation device stores an operating system, a database, and computer-readable instructions. The database may store an information sequence. When the computer-readable instructions are executed by the processor, the processor may implement a speech synthesis model generation method.

[0220] The processor of the speech synthesis model generation device is used to provide computing and control capabilities to support the operation of the entire speech synthesis model generation device. The memory of the speech synthesis model generation device may store computer-readable instructions, which, when executed by the processor, may enable the processor to execute the speech synthesis model generation method of the present application. The network interface of the speech synthesis model generation device is used to connect and communicate with a terminal.

[0221] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the speech synthesis model generation device to which the scheme of the present application is applied. The specific speech synthesis model generation device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0222] In this embodiment, the processor is used to execute Figure 8 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to realize data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the speech synthesis model generation device of this application, and the server can call the program code and data of the server to execute the functions of all modules.

[0223] The present application also provides a non-volatile readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the speech synthesis model generation method of any embodiment of the present application.

[0224] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.

[0225] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0226] To summarize, the present application realizes the automatic generation and selection of the vocoder with the help of the controller, so that the obtained speech synthesis model meets the requirements of mobile device deployment, can obtain good performance after being deployed on the mobile device, and can meet the requirements of model miniaturization and high real-time performance in the speech synthesis scenario.

Claims

1. A method for generating a speech synthesis model, characterized in that: include: calling a controller, and generating a structural code of the vocoder by the controller; Constructing a vocoder according to the structural coding, the vocoder comprising a conditional network and an autoregressive network generated according to the structural coding; Iteratively training the vocoder to a converged state using a training set, and performing a gradient update on the controller and iteratively generating a new vocoder before the controller reaches convergence according to a performance score of the vocoder on a test set; After the controller reaches convergence, the vocoder is selected as the speech synthesis model based on the performance score.

2. The method for generating a speech synthesis model according to claim 1, characterized in that: The method of constructing a vocoder according to structural coding includes: constructing an upsampling network in a conditional network of a vocoder according to first encoding information in the structural encoding, wherein the first encoding information includes layer types corresponding to a plurality of convolutional layers of the upsampling network and the corresponding number of channels; constructing a gated recurrent unit in an autoregressive network of a vocoder according to second encoding information in the structural encoding, wherein the second encoding information includes an operation type corresponding to a structural node of the gated recurrent unit and its corresponding activation type; The conditional network and the autoregressive network are constructed as a vocoder according to a preset topology.

3. The method for generating a speech synthesis model according to claim 1, characterized in that: Iteratively training the vocoder to a converged state using a training set includes: Calling a single training sample in the training set, obtaining a plurality of continuous speech frames of a predetermined duration therein to construct a speech frame sequence, and extracting acoustic features corresponding to the speech sequence; Inputting the acoustic feature into a conditional network of a vocoder, obtaining global feature information of the acoustic feature through a residual network therein, obtaining local feature information of the acoustic feature at multiple scales through an upsampling network therein, and obtaining comprehensive feature information consisting of the global feature information and the local feature information; Extracting relatively stable style features in the speech frame sequence from the comprehensive feature information through a gated recurrent unit of an autoregressive network in a vocoder to obtain predicted feature information; generating subsequent speech frames of the speech frame sequence according to the predicted feature information via a classification network in the vocoder; The loss value of the subsequent speech frame is calculated by using the speech frame that is consecutively following in time sequence in the speech frame sequence in the training sample, and the iterative training of the vocoder is controlled according to the loss value.

4. The method for generating a speech synthesis model according to claim 1, characterized in that: According to the performance score of the vocoder on the test set, the controller is gradient updated and a new vocoder is iteratively generated before the controller reaches convergence, including: Testing the vocoder using the test samples in the test set to obtain a performance score, the performance score including a quality score obtained after the test samples are processed by the vocoder; The controller is optimized by policy gradient according to the performance score, and the controller is iteratively called to generate a new vocoder when the controller fails to reach a convergence state.

5. The method for generating a speech synthesis model according to claim 1, characterized in that: After the controller reaches convergence, the step of selecting the vocoder as the speech synthesis model according to the performance score includes: Training the speech synthesis model to a convergence state; The speech synthesis model is configured to smoothly connect subsequent speech frames generated according to the speech stream to the speech stream.

6. The method for generating a speech synthesis model according to claim 5, characterized in that: The step of training the speech synthesis model to a convergence state includes: The first stage of training is performed on the speech synthesis model using the first type of training samples in the data set, and the speech synthesis model is trained to a convergence state. In the first stage of training, the training is performed according to the preset weight sparsification target; The second stage of training is performed on the speech synthesis model using the second type of training samples in the data set, and the speech synthesis model is trained to a convergence state; Solidifying the weights of the conditional network of the speech synthesis model, performing the third stage of training on the speech synthesis model using the second type of training samples in the data set, and training the speech synthesis model to a convergence state to adjust the weights of the autoregressive network in the speech synthesis model; Among them, the training samples are audio data, including multiple temporally continuous speech frames. The temporally consecutive speech frames are used to calculate the loss values ​​of subsequent speech frames generated by the speech synthesis model corresponding to the temporally consecutive speech frames. The second type of training samples are audio data corresponding to pure human voice fragments.

7. The method for generating a speech synthesis model according to claim 6, characterized in that: The third stage of training the speech synthesis model is carried out using the second type of training samples in the data set, including: Replacing the acoustic features of a preset number of subsequent speech frames that are sequentially consecutive to the last speech frame in the speech frame sequence sampled from each second type of training sample with mask representations; The speech synthesis model iteratively generates subsequent speech frames corresponding to the plurality of subsequent speech frames based on the speech frame sequence of the training sample; The loss values ​​of the corresponding subsequent speech frames are calculated according to the multiple subsequent speech frames in the second type of training samples, and the weights of the autoregressive network in the speech synthesis model are corrected according to the loss values.

8. A speech synthesis model generation device, characterized in that: include: A code generation module is configured to call the controller, and the controller generates a structural code of the vocoder; A vocoder construction module, configured to construct a vocoder according to the structural coding, the vocoder comprising a conditional network and an autoregressive network generated according to the structural coding; an iterative decision module, configured to iteratively train the vocoder to a converged state using a training set, and to perform a gradient update on the controller and iteratively generate a new vocoder before the controller reaches convergence according to a performance score obtained by the vocoder on a test set; The model output module is configured to select the vocoder as the speech synthesis model based on the performance score after the controller reaches convergence.

9. A speech synthesis model generation device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A non-volatile readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

11. A computer program product, characterized in that The method comprises a computer program / instruction, which implements the steps of the method according to any one of claims 1 to 7 when executed by a processor.

Images