Speech style transfer model training, speech style transfer method and device

Abstract

The present disclosure provides a training of a speech style transfer model, a speech style transfer method and a device, and relates to the field of artificial intelligence technology, and in particular to the field of speech synthesis. The method comprises: inputting the spectral features of the sample audio into the content coding network and the speaker coding network of the speech style transfer model respectively, obtaining the content feature vector and the speaker feature vector of the sample audio, and extracting the fundamental frequency of the sample audio; determining the mutual information of the content feature vector, the speaker feature vector and the fundamental frequency; inputting the content feature vector, the speaker feature vector and the fundamental frequency into the spectral decoding network of the speech style transfer model to obtain the predicted spectral features; and updating the parameters of the speech style transfer model according to the predicted spectral features and the mutual information. The method optimizes the effect of speech style transfer.

Description

Technical Field

[0001] The present disclosure relates to speech synthesis technology in the field of artificial intelligence technology, and in particular to the training of a speech style transfer model, a speech style transfer method and a device. Background Art

[0002] Voice style transfer is often used in scenarios such as voice changing systems, voice chat, and online games. It outputs the sound of one voice style in other voice styles to hide the user's identity or increase entertainment, etc.

[0003] Speech style transfer is often achieved using a speech style transfer model, which is used to extract the content features of the source audio and the speaker features of the target audio, and output the spectral features of the source audio content but the target audio speaker style. Finally, a vocoder is used to convert the spectral features into audio output.

[0004] When training the speech style transfer model, the content features and speaker features of the sample audio are extracted, the predicted spectral features of the sample audio are output, and the model parameters are updated using the difference between the predicted spectral features and the actual spectral features of the sample audio. Since there is a dependency between the content features and speaker features of the sample audio, the speech style transfer effect is poor during the application process. Summary of the invention

[0005] The present disclosure provides a speech style transfer model training, speech style transfer method and device that optimize the speech style transfer effect.

[0006] According to one aspect of the present disclosure, there is provided a

[0007] The training method of the speech style transfer model includes:

[0008] Inputting the spectral features of the sample audio into the content encoding network and the speaker encoding network of the speech style transfer model respectively, obtaining the content feature vector and the speaker feature vector of the sample audio, and extracting the fundamental frequency of the sample audio;

[0009] Determining mutual information among the content feature vector, the speaker feature vector and the pitch frequency;

[0010] Inputting the content feature vector, the speaker feature vector and the fundamental frequency into the spectrogram decoding network of the speech style transfer model to obtain predicted spectrogram features;

[0011] The parameters of the speech style transfer model are updated according to the predicted spectral features and the mutual information.

[0012] According to another aspect of the present disclosure, a method for voice style transfer is provided, comprising:

[0013] Acquire a first spectral feature of the source audio and a second spectral feature of the target audio;

[0014] Inputting the first spectral feature into a content coding network of a speech style transfer model to obtain a content feature vector of the source audio, and extracting a fundamental frequency of the source audio;

[0015] Inputting the second spectral feature into a speaker encoding network of a speech style transfer model to obtain a speaker feature vector of the target audio;

[0016] Inputting the content feature vector of the source audio, the speaker feature vector of the target audio, and the fundamental frequency of the source audio into the spectrogram decoding network of the speech style transfer model to obtain the spectrogram features to be output, and generating the audio to be output according to the spectrogram features to be output;

[0017] Wherein, the speech style transfer model is trained according to the aforementioned method.

[0018] According to another aspect of the present disclosure, a training device for a speech style transfer model is provided, comprising:

[0019] A feature extraction module, used to input the spectral features of the sample audio into the content coding network and the speaker coding network of the speech style transfer model respectively, to obtain the content feature vector and the speaker feature vector of the sample audio, and to extract the fundamental frequency of the sample audio;

[0020] A mutual information calculation module, used to determine the mutual information of the content feature vector, the speaker feature vector and the fundamental frequency;

[0021] A spectrum decoding module, used for inputting the content feature vector, the speaker feature vector and the fundamental frequency into the spectrum decoding network of the speech style transfer model to obtain predicted spectrum features;

[0022] An updating module is used to update the parameters of the speech style transfer model according to the predicted spectral features and the mutual information.

[0023] According to another aspect of the present disclosure, a speech style transfer apparatus is provided, comprising:

[0024] An acquisition module, used to acquire a first spectrum feature of the source audio and a second spectrum feature of the target audio;

[0025] A content encoding module, configured to input the first spectral feature into a content encoding network of a speech style transfer model to obtain a content feature vector of the source audio and extract a fundamental frequency of the source audio;

[0026] A speaker encoding module, used for inputting the second spectral feature into a speaker encoding network of a speech style transfer model to obtain a speaker feature vector of the target audio;

[0027] A spectrum decoding module, used for inputting the content feature vector of the source audio, the speaker feature vector of the target audio and the fundamental frequency of the source audio into the spectrum decoding network of the speech style transfer model to obtain the spectrum features to be output, and generating the audio to be output according to the spectrum features to be output;

[0028] Wherein, the speech style transfer model is trained according to the aforementioned method.

[0029] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0030] at least one processor; and

[0031] a memory communicatively connected to the at least one processor; wherein,

[0032] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect above.

[0033] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0034] According to another aspect of the present disclosure, a computer program product is provided, the program product comprising: a computer program, the computer program being stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, and the at least one processor executing the computer program so that the electronic device executes the method described in the first aspect.

[0035] According to the technical solution disclosed in the present invention, the effect of speech style transfer is optimized.

[0036] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0038] Figure 1 It is a flowchart of a method for training a speech style transfer model provided in an embodiment of the present disclosure;

[0039] Figure 2 is a schematic diagram of a speech style transfer model provided according to an embodiment of the present disclosure;

[0040] Figure 3 is a schematic diagram of a content encoding network provided according to an embodiment of the present disclosure;

[0041] Figure 4 is a flow chart of a method for voice style transfer provided according to an embodiment of the present disclosure;

[0042] Figure 5 is a structural schematic diagram of a training device for a speech style transfer model provided according to an embodiment of the present disclosure;

[0043] Figure 6 is a structural schematic diagram of a speech style transfer device provided according to an embodiment of the present disclosure;

[0044] Figure 7 It is a schematic block diagram of an electronic device for implementing the method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0046] The speech style transfer model usually includes at least a content encoding network, a speaker encoding network, and a spectrogram decoding network. The content encoding network is used to extract the content features of the source audio, the speaker encoding network is used to extract the speaker features of the target audio, and the spectrogram decoding network is used to output the spectrogram features according to the content features of the source audio and the speaker features of the target audio. Finally, the spectrogram features are converted into audio, which is the audio with the content of the source audio but the speech style of the target audio speaker.

[0047] When training the speech style transfer model, for the sample audio, the content features of the sample audio are extracted through the content coding network, and at the same time, the speaker features of the sample audio are extracted through the speaker coding network. Finally, the spectrogram decoding network outputs the predicted spectrogram features of the sample audio based on the content features of the sample audio and the speaker features of the sample audio. The content of the audio corresponding to the predicted spectrogram features is the content of the sample audio and the speech style is the style of the speaker of the sample audio. The model parameters are updated using the difference between the predicted spectrogram features and the real spectrogram features of the sample audio. In this way, after the model training is completed, when using the speech style transfer model for speech style transfer, it is only necessary to change the input of the speaker coding network to the target audio so that the speaker coding network outputs the speaker features of the target audio, and the speech style transfer can be achieved.

[0048] However, when training the speech style transfer model, the content features and speaker features extracted from the sample audio may not be decoupled, that is, there may be a dependency between the content features and the speaker features. This may lead to poor speech style transfer effect after the input of the speaker encoding network is changed to the target audio in the application after the training is completed.

[0049] To this end, the embodiment of the present disclosure proposes that in the speech style transfer model, in addition to using the content encoding module to extract content features and using the speaker encoding module to extract speaker features, the fundamental frequency of the audio is also extracted, and the three features are used to determine the predicted spectral features. When training the model, in addition to updating the model parameters according to the predicted spectral features, the mutual information (MI) between the content features, speaker features and fundamental frequency is also used. The mutual information between the two features can be regarded as the amount of information contained in one feature about another random feature, or the uncertainty reduced by one feature due to the knowledge of another feature, which is used to measure the mutuality between the two features. In the training process, by reducing the mutual information, the content features, speaker features and fundamental frequency are decoupled as much as possible, that is, the dependency between them is reduced, so that the information contained in each feature does not overlap, so that it can be ensured that in the subsequent application process, the input of the speaker encoding network is changed to the target audio. After obtaining the speaker features of the target audio, the spectral features of the speech style of the target audio can be obtained by using the model, which is not affected or interfered by other features, and the speech style transfer effect is better.

[0050] The present disclosure provides a training of a speech style transfer model, a speech style transfer method and a device, which are applied to the field of speech synthesis in the field of artificial intelligence technology, and can be specifically applied to scenarios such as voice changing systems, voice chats, and online games to optimize the effect of speech style transfer.

[0051] The following will describe in detail the training method of the speech style transfer model and the speech style transfer method provided by the present disclosure through specific embodiments. It is understandable that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0052] Figure 1 1 is a flow chart of a method for training a speech style transfer model according to an embodiment of the present disclosure. The method is performed by a training device for a speech style transfer model, which can be implemented by software and / or hardware. Figure 1 As shown, the method includes:

[0053] S101, inputting the spectral features of the sample audio into the content encoding network and the speaker encoding network of the speech style transfer model respectively, obtaining the content feature vector and the speaker feature vector of the sample audio, and extracting the fundamental frequency of the sample audio.

[0054] For example, the sample audio may adopt open source data sets such as Aishell, LJSpeech, VCTK, etc., and the spectral features of the sample audio may be MFCC, PLP or Fbank features, etc., which are not limited in the embodiments of the present application. The spectral features of the sample audio are input into the content coding network to obtain the content feature vector of the sample audio; the spectral features of the sample audio are input into the speaker coding network to obtain the speaker feature vector of the sample audio.

[0055] The speaker coding network models the characteristics of the speaker identity and extracts the speaker's features. In addition, phonemes that are not related to the speaker's characteristics, such as stress and pause, are characterized by the fundamental frequency. The fundamental frequency of the sample audio can be obtained by digital signal processing.

[0056] S102: Determine the mutual information of the content feature vector, the speaker feature vector, and the fundamental frequency.

[0057] In order to achieve decoupling between the three features of content feature vector, speaker feature vector and fundamental frequency and reduce the dependency between them through model training, the mutual information between these three features is introduced in the model training process in the embodiment of the present disclosure. It can be understood that the mutual information between the three features here may refer to the mutual information between each of them. The lower the mutual information between two features, the lower the correlation between them, and the less overlapping the information they each contain.

[0058] S103, inputting the content feature vector, the speaker feature vector and the fundamental frequency into the spectrogram decoding network of the speech style transfer model to obtain predicted spectrogram features.

[0059] The input of the spectrogram decoding network is the content feature vector, the speaker feature vector and the fundamental frequency, and the output is the predicted spectrogram feature. Since the content feature vector, the speaker feature vector and the fundamental frequency are all features of the sample audio, the spectrogram decoding network in this step actually reconstructs the spectrogram features of the sample audio based on the above three features to obtain the predicted spectrogram features.

[0060] S104: Update the parameters of the speech style transfer model according to the predicted spectral features and mutual information.

[0061] The above steps have shown that the predicted spectral features are to reconstruct the spectral features of the sample audio. Therefore, the parameters of the speech style transfer model can be adjusted according to the predicted spectral features and the spectral features of the sample audio. At the same time, in order to ensure the decoupling between the three features of the content feature vector, the speaker feature vector and the fundamental frequency, reducing the mutual information is also taken as the goal of training optimization. According to the predicted spectral features and the mutual information, the parameters of the speech style transfer model are updated. On the basis of ensuring the accuracy of the predicted spectral features, the content feature vector, the speaker feature vector and the fundamental frequency are decoupled.

[0062] The method of the disclosed embodiment extracts the content feature vector, speaker feature vector and fundamental frequency of the sample audio, and uses these three features to determine the predicted spectral features. When training the model, in addition to updating the model parameters according to the predicted spectral features, the mutual information between the content feature vector, speaker feature vector and fundamental frequency of the sample audio is also added, and reducing the mutual information is also used as an optimization goal of model training. During the training process, the content feature vector, speaker feature vector and fundamental frequency are decoupled by reducing the mutual information, so that in the subsequent application process, the input of the speaker coding network is changed to the target audio, so that the speaker coding network outputs the speaker feature vector of the target audio, that is, the spectral decoding network accurately outputs the spectral features of the speech style of the target audio without being affected or interfered by other features, so that the speech style transfer effect is better.

[0063] Based on the above embodiments, the implementation method of each step is further described.

[0064] Optionally, in step S101, the spectral features of the sample audio are input into the content coding network of the speech style transfer model to obtain a content feature vector of the sample audio, including:

[0065] The spectral feature is input into a first feature extraction network in a content coding network to obtain a first feature vector, wherein the frame number of the first feature vector is lower than the frame number of the spectral feature; the first feature vector is input into a quantizer in the content coding network to determine a second feature vector having the highest similarity to the first feature vector; and the second feature vector is updated using a comparative prediction coding method to obtain a content feature vector.

[0066] Optionally, in step S104, the parameters of the speech style transfer model are updated according to the predicted spectral features and mutual information, including: determining a first loss of a content encoding network according to a content feature vector; determining a second loss of a spectral decoding network according to the predicted spectral features; and updating the parameters of the speech style transfer model according to the first loss, the second loss and the mutual information.

[0067] Optionally, a first sub-loss is determined according to the second eigenvector, and a second sub-loss is determined according to the content eigenvector; and the sum of the first sub-loss and the second sub-loss is determined as the first loss.

[0068] The following reference Figure 2 and Figure 3 Provide explanation.

[0069] The content coding network is composed of {h-net+quantizer+g-net}, where h-net is the first feature extraction network. Optionally, h-net is composed of {Conv+{LayerNorm+Linear+ReLU}*4+Linear}; the quantizer is a codebook composed of multiple learnable vectors, for example, 512 64-dimensional learnable vectors; and g-net is composed of a unidirectional RNN.

[0070] The input of the content coding network is the spectral features of the sample audio. For example, the dimension of the spectral features can be 20, each frame is 25ms long, and the frame shift is 10ms. The content coding network can carry a certain number of contexts during calculation, such as 2 frames of context each. The output of the content coding network is a content feature vector extracted from the sample audio.

[0071] Among them, h-net maps the spectral features X (T frames) of the input sample audio to the first feature vector Z (for example, T / 2 frames), and extracts high-level features from the spectral features of the sample audio by compressing the spectral features X of the sample audio, so that the redundant information in the obtained first feature vector Z is less. After that, the sampling dictionary learning algorithm (DictionaryLearning) vector quantization (Vector Quantization, VQ) method is used to find the most similar vector to each column of the first feature vector Z from the codebook to form the second feature vector In this way, the first eigenvector Z is compressed into a lower dimensional and compact space, removing unimportant details and retaining basic linguistic information. The compression mapping of the first eigenvector Z is completed by minimizing the vector quantization loss. Determine the first sub-loss, namely the vector quantization loss L VQ as follows:

[0072]

[0073] Where K is the number of sample audios, and sg represents the stop-gradient.

[0074] The input of g-net is the second eigenvector G-net is used to calculate the second eigenvector Update to get the updated Updated This is the content feature vector output by the content coding network. Among them, g-net uses the parameter matrix W to multiply the second feature vector The second eigenvector During the training process, in order to make the second eigenvector It is possible to learn more local structural information while discarding the underlying low-level information and noise. In the embodiment of the present disclosure, an unsupervised contrastive predictive coding (CPC) method is introduced to extract compact potential feature representations using future or contextual information, and autoregressive modeling and noise contrast estimation are combined with predictive coding to learn abstract representations in an unsupervised manner. By minimizing the information noise contrast estimation (Information Noise-Contrastive Estimation, lnfoNCE) loss L CPC VQ (as encoder) and g-net (as autoregressive model) are jointly trained, that is, the second sub-loss, i.e., the information-noise contrast loss L, is determined according to the content feature vector. CPC as follows:

[0075]

[0076] Where, T′=T / 2-M,W m (m=1, 2, ..., M); K is the number of sample audios; W is the parameter matrix to be trained in g-net, M is a constant, and M in the example is 6; is a positive sample, which is from the negative sample set Ω k，t，m The number of samples in the example set is 10; r k，tIt is the second feature vector used in g-net to predict the time after t. The middle value of .

[0077] In this way, the sum of the first sub-loss and the second sub-loss is determined as the first loss of the content coding network, that is, the total loss of each part in the content coding network. By calculating the first sub-loss and the second sub-loss, the accuracy of the information contained in the content feature vector output by the trained content coding network can be guaranteed.

[0078] Optionally, the speaker encoding network is composed of {ConvBank*8+Conv1d*12+average-pooling+Linear*4}. The input of the speaker encoding network is the spectral feature X of the sample audio. The ConvBank layer is used to expand the receptive field and capture long-term information. Finally, the pooling layer is used to aggregate the information features and finally output the speaker feature vector s.

[0079] The speaker coding network mainly models the characteristics of the speaker. The speaker's speaking characteristics are composed of conscious and unconscious activities, which are generally very variable. From the perspective of the spectrum, there are three aspects. On the one hand, there are internal stable coarse-grained characteristics, which are determined by the speaker's physiological structure characteristics. Different speakers have different subjective auditory perceptions, which are manifested in the low-frequency area of ​​the spectrum, such as the average pitch, the spectral envelope reflecting the impulse response of the vocal tract, the relative amplitude and position of the formant, etc. On the other hand, there are unstable short-term acoustic characteristics, such as the sharp and rapid jitter of the pronunciation speed, frequency, loudness, and the refined structure of the spectrum. These can reflect the changes in the speaker's psychology or spirit. Different emotions and intentions can be expressed by changing these during conversation. The above two aspects of features are extracted by the speaker coding network. In the speaker coding network, features of different levels are extracted through a multi-layer convolutional network, and then integrated to ensure the accuracy of the speaker's characteristics.

[0080] The last aspect is stress, pauses, etc. which are unrelated to the speaker himself, and they are characterized by the fundamental frequency p. The fundamental frequency p can be determined by digital signal processing methods such as discrete Fourier transform (DFT) and harmonic summation method.

[0081] After the content feature vector, speaker feature vector and fundamental frequency of the sample audio are determined, the mutual information between them can be determined. In the disclosed embodiment, the upper limit of the mutual information between the content feature vector, the speaker feature vector and the fundamental frequency is determined by comparing the logarithmic ratio upper limit algorithm; the sum of the upper limits of the mutual information between the content feature vector, the speaker feature vector and the fundamental frequency is determined as the mutual information between the content feature vector, the speaker feature vector and the fundamental frequency.

[0082] As described in the above embodiments, if there is no clear constraint on the three features of the content feature vector, speaker feature vector and fundamental frequency, it is very likely that the three features will be dependent on each other, for example, the content feature contains part of the speaker feature, and the speaker feature contains part of the fundamental frequency, etc., which will lead to unsatisfactory effect of speech style transfer. Therefore, in the embodiments disclosed herein, from the perspective of information theory, mutual information is used to constrain the dependency between the above features. In order to ensure that the mutual information between them is reduced, the algorithm of Contrastive Log-ratio Upper Bound (CLUB) is introduced in the embodiments disclosed herein to estimate the upper limit of the mutual information between them, and the upper limit is used as the mutual information between them. In the training process, reducing the mutual information is used as the optimization goal, so as to ensure that the correlation of the above three features extracted by the model obtained after training is the lowest.

[0083] In the related art, the mutual information I(u, v) between two features u and v is calculated by calculating the relative entropy (Kullback-Leibler Divergence) between the joint distribution and the marginal distribution of the two features, that is:

[0084] I(u,v)=D KL (P(u, v); P(u)P(v))

[0085] After the introduction of the CLUB algorithm, the upper limit of the mutual information is as follows:

[0086]

[0087] in, Represents an estimate of the true distribution of u given v, Through the neural network θ u，v To calculate it. Optionally, estimate it through the variable estimation neural network {Linear+ReLU+Linear} The mean of is estimated by {Linear+ReLU+Linear+Tanh} The log-likelihood is calculated and the neural network θ is updated by maximizing the following loss u，v :

[0088]

[0089] Substituting the content feature vector, speaker feature vector and pitch frequency respectively, we can get the mutual information between these three features as follows:

[0090]

[0091]

[0092]

[0093] in Therefore, the overall mutual information of the content feature vector, speaker feature vector and fundamental frequency is L MI :

[0094]

[0095] Optionally, the spectrogram decoding network is composed of {LSTM+Conv1d+LSTM*2+Linear}. The input of the spectrogram decoding network is the concatenation of the content feature vector of the sample audio, the speaker feature vector and the fundamental frequency. The output is the predicted spectrogram feature. In order to make the predicted spectrogram feature approach the real spectrogram feature of the sample audio, the error loss, i.e., the second loss, is calculated by the mean square error (MSE) to update the spectrogram decoding network. The error loss L REC as follows:

[0096]

[0097] When the content feature vector, the speaker feature vector and the pitch frequency are concatenated, the content feature vector may be upsampled, that is, expanded to T frames, for example, by linear interpolation. In addition, the speaker feature vector is copied and repeated to T frames.

[0098] Through the above process, the first loss, the second loss and the mutual information are obtained, and the sum of the first loss, the second loss and the mutual information is determined as the total loss; according to the total loss, the parameters of the speech style transfer model are updated.

[0099] That is, the total loss L of the speech style transfer model VC for:

[0100] L VC =L VQ +LCPC +L REC +λ MI L MI

[0101] Among them, λ MI is a constant weight (usually 0.1 or 0.01, etc.), used to adjust L MI Weight in the total loss.

[0102] Through the method of the embodiment of the present disclosure, in the process of training the speech style transfer model, important linguistic information is retained in the content feature vector through vector quantization, and unnecessary redundant information is removed. At the same time, the model is updated using mutual information as a loss. Through an unsupervised training method, the content feature vector, the speaker feature vector and the fundamental frequency are decoupled. Therefore, in the application process after the training is completed, the target audio is input into the speaker encoding network, and the model can output the spectral characteristics of the speech style of the target audio.

[0103] Figure 4 1 is a flow chart of a method for voice style transfer according to an embodiment of the present disclosure. The method is performed by a voice style transfer device, which can be implemented by software and / or hardware. Figure 4 As shown, the method includes:

[0104] S401: Obtain a first spectrum feature of a source audio and a second spectrum feature of a target audio.

[0105] S402: Input the first spectrum feature into the content coding network of the speech style transfer model to obtain a content feature vector of the source audio, and extract the fundamental frequency of the source audio.

[0106] S403: Input the second spectral feature into a speaker encoding network of a speech style transfer model to obtain a speaker feature vector of the target audio.

[0107] S404: Input the content feature vector of the source audio, the speaker feature vector of the target audio, and the fundamental frequency of the source audio into the spectrogram decoding network of the speech style transfer model to obtain the spectrogram features to be output, and generate the audio to be output according to the spectrogram features to be output.

[0108] The speech style transfer model is trained according to the method of the aforementioned embodiment.

[0109] In the disclosed embodiment, the functions of each part in the speech style transfer model are similar to those in the aforementioned embodiment, and will not be repeated here. During the application process, the content coding network extracts the content feature vector of the source audio, and the speaker coding network extracts the speaker feature vector of the target audio. The spectrogram decoding network further obtains the output spectrum according to the content feature vector of the source audio, the speaker feature vector of the target audio, and the fundamental frequency of the source audio, and finally obtains the output audio through the output spectrum. The content of the output audio is the content of the source audio, and the speech style is the speech style of the target audio. By training the speech style transfer model using the method in the aforementioned embodiment, it can be ensured that there is no dependency and interference between the features during the application process, and the speech style transfer effect is better.

[0110] Figure 5 Schematic diagram of a training device for a speech style transfer model provided according to an embodiment of the present disclosure. Figure 5 As shown, the training device 500 of the speech style transfer model includes:

[0111] The feature extraction module 501 is used to input the spectral features of the sample audio into the content coding network and the speaker coding network of the speech style transfer model respectively, obtain the content feature vector and the speaker feature vector of the sample audio, and extract the fundamental frequency of the sample audio;

[0112] A mutual information calculation module 502, used to determine the mutual information of the content feature vector, the speaker feature vector and the fundamental frequency;

[0113] The spectrogram decoding module 503 is used to input the content feature vector, the speaker feature vector and the fundamental frequency into the spectrogram decoding network of the speech style transfer model to obtain the predicted spectrogram feature;

[0114] The updating module 504 is used to update the parameters of the speech style transfer model according to the predicted spectral features and mutual information.

[0115] In one embodiment, the update module 504 includes:

[0116] A first determining unit, configured to determine a first loss of a content coding network according to a content feature vector;

[0117] A second determining unit, configured to determine a second loss of the spectrum decoding network according to the predicted spectrum feature;

[0118] An updating unit is used to update parameters of the speech style transfer model according to the first loss, the second loss and the mutual information.

[0119] In one embodiment, the updating unit includes:

[0120] A first determining subunit, configured to determine the sum of the first loss, the second loss and the mutual information as the total loss;

[0121] The update subunit is used to update the parameters of the speech style transfer model according to the total loss.

[0122] In one embodiment, the feature extraction module 501 includes:

[0123] A first extraction unit is used to input the spectrum feature into a first feature extraction network in the content coding network to obtain a first feature vector, wherein the number of frames of the first feature vector is smaller than the number of frames of the spectrum feature;

[0124] A second extraction unit, configured to input the first feature vector into a quantizer in the content coding network to determine a second feature vector having the highest similarity to the first feature vector;

[0125] The third extraction unit is used to update the second feature vector by using a contrast prediction coding method to obtain a content feature vector.

[0126] In one implementation, the first determining unit includes:

[0127] A second determining subunit, configured to determine the first sub-loss according to the second feature vector, and determine the second sub-loss according to the content feature vector;

[0128] The third determining subunit is configured to determine the sum of the first sub-loss and the second sub-loss as the first loss.

[0129] In one implementation, the mutual information calculation module 502 includes:

[0130] A third determining unit is used to determine the upper limit of mutual information between the content feature vector, the speaker feature vector and the fundamental frequency by comparing the logarithmic ratio upper limit algorithm;

[0131] The fourth determining unit is used to determine the sum of upper limits of mutual information between the content feature vector, the speaker feature vector and the fundamental frequency as the mutual information between the content feature vector, the speaker feature vector and the fundamental frequency.

[0132] The device of the embodiment of the present disclosure can be used to execute the training method of the speech style transfer model in the above method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0133] Figure 6 is a structural diagram of a speech style transfer device provided according to an embodiment of the present disclosure. Figure 6 As shown, the speech style transfer device 600 includes:

[0134] An acquisition module 601 is used to acquire a first spectrum feature of a source audio and a second spectrum feature of a target audio;

[0135] A content encoding module 602 is used to input the first spectrum feature into a content encoding network of a speech style transfer model to obtain a content feature vector of the source audio and extract a fundamental frequency of the source audio;

[0136] A speaker encoding module 603, used for inputting the second spectrum feature into a speaker encoding network of the speech style transfer model to obtain a speaker feature vector of the target audio;

[0137] The spectrogram decoding module 604 is used to input the content feature vector of the source audio, the speaker feature vector of the target audio, and the fundamental frequency of the source audio into the spectrogram decoding network of the speech style transfer model to obtain the spectrogram features to be output, and generate the audio to be output according to the spectrogram features to be output;

[0138] The speech style transfer model is trained according to the method of the aforementioned embodiment.

[0139] The device of the embodiment of the present disclosure can be used to execute the speech style transfer method in the above method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0140] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a non-transitory computer-readable storage medium storing computer instructions.

[0141] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, the program product includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0142] Figure 7 Schematic block diagram of an electronic device for implementing the method of the embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0143] like Figure 7As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0144] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0145] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the training method of the voice style transfer model or the voice style transfer method. For example, in some embodiments, the training method of the voice style transfer model or the voice style transfer method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the training method of the voice style transfer model or the voice style transfer method described above may be executed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the training method of the speech style transfer model or the speech style transfer method in any other appropriate manner (for example, by means of firmware).

[0146] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0148] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0149] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0150] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0151] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0152] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in this disclosure can be achieved, and this document is not limited here.

[0153] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a speech style transfer model, comprising: Inputting the spectral features of the sample audio into the content encoding network and the speaker encoding network of the speech style transfer model respectively, obtaining the content feature vector and the speaker feature vector of the sample audio, and extracting the fundamental frequency of the sample audio; Determining mutual information of the content feature vector, the speaker feature vector, and the pitch frequency; Inputting the content feature vector, the speaker feature vector and the fundamental frequency into the spectrogram decoding network of the speech style transfer model to obtain predicted spectrogram features; updating the parameters of the speech style transfer model according to the predicted spectral features and the mutual information; The updating of the parameters of the speech style transfer model according to the predicted spectral features and the mutual information comprises: Determining a first loss of the content coding network according to the content feature vector; Determining a second loss of the spectrogram decoding network according to the predicted spectrogram feature; Update parameters of the speech style transfer model according to the first loss, the second loss and the mutual information.

2. The method according to claim 1, wherein: The updating of the parameters of the speech style transfer model according to the first loss, the second loss and the mutual information comprises: Determine the sum of the first loss, the second loss and the mutual information as a total loss; According to the total loss, the parameters of the speech style transfer model are updated.

3. The method according to claim 1, wherein: Inputting the spectral features of the sample audio into the content encoding network of the speech style transfer model to obtain a content feature vector of the sample audio, including: Inputting the spectrum feature into a first feature extraction network in the content coding network to obtain a first feature vector, wherein the number of frames of the first feature vector is smaller than the number of frames of the spectrum feature; Inputting the first feature vector into a quantizer in the content coding network to determine a second feature vector having the highest similarity to the first feature vector; The second feature vector is updated by using a comparative prediction coding method to obtain the content feature vector.

4. The method according to claim 3, wherein: Determining a first loss of the content coding network according to the content feature vector includes: Determine a first sub-loss according to the second feature vector, and determine a second sub-loss according to the content feature vector; The sum of the first sub-loss and the second sub-loss is determined as the first loss.

5. The method according to any one of claims 1 to 4, wherein: The determining of the mutual information of the content feature vector, the speaker feature vector and the fundamental frequency comprises: Determine the upper limit of mutual information between the content feature vector, the speaker feature vector and the fundamental frequency by comparing the logarithmic ratio upper limit algorithm; The sum of upper limits of mutual information between the content feature vector, the speaker feature vector and the fundamental pitch frequency is determined as the mutual information between the content feature vector, the speaker feature vector and the fundamental pitch frequency.

6. A method for voice style transfer, comprising: Acquire a first spectral feature of the source audio and a second spectral feature of the target audio; Inputting the first spectral feature into a content coding network of a speech style transfer model to obtain a content feature vector of the source audio, and extracting a fundamental frequency of the source audio; Inputting the second spectral feature into a speaker encoding network of a speech style transfer model to obtain a speaker feature vector of the target audio; Inputting the content feature vector of the source audio, the speaker feature vector of the target audio, and the fundamental frequency of the source audio into the spectrogram decoding network of the speech style transfer model to obtain the spectrogram features to be output, and generating the audio to be output according to the spectrogram features to be output; Wherein, the speech style transfer model is trained according to the method according to any one of claims 1-5.

7. A training device for a speech style transfer model, comprising: A feature extraction module, used to input the spectral features of the sample audio into the content coding network and the speaker coding network of the speech style transfer model respectively, to obtain the content feature vector and the speaker feature vector of the sample audio, and to extract the fundamental frequency of the sample audio; A mutual information calculation module, used to determine the mutual information of the content feature vector, the speaker feature vector and the fundamental frequency; A spectrum decoding module, used for inputting the content feature vector, the speaker feature vector and the fundamental frequency into the spectrum decoding network of the speech style transfer model to obtain predicted spectrum features; An updating module, used for updating the parameters of the speech style transfer model according to the predicted spectral features and the mutual information; Wherein, the update module includes: A first determining unit, configured to determine a first loss of the content coding network according to the content feature vector; A second determining unit, configured to determine a second loss of the spectrogram decoding network according to the predicted spectrogram feature; An updating unit is used to update the parameters of the speech style transfer model according to the first loss, the second loss and the mutual information.

8. The device according to claim 7, wherein: The updating unit comprises: A first determining subunit, configured to determine the sum of the first loss, the second loss and the mutual information as a total loss; An updating subunit is used to update the parameters of the speech style transfer model according to the total loss.

9. The device according to claim 7, wherein: The feature extraction module comprises: A first extraction unit, configured to input the spectrum feature into a first feature extraction network in the content coding network to obtain a first feature vector, wherein the number of frames of the first feature vector is smaller than the number of frames of the spectrum feature; a second extraction unit, configured to input the first feature vector into a quantizer in the content coding network to determine a second feature vector having the highest similarity to the first feature vector; The third extraction unit is used to update the second feature vector by using a comparative prediction coding method to obtain the content feature vector.

10. The device according to claim 9, wherein: The first determining unit includes: a second determining subunit, configured to determine a first sub-loss according to the second feature vector, and to determine a second sub-loss according to the content feature vector; The third determining subunit is configured to determine the sum of the first sub-loss and the second sub-loss as the first loss.

11. The device according to any one of claims 7 to 10, wherein: The mutual information calculation module comprises: A third determining unit is used to determine the upper limit of mutual information between the content feature vector, the speaker feature vector and the fundamental frequency by comparing the logarithmic ratio upper limit algorithm; The fourth determining unit is used to determine the sum of upper limits of mutual information between the content feature vector, the speaker feature vector and the fundamental pitch frequency as the mutual information between the content feature vector, the speaker feature vector and the fundamental pitch frequency.

12. A speech style transfer device, comprising: An acquisition module, used to acquire a first spectrum feature of the source audio and a second spectrum feature of the target audio; A content encoding module, configured to input the first spectral feature into a content encoding network of a speech style transfer model to obtain a content feature vector of the source audio and extract a fundamental frequency of the source audio; A speaker encoding module, used for inputting the second spectral feature into a speaker encoding network of a speech style transfer model to obtain a speaker feature vector of the target audio; A spectrum decoding module, used for inputting the content feature vector of the source audio, the speaker feature vector of the target audio and the fundamental frequency of the source audio into the spectrum decoding network of the speech style transfer model to obtain the spectrum features to be output, and generating the audio to be output according to the spectrum features to be output; Wherein, the speech style transfer model is trained according to the method according to any one of claims 1-5.

13. An electronic device, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method of any one of claims 1 to 6.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images