Cross-lingual speech conversion method fusing an enhanced coding module and a codec structure of an LGNet network

Abstract

The application discloses a cross-language speech conversion method of a coding-decoding structure of a fusion enhanced coding module and an LGNet network, comprising a training stage and a conversion stage, in the training stage, an encoder disentangles acoustic features to obtain speaker information representation and content representation; a U-shaped connection is adopted between the encoder and the decoder, and the speaker information representation is transmitted from the encoder to the decoder; an LGNet network further optimizes the extracted content representation; the decoder reconstructs the obtained speaker information representation and the optimized content representation; the application introduces an enhanced coding module in the encoder, and improves the quality of the converted speech; the LGNet network is used to make the optimized content representation of the source sentence and the speaker information representation of the target sentence fully fused in the adaptive instance normalization layer in the decoder, further improves the naturalness of the converted speech and the speaker similarity, and thus high-quality cross-language speech conversion is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cross-lingual speech conversion, in particular to a cross-lingual speech conversion method of a coding-decoding structure fusing an enhanced coding module and an LGNet network. BACKGROUND

[0002] Speech conversion is an important branch of speech signal processing field, and the research of the technology has important theoretical value and application prospect. Speech conversion is a technology of converting the identity feature of a speaker in a source sentence into the identity feature of a speaker in a target sentence while ensuring that the semantic feature of the source sentence is unchanged. Traditional speech conversion focuses on solving the problem of same-lingual speech conversion, that is, the language of the source and target sentences is required to be the same. Cross-lingual speech conversion breaks the same-lingual limitation, and the language and text of the source and target sentences are all different.

[0003] Thanks to the development of deep learning, cross-lingual voice conversion has made a breakthrough. In 2018, Mohammadi et al. proposed a cross-lingual voice conversion method based on factorized hierarchical variational autoencoder model (Investigation of using disentangled and interpretable representations for one-shot cross-lingual voice conversion. Mohammadi S H, Kim T. arXiv preprint arXiv:1808.05294, 2018.), which uses the FHVAE model to model the multi-scale information in speech. When implementing cross-lingual voice conversion, the semantic content representation and speaker information representation are first extracted from the acoustic features of different languages, and then the two are fused to obtain the converted acoustic features, so as to generate the converted speech. In 2020, Tobing et al. proposed a cross-lingual voice conversion method based on CycleVAE model (Baseline system of Voice Conversion Challenge 2020 with cyclic variational autoencoder and Parallel WaveGAN. Tobing P L, Wu Y C, Toda T. arXiv preprint arXiv:2010.04429, 2020.). This method uses the disentanglement ability of VAEs model to extract the latent encoding vector representing semantic content from the acoustic features, and uses the one-hot encoding vector to represent the speaker identity to guide the model to generate the reconstructed source acoustic features and converted acoustic features, and then input the converted acoustic features into the model again for cyclic operation. In 2022, Lu and Wang et al. proposed a cross-lingual voice conversion method based on β-VAE (Disentangled Speech Representation Learning for One-Shot Cross-Lingual Voice Conversion Using β-VAE. Lu H, Wang D, Wu X, et al. 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023:814-821.). This method takes the KL divergence between content representation and acoustic features, speaker information representation and acoustic features as the upper bound of the mutual information between the representations and acoustic features. At the same time, β c and β sTwo weight parameters are used to control the amount of data information that the content representation and the speaker information representation can capture. Dai proposed a cross-lingual voice conversion method based on activation guidance and inner convolution (Research on Cross-lingual Voice Conversion Based on Activation Guidance and Inner Convolution. Dai Shaoliang. Nanjing University of Posts and Telecommunications, 2022. DOI: 10.27251 / d.cnki.gnjdc.2022.001228.). This method uses a U-shaped connection encoder-decoder structure, and uses an activation guidance function as a soft bottleneck to extract semantic content representations of different languages, thereby realizing cross-lingual voice conversion between arbitrary speakers.

[0004] The various cross-lingual voice conversion methods mentioned above have developed from conversion in a closed set scenario to an open set scenario, i.e., arbitrary cross-lingual voice conversion between arbitrary pairs. Although the application scenarios have been expanded, the quality of the converted voice is still not satisfactory, and there are problems such as incomplete semantic content and low speaker similarity to some extent. SUMMARY

[0005] To solve the above problems, the application provides a cross-lingual speech conversion method of a coding-decoding structure of a fusion of an enhanced coding module and an LGNet network, which introduces an enhanced coding module in an encoder E. In the enhanced coding module, a recursive gate convolution is used to replace the ordinary convolution in the original coding module, which can extend the second-order interaction in self-attention to any order, realize high-order spatial interaction with limited complexity, and enable the encoder E to extract the content representation C1 of the source sentence containing more abundant information, thereby effectively solving the problem of incomplete semantic content of the converted speech in the existing cross-lingual speech conversion method and improving the quality of the converted speech. Further, an LGNet network is added between the encoder E and the decoder D. Learning from the success of the Transformer, the LGNet network replaces the batch normalization used in the traditional ResNet network with layer normalization, and only retains the normalization after the one-dimensional deep convolution layer. With the help of the layer normalization, the statistical information on the feature dimension of each sample is used to normalize each sample independently, thereby improving the generalization ability of the model. In addition, reducing the use of normalization reduces the sensitivity of the model to small perturbations. The ReLU activation function is replaced with the GELU activation function, and only the activation function after the first fully connected layer is retained. With the help of the stronger non-linear representation ability of the GELU activation function, the complex non-linear relationship can be better approximated, so that the network can learn more complex feature representations and improve the expression ability of the model. In addition, reducing the use of activation functions can reduce the probability of problems such as gradient disappearance and improve the stability of the network. These changes can improve the expression ability of the network. In addition, the LGNet network uses one-dimensional deep convolution to replace one-dimensional ordinary convolution. Therefore, compared with adding a ResNet network to the conversion model, adding an LGNet network reduces the computational complexity and thus reduces the training time of the conversion model. The content representation C1 of the source sentence is further optimized by using the LGNet network to obtain an optimized content representation LG(C1) of the source sentence, so that the information in the content representation C1 of the source sentence containing more abundant information can be fully expressed. In the decoder, the optimized content representation LG(C1) of the source sentence is fully fused with the speaker information representation S2 of the target sentence in the adaptive instance normalization layer, which improves the naturalness and speaker similarity of the converted speech, and finally realizes high-quality cross-lingual speech conversion in an open set.

[0006] The application is implemented by the following technical solutions:

[0007] A cross-lingual speech conversion method of a coding-decoding structure of a fusion of an enhanced coding module and an LGNet network, comprising a training phase and a conversion phase,

[0008] The training phase comprises the following steps:

[0009] Step S1: obtaining a large amount of training corpus, the training corpus being composed of sentences of multiple speakers in different languages;

[0010] Step S2: preprocessing and feature extraction of the obtained training corpus; 128 frames are randomly selected from the log-mel spectrum features of each training corpus, and a log-mel spectrum feature segment is obtained by splicing, which is represented as acoustic feature X;

[0011] Step S3: inputting the acoustic feature X into a conversion model for model training, the conversion model comprising an encoder E, an LGNet network and a decoder D; the encoder E disentangles the input acoustic feature X to obtain a speaker information representation S and a content representation C; the U-shaped connection is adopted between the encoder E and the decoder D, so as to pass the speaker information representation S from the encoder E to the decoder D; the LGNet network further optimizes the content representation C extracted by the encoder E to obtain LG(C), so that the information in the content representation is fully expressed; the decoder D reconstructs the obtained speaker information representation S and the optimized content representation LG(C) to obtain the reconstructed log-mel spectrum feature D(LG(C), S), which is represented as acoustic feature X'; the present application improves the encoder to improve the feature extraction capability, and the content representation containing richer information can be extracted; the U-shaped connection is adopted between the encoder and the decoder, allowing the low-level features to be directly passed to the high-level, so that the feature information of the low-level and the high-level is fused together. In this way, the network can not only pay attention to details and local features when extracting features, but also utilize global context information to improve the richness and expression capability of the features;

[0012] Step S4: during the training of the conversion model, the reconstruction loss function of the conversion model is made as small as possible, the hyperparameters of the conversion model are set to minimize the objective function, and the iteration times are set until the set iteration times are reached, so that the trained conversion model is obtained;

[0013] The conversion stage comprises the following steps:

[0014] Step S5: selecting sentences in different languages as to-be-converted corpus, selecting source sentences and target sentences in different languages from the to-be-converted corpus, and extracting log-mel spectrum features of the source sentences and log-mel spectrum features of the target sentences, respectively, which are represented as acoustic features X1 and X2;

[0015] Step S6: inputting the acoustic features X1 and X2 into the trained conversion model to reconstruct the log-mel spectrum features of the converted sentences, which are represented as acoustic features X 1→2 ;

[0016] Step S7: using a Griffin-Lim vocoder to obtain the acoustic features X1→2 The converted speech waveform is outputted, thereby obtaining a synthesized speech of cross-lingual speech conversion. The Griffin-Lim vocoder has fast calculation speed and relatively simple implementation.

[0017] Preferably, the training corpus obtained is preprocessed and feature extracted in step S2, and the specific steps are as follows: the sampling rate of the training corpus is set to 24 kHz, the number of points of fast Fourier transform is set to 2048, the window length is set to 1200, and the window moving length is set to 300.

[0018] Preferably, in step S3, the encoder E in the conversion model is composed of 2 one-dimensional convolution layers, 6 enhanced encoding modules, 6 instance normalization layers and 1 activation guide function layer, the enhanced encoding module is composed of 1 recurrent gate convolution layer, 1 batch normalization layer, 1 LeakyReLU activation function and 1 one-dimensional convolution layer, and the enhanced encoding module and the instance normalization layer are alternately placed.

[0019] Preferably, the recurrent gate convolution in the enhanced encoding module expands the second-order interaction in self-attention to an arbitrary order, and the encoder E obtains a feature vector As the input of the recurrent gate convolution in the enhanced encoding module, the input linear projection layer φ in obtains a group of projection features p0and The formula of the input linear projection layer is as follows:

[0020]

[0021] wherein, H and W represent the height and width of the feature vector Z; C represents the number of channels; n represents the order of the high-order interaction; C0, …, C n-1 represent the number of channels corresponding to different projection features; after the projection features are obtained, they are sequentially input into the recurrent gate convolution for recursive operation, and the operation formula is as follows:

[0022] p k+1 =f k (q k )⊙g k (p k ) / α,k=0,1,…,n-1

[0023] wherein, α represents a scaling coefficient; f k (·) represents a deep convolution calculation; g k (·) represents a dimension mapping function, and ⊙ represents a dot product operation; the output p n of the last step of operation is fed to the output linear projection layer φ outThe final result of the recursive gated convolution is obtained.

[0024] Further improvement, in step S3, the LGNet network in the conversion model is composed of 6 LGBlock convolution modules, each LGBlock convolution module is composed of 1 one-dimensional deep convolution layer, 1 layer normalization layer, 1 GELU activation function, 1 regularization layer and 2 fully connected layers; each LGBlock convolution module is connected by residual connection.

[0025] Preferably, the LGNet network replaces the batch normalization used in the traditional ResNet network with layer normalization, and only the normalization after the one-dimensional deep convolution layer is retained. By using the statistical information in the feature dimension of each sample through layer normalization, each sample has independent normalization, thereby improving the generalization ability of the model. In addition, reducing the use of normalization can reduce the sensitivity of the model to small perturbations; the ReLU activation function is replaced with a GELU activation function, and only the activation function after the first fully connected layer is retained. By virtue of the stronger nonlinear representation ability of the GELU activation function, it can better approximate complex nonlinear relationships, so that the network can learn more complex feature representations and improve the expression ability of the model. In addition, reducing the use of activation functions can reduce the probability of problems such as gradient disappearance, improving the stability of the network. These changes can improve the expression ability of the network.

[0026] In addition, since the LGNet network replaces one-dimensional ordinary convolution with one-dimensional deep convolution, adding the LGNet network in the conversion model will reduce the amount of calculation compared to adding the ResNet network, thereby reducing the training time of the conversion model.

[0027] The number of convolution kernels of one-dimensional deep convolution is equal to 1, and each convolution kernel is only responsible for one channel of input features, so the number of convolution kernels is equal to the number of channels of input features, so that the number of channels of output features is equal to the number of channels of input features; assuming D f is the width and height of the input feature, D k is the height and width of the convolution kernel, and M is the number of channels of the input feature; the amount of calculation of convolution is equal to the height of the convolution kernel x the width of the convolution kernel x the number of channels of the convolution kernel x the number of convolution kernels x the height of the input feature x the width of the input feature, assuming that the step is equal to 1; the amount of calculation of one-dimensional deep convolution is:

[0028] D k ×D k ×1×M×D f ×D f .

[0029] Further improvement, in step 4, the training process is as follows:

[0030] Step S4-1: input the acoustic feature X extracted from the training corpus into the encoder E to train the ability of the encoder E to extract the content representation C and the speaker information representation S;

[0031] Step S4-2: in the encoder E, the acoustic feature X is converted into a feature vector Z after passing through the first one-dimensional convolutional layer in the encoder E, and then converted into a feature vector Z' after passing through the enhanced encoding module;

[0032] Step S4-3: the first instance normalization layer in the encoder E takes the feature vector Z' as input to extract the speaker information representation S = {μ(Z'), σ(Z')}, and passes the speaker information representation S to the decoder D through the U-shaped connection; and normalizes Z' using the extracted speaker information to obtain an instance normalized representation, which is input into the last one-dimensional convolutional layer in the encoder E to obtain a feature vector Z";

[0033] Step S4-4: use the activation guide function layer as a bottleneck to further filter out the speaker information in the feature vector Z" to obtain a pure content representation C;

[0034] Step S4-5: input the pure content representation C into the LGNet network to further extract and optimize, to obtain an optimized representation LG(C) that fully expresses the information in the content representation;

[0035] Step S4-6: input the optimized content representation LG(C) and the speaker information representation S into the decoder D to train the ability of the decoder D to reconstruct the acoustic feature X';

[0036] Step S4-7: input the feature vector Y obtained by passing the content representation LG(C) through the first one-dimensional convolutional layer in the decoder and the speaker representation S extracted by the encoder E into the adaptive instance normalization layer in the decoder to strengthen the speaker information in the reconstructed acoustic feature X';

[0037] Step S4-8: jointly train the encoder E, the LGNet network and the decoder D, repeat the process of steps S4-1 to S4-7, minimize the reconstruction loss function until the set number of iterations is reached, to obtain the trained conversion model.

[0038] Preferably, the reconstruction loss function is represented as:

[0039] L rec (θ E ,θ D )=E X～p(X) [||X′-X||1]

[0040] wherein X represents the input acoustic feature of the encoder E; X' represents the reconstructed acoustic feature of the decoder D, EX～p(X) [·] represents the loss expectation between the reconstructed acoustic feature X' of the decoder D and the input acoustic feature X of the encoder E; ||·||1 represents the L1 norm; θ E represents the parameters corresponding to the encoder E; θ D represents the parameters corresponding to the decoder D.

[0041] Preferably, the objective function is represented as:

[0042]

[0043] wherein θ E represents the parameters corresponding to the encoder E; θ D represents the parameters corresponding to the decoder D, L rec represents the reconstruction loss function of the model, λ rec The hyperparameter represents the weight of the reconstruction loss function.

[0044] Further improvement, the conversion process in step S6 is specifically as follows:

[0045] Step S6-1: input the acoustic features X1 and X2 described in step S5 into the trained conversion model, and use the encoder E to extract the content representation C1 of the source sentence and the speaker information representation S2 of the target sentence;

[0046] Step S6-2: the content representation C1 of the source sentence is further extracted and optimized through the LGNet network to obtain the optimized content representation LG(C1), so that the information in the content representation is fully expressed; the speaker information representation S2 of the target sentence is directly transmitted to the decoder D through the U-shaped connection;

[0047] Step S6-3: the decoder D fully fuses the optimized content representation LG(C1) of the source sentence and the speaker information representation S2 of the target sentence in the adaptive instance normalization layer in the decoder, and then decodes through the decoding module, and finally obtains the reconstructed conversion log-mel spectrum feature D(LG(C1), S2), which is represented as the acoustic feature X 1→2 .

[0048] The present application has the following beneficial effects: (1) Compared with the prior art, the conversion model in the present application first extracts content representation containing more rich information by using the encoder E fused with the enhanced encoding module, and then further optimizes the extracted content representation by the LGNet network to realize high-quality cross-lingual speech conversion. In the encoder E, the enhanced encoding module is introduced, and the ordinary convolution in the original encoding module is replaced with the recursive gate convolution, which can extend the second-order interaction in self-attention to any order, realizing high-order spatial interaction with limited complexity. After introducing the enhanced encoding module in the encoder E, the ability of the encoder E to extract features is further improved, and the content representation C1 of the source sentence containing more rich information is obtained, thereby solving the problem of incomplete semantic content in the current cross-lingual speech conversion method and improving the quality of the converted speech.

[0049] (2) The present application adds the LGNet network between the encoder E and the decoder D. The LGNet network learns from the success experience of the Transformer, replaces the batch normalization used in the traditional ResNet network with the layer normalization, replaces the ReLU activation function with the GELU activation function, and only retains the normalization after the one-dimensional deep convolution layer and the activation function after the first fully connected layer, reducing the use of normalization and activation function. These changes can improve the expression ability of the network. In addition, the LGNet network replaces the one-dimensional ordinary convolution with the one-dimensional deep convolution. Therefore, compared with adding the ResNet network in the conversion model, adding the LGNet network can reduce the calculation amount, thereby reducing the training time of the conversion model. The LGNet network further optimizes the content representation C1 of the source sentence to obtain the optimized content representation LG(C1) of the source sentence, so that the information in the content representation is fully expressed. In the decoder D, the optimized content representation LG(C1) of the source sentence and the speaker information representation S2 of the target sentence are fully fused in the adaptive instance normalization layer, thereby further improving the naturalness and speaker similarity of the converted speech. Since the instance normalization layer in the encoder E of the present application can extract the speaker information representation S of any speaker, and the activation guide function layer can extract the content representation C of different languages, after training using a large number of speaker corpus in the training stage, the conversion model in the present application can convert the speaker sentence not participating in the training in the conversion stage, that is, it is suitable for the cross-lingual speech conversion task in the open set situation. Therefore, the present application is a cross-lingual speech conversion method with high speech quality and high speaker similarity in the open set situation. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is the principle schematic diagram of the training stage in the present application.

[0051] Figure 2 is the principle schematic diagram of the conversion stage in the present application.

[0052] Figure 3 is a network structure diagram of the encoder E in the model described in the embodiments of the present application.

[0053] Figure 4 is a partial network structure diagram of the encoder E containing the detailed structure of the enhanced encoding module.

[0054] Figure 5 is a network structure diagram of the LGNet in the model described in the embodiments of the present application.

[0055] Figure 6 is a partial network structure diagram of the LGNet containing the detailed structure of the LGBlock convolutional module.

[0056] Figure 7 is a network structure diagram of the decoder D in the model described in the embodiments of the present application.

[0057] Figure 8 is a partial network structure diagram of the decoder D containing the detailed structure of the decoding module. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0059] A cross-lingual speech conversion method fusing an encoder-decoder structure of an enhanced encoding module and an LGNet network includes a training stage and a conversion stage. The training stage is used to obtain a conversion model and parameters required for cross-lingual speech conversion, and the conversion stage is used to convert speaker information of a source sentence into speaker information of a target sentence while keeping the semantic content of the source sentence unchanged.

[0060] As shown in Figure 1 , the training stage includes the following steps:

[0061] Step S1: A large amount of training corpus is obtained, which is from the AISHELL-3 Chinese corpus and the CSTR VCTK English corpus; from the obtained Chinese and English corpora, 40 speakers in the AISHELL-3 Chinese corpus are selected, each speaker has about 200 sentences, and each sentence has a duration of 2-10s. 40 speakers in the CSTR VCTK English corpus are selected, each speaker has about 200 sentences, and each sentence has a duration of 2-8s. The selected sentences of Chinese and English are jointly used to form training corpus for training of the conversion model, so that the conversion model can fully learn the pronunciation characteristics and semantic content of different languages, thereby ensuring the performance of the conversion model;

[0062] Step S2: Acoustic features are obtained, and the obtained training corpus is preprocessed and feature extracted. Specifically, the sampling rate of the training corpus is set to 24kHz, the number of fast Fourier transform points is set to 2048, the window length is set to 1200, and the window moving length is set to 300. 128 frames are randomly selected from the log-mel spectrum features of each sentence of the training corpus, and a log-mel spectrum feature segment is obtained by splicing, which is represented as acoustic feature X;

[0063] Step S3: The acoustic feature X obtained in step S2 is input into the conversion model for model training. The network in the present example adopts an encoder-decoder structure. The acoustic feature X is first input into the enhanced encoding module in the encoder E to obtain a preliminary feature containing rich information. Then, the content and speaker information in the preliminary feature are disentangled through the instance normalization layer and the activation guide function layer, to obtain the speaker information representation S and the content representation C, respectively. Specifically, the instance normalization is used to dynamically extract the speaker information representation S, and the activation guide function is used as a bottleneck to extract the content representation C. The content representation C is further optimized by the LGNet network to obtain LG(C), so that the information in the content representation can be fully expressed. Then, the speaker information representation S is transmitted from the encoder E to the decoder D through the U-shaped connection between the encoder E and the decoder D. The two representations are fully fused by using the adaptive instance normalization layer, and then decoded and reconstructed by the decoding module, to obtain the reconstructed log-mel spectrum feature D(LG(C), S), which is represented as acoustic feature X';

[0064] The conversion model mainly includes an encoder E, an LGNet network and a decoder D:

[0065] (1) In encoder E, acoustic features X are first extracted into preliminary features containing rich information through an enhancement coding module. Then, an instance normalization layer is used to extract speaker information representation S from the preliminary features. After instance normalization, the speaker information is removed while retaining the content information of the preliminary features, thus separating the content and speaker information in the preliminary features. Finally, the features after the last instance normalization process are passed through a one-dimensional convolutional layer and then through an activation guidance function layer to obtain a clean content representation C, achieving deentanglement. Its structure is as follows: Figure 3 As shown. Encoder E consists of two one-dimensional convolutional layers, six augmentation coding modules, six instance normalization layers, and one activation guidance function layer. The augmentation coding modules are composed of... Figure 4 The diagram shows a structure consisting of a recursively gated convolutional layer, a batch normalization layer, a LeakyReLU activation function, and a one-dimensional convolutional layer. The recursively gated convolution can extend the second-order interactions in self-attention to arbitrary orders, enabling high-order spatial interactions with finite complexity and improving the encoder E's ability to extract content features. The output of the enhanced encoding module is connected to an instance normalization layer to extract speaker information representation S while filtering out speaker information from the content information. Figure 4 In the diagram, A represents the enhancement coding module and instance normalization layer as a whole, which are placed in 6 groups in sequence (the enhancement coding module and instance normalization layer are placed alternately); finally, the activation guidance function layer is used to further filter out speaker information in the content information, resulting in a clean content representation C.

[0066] In encoder E, the enhancement coding module utilizes recursively gated convolutions to extend the second-order interactions in self-attention to arbitrary orders, achieving higher-order spatial interactions with finite complexity and improving encoder E's ability to extract content features. The feature vector obtained after passing acoustic features X through the first one-dimensional convolutional layer in encoder E is... As input to the recursive gated convolution in the augmented coding module, It refers to the real number field in mathematics; the input is fed into the linear projection layer φ of the recursive gated convolution. in A set of projection features p0 and Projection features p0 and The formula for the input linear projection layer is the same as that for the feature vector Z in terms of height and width:

[0067]

[0068] in, H and W represent the height and width of the eigenvector Z, respectively, C represents the number of channels, and n represents the order of higher-order interactions, C0,…,C n-1 This represents the number of channels corresponding to different projection features. After obtaining the projection features, they are sequentially input into a recursive gated convolution for recursive computation. The computation formula is as follows:

[0069] p k+1 = f k (q k )⊙g k (p k ) / α, k = 0, 1, …, n-1

[0070] wherein, a represents a scaling coefficient, used for stabilizing training. f k (·) represents a deep convolution calculation. g k (·) represents a dimension mapping function, used for matching dimensions at different orders. ⊙ represents a dot product operation. The output p n of the last step of operation is fed to the output linear projection layer φ out to obtain the final result of the recurrent gated convolution.

[0071] In addition, the formula of the instance normalization layer is:

[0072]

[0073]

[0074]

[0075] wherein, Z' represents a feature vector after the enhanced encoding module, serving as the input of the instance normalization layer. μ nc (Z') and σ nc (Z') respectively represent the mean and standard deviation of Z' along the c direction, used for representing the speaker information representation S. Z n ' ci represents the i-th frame size of Z' along the c direction given batch n and channel number c. L represents the frame length. In the present instance, n is a positive integer between [1, 64], c is a positive integer between [1, 512], L is 128, and ε is a very small value to prevent the standard deviation from being 0.

[0076] Finally, in the encoder E, an activation guide function layer is used as a bottleneck to further filter out the speaker information, to obtain a pure content representation C. The activation guide function layer is represented as:

[0077]

[0078] wherein, Z" represents a feature vector after the last one-dimensional convolution layer in the encoder E, serving as the input of the activation guide function layer. a represents a hyperparameter. In the present instance, a is set to 0.1.

[0079] (2) In the LGNet network, the content representation C output by the encoder E is taken as input, and a feature vector LG(C) is generated as the optimized content representation. The structure of the LGNet network is shown in Figure 5 The LGNet network is composed of 6 LGBlock convolution modules, and each convolution module is composed of a 1-dimensional deep convolution layer, a layer normalization layer, a GELU activation function, a regularization layer, and two fully connected layers as shown in Figure 6 Each LGBlock convolution module is connected using a residual connection. The LGNet network draws on the successful experience of the Transformer, replacing the batch normalization used in the traditional ResNet network with layer normalization, replacing the ReLU activation function with the GELU activation function, and only retaining the normalization after the 1-dimensional deep convolution layer and the activation function after the first fully connected layer, reducing the use of normalization and activation functions. These changes can all improve the expression ability of the network. In addition, since the LGNet network replaces the ordinary convolution with a deep convolution, adding the LGNet network in the conversion model will reduce the computational complexity compared to adding the ResNet network, thereby reducing the training time of the conversion model. Specifically, the number of channels of each convolution kernel of the deep convolution is equal to 1, and each convolution kernel is only responsible for one channel of the input feature, so the number of convolution kernels is equal to the number of channels of the input feature, so that the number of channels of the output feature is equal to the number of channels of the input feature. Each convolution kernel of the ordinary convolution will be convolved with each dimension of the input feature, so the number of channels of each convolution kernel is equal to the number of channels of the input feature, and the number of channels of the output feature is equal to the number of convolution kernels. A specific analysis of the computational complexity of the two types of convolution is as follows: assuming D f is the width and height of the input feature, D k is the height and width of the convolution kernel, M is the number of channels of the input feature, and N is the number of channels of the output feature. The computational complexity of convolution is approximately equal to the height of the convolution kernel x the width of the convolution kernel x the number of channels of the convolution kernel x the number of convolution kernels x the height of the input feature x the width of the input feature (assuming the step size is equal to 1). The computational complexity of the ordinary convolution is:

[0080] D k x D k x M x N x D f x D f

[0081] The computational complexity of the deep convolution is:

[0082] D k x D k x 1 x M x D f x D f

[0083] Therefore, the computational complexity of the deep convolution is 1 / N of the ordinary convolution.

[0084] (3) In the decoder D, the adaptive instance normalization layer fully fuses the LGNet network optimized content representation LG(C) and the speaker information representation S, to provide the speaker information for the reconstructed acoustic feature X', and further obtain the reconstructed acoustic feature X' highly similar to the input acoustic feature X, as shown in the structure of Figure 7 The decoder D includes 1 one-dimensional convolution layer, 6 adaptive instance normalization layers, 6 decoding modules, 1 gated recurrent unit layer, and 1 fully connected layer. The decoding module is composed of 2 one-dimensional convolution layers, 1 batch normalization layer, and 1 LeakyReLU activation function shown in Figure 8 The adaptive instance normalization layer fuses the speaker information representation S into the optimized content representation LG(C), so that the converted sentence is closer to the target speaker in speaker similarity. The output of the adaptive instance normalization layer is connected to the decoding module, and the fused feature is decoded. Figure 8 In the structure shown in FIG. 6B, B represents the adaptive instance normalization layer and the decoding module as a whole, and 6 groups are placed in turn (the adaptive instance normalization layer and the decoding module are placed alternately). The gated recurrent unit layer can capture the semantic correlation between the frame sequence, and output the converted spectral feature through the fully connected layer, so as to optimize the reconstructed acoustic feature X'.

[0085] The formula of the adaptive instance normalization layer is:

[0086] AdaIN(σ(Z'), μ(Z'), Y) = σ(Z') IN(Y) + μ(Z')

[0087] Where Y represents the feature vector obtained after the first one-dimensional convolution layer in the decoder, as the input of the adaptive instance normalization layer. μ(Z') and σ(Z') represent the speaker information representation S extracted by the encoder E. IN(Y) represents the instance normalization of Y.

[0088] The objective function of the entire conversion model is represented as:

[0089]

[0090] Where θ E represents the parameters corresponding to the encoder E, θ D represents the parameters corresponding to the decoder D, L rec represents the reconstruction loss function of the model, λ rec is a hyperparameter representing the weight of the reconstruction loss function.

[0091] The reconstruction loss function is represented as:

[0092] L rec (θ E ,θ D )=EX～p(X) [||X'-X||1]

[0093] wherein X represents the input acoustic feature of the encoder E, X' represents the reconstructed acoustic feature of the decoder D, E X～p(X) [·] represents the loss expectation between the reconstructed acoustic feature X' of the decoder D and the input acoustic feature X of the encoder E, ||·||1 represents the L1 norm.

[0094] Step S4: In the process of training the conversion model, the difference between the reconstructed acoustic feature X' and the input acoustic feature X is made smaller and smaller through continuous iterative training, so that the reconstruction loss function of the conversion model is as small as possible. The hyperparameters of the conversion model are set to minimize the objective function until the set number of iterations, so as to obtain the trained conversion model. Due to the difference in specific settings of neural networks and the difference in performance of experimental equipment, the number of iterations set is also different. In this experiment, the number of iterations is set to 200000 times;

[0095] Step S4-1: input the acoustic feature X extracted from the training corpus into the encoder E to train the ability of the encoder E to extract the content representation C and the speaker information representation S;

[0096] Step S4-2: in the encoder E, the acoustic feature X obtains a feature vector Z after passing through the first one-dimensional convolutional layer in the encoder E, and then obtains a feature vector Z' after passing through the enhanced encoding module;

[0097] Step S4-3: the instance normalization layer takes the feature vector Z' as input to extract the speaker information representation S={μ(Z'), σ(Z')}, and passes the speaker information representation S to the decoder D through the U-shaped connection; and normalizes Z' using the extracted speaker information to obtain an instance normalized representation, which is input into the last one-dimensional convolutional layer in the encoder E to obtain a feature vector Z";

[0098] Step S4-4: use the activation guide function layer as a bottleneck to further filter out the speaker information in the feature vector Z" to obtain a pure content representation C;

[0099] Step S4-5: input the pure content representation C into the LGNet network to further extract and optimize, to obtain an optimized representation LG(C) that fully expresses the information in the content representation;

[0100] Step S4-6: input the optimized content representation LG(C) and the speaker information representation S into the decoder D to train the ability of the decoder D to reconstruct the acoustic feature X';

[0101] Step S4-7: input the feature vector Y obtained after the content representation LG(C) is passed through the first one-dimensional convolutional layer in the decoder and the speaker representation S extracted by the encoder E into the adaptive instance normalization layer to enhance the speaker information in the reconstructed acoustic feature X';

[0102] Step S4-8: jointly train the encoder E, the LGNet network and the decoder D, repeat the process of steps S4-1 to S4-7, minimize the reconstruction loss function until the set number of iterations is reached, so as to obtain the trained conversion model.

[0103] As shown in Figure 2 , the conversion stage includes the following steps:

[0104] Step S5: obtain the to-be-converted corpus, which is from the AISHELL-3 Chinese corpus and the CSTR VCTK English corpus; from the obtained Chinese and English corpora, 8 speakers in the AISHELL-3 Chinese corpus are randomly selected, each speaker has about 200 sentences, and 10 sentences are randomly selected from the 200 sentences, each with a duration of 2-10s; 8 speakers in the CSTR VCTK English corpus are selected, each speaker has about 200 sentences, and 10 sentences are randomly selected from the 200 sentences, each with a duration of 2-8s. The selected sentences in Chinese and English are jointly composed into the to-be-converted corpus, and the log-mel spectrum features of the source sentences and the target sentences in the to-be-converted corpus are extracted to represent the acoustic features X1 and X2 respectively;

[0105] Step S6: input the acoustic features X1 and X2 into the trained conversion model to reconstruct the log-mel spectrum features D(LG(C1), S2) of the converted sentence, which is represented as the acoustic feature X 1→2 ;

[0106] Step S6-1: input the acoustic features X1 and X2 extracted in step S5 into the trained conversion model, and the encoder E of the conversion model extracts the content representation C1 of the source sentence and the speaker information representation S2 of the target sentence respectively;

[0107] Step S6-2: input the content representation C1 of the source sentence obtained in step S6-1 into the LGNet network of the conversion model for further extraction and optimization to obtain the optimized content representation LG(C1) of the source sentence, and fully express the information in the content representation; the speaker information representation S2 of the target sentence is directly transmitted to the decoder D through the U-shaped connection;

[0108] Step S6-3: input the optimized content representation LG(C1) of the source sentence and the speaker information representation S2 of the target sentence into the decoder D to reconstruct the log-mel-spectral feature D(LG(C1), S2) of the converted sentence, which represents the acoustic feature X 1→2 ;

[0109] Step S7: use the Griffin-Lim vocoder to convert the acoustic feature X of the converted sentence obtained in step S6-3 into a speech waveform for output, thereby obtaining the synthesized speech of the cross-lingual speech conversion. 1→2

[0110] The above describes exemplary embodiments of the present application, which do not limit the patent protection scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, which is made by using the content of the specification and drawings, is also included in the patent protection scope of the present application.​

Claims

1. A cross-lingual speech conversion method of a codec structure fusing an enhancement coding module and an LGNet network, characterized by, The method comprises a training phase and a conversion phase, The training phase comprises the following steps: Step S1: obtaining a large amount of training corpus, the training corpus being composed of sentences of multiple speakers in different languages; Step S2: preprocessing and feature extraction are performed on the acquired training corpus; 128 frames are randomly selected from the log-mel spectrum features of each training corpus and are spliced to obtain a log-mel spectrum feature segment, which is represented as an acoustic feature ; Step S3: The acoustic features The input is fed into a transformation model for model training, the transformation model including an encoder. LGNet network and decoder The encoder acoustic features of the input De-entanglement is performed to obtain speaker information representation. and content representation The encoder and decoder The speakers are represented using a U-shaped connection. From encoder Passed to decoder The LGNet network supports the encoder. Extracted content representation Further optimization yielded This allows the information in the content representation to be fully expressed; the decoder Represent the acquired speaker information and optimized content representation Reconstruction is performed to obtain the reconstructed log-Mel spectrum features. Characterized by acoustic features ; The encoder in the conversion model consisting of 2 one-dimensional convolutional layers, 6 enhanced encoding modules, 6 instance normalization layers and 1 activation guide function layer, the enhanced encoding module consists of 1 recurrent gate convolutional layer, 1 batch normalization layer, 1 LeakyReLU activation function and 1 one-dimensional convolutional layer, the enhanced encoding module and the instance normalization layer are placed alternately; Step S4: in the process of training the conversion model, the reconstruction loss function of the conversion model is made as small as possible, the hyperparameters of the conversion model are set to minimize the objective function until the set number of iterations, so as to obtain the trained conversion model; The conversion phase comprises the following steps: Step S5: Selecting sentences in different languages as the to-be-converted corpus, selecting source sentences and target sentences in different languages from the to-be-converted corpus respectively, extracting log-mel spectrum features of the source sentences and log-mel spectrum features of the target sentences respectively, and representing the log-mel spectrum features as acoustic features respectively and ; Step S6: inputting the acoustic feature and into the trained conversion model to reconstruct the log-mel spectrum feature of the converted sentence, represented as acoustic feature ; Step S7: The obtained acoustic features of the converted sentence are converted into a speech waveform using a Griffin-Lim vocoder, and output, thereby obtaining a synthesized speech of cross-lingual speech conversion. converted into a speech waveform using a Griffin-Lim vocoder, and output, thereby obtaining a synthesized speech of cross-lingual speech conversion.

2. The cross-lingual speech conversion method of claim 1, wherein the encoder-decoder structure fuses the enhanced coding module and the LGNet network. The pre-processing and feature extraction of the obtained training corpus in step S2 are specifically as follows: the sampling rate of the training corpus is set to 24 kHz, the number of points of fast Fourier transform is set to 2048, the window length is set to 1200, and the window moving length is set to 300.

3. The cross-lingual speech conversion method of claim 1, wherein the encoder-decoder structure fuses the enhanced coding module and the LGNet network. The recursive gated convolution in the enhanced encoding module extends the second-order interaction in self-attention to an arbitrary order, and the encoder The acoustic features The feature vector obtained after the first one-dimensional convolution layer in the encoder The feature vector obtained after the first one-dimensional convolution layer in the encoder The input linear projection layer input into the recursive gated convolution as the input of the recursive gated convolution in the enhanced encoding module A set of projected features And The formula of the input linear projection layer is: ； wherein, ; and denote the height and width of the feature vector ; denote the number of channels; denote the order of high-order interaction; denote the number of channels corresponding to different projection features; after obtaining the projection features, they are sequentially input into the recursive gate convolution for recursive operation, and the operation formula is: ， ； wherein, denotes a scaling factor; denotes a depthwise convolution computation; denotes a dimension mapping function, denotes a dot product operation; the output of the last step of the operation is fed to an output linear projection layer which results in the final result of the recurrent gated convolution.

4. The cross-lingual speech conversion method of claim 1, wherein the fusion of the enhanced coding module and the codec structure of the LGNet network is characterized by, In step S3, the LGNet network in the conversion model is composed of 6 LGBlock convolution modules, each LGBlock convolution module is composed of 1 one-dimensional deep convolution layer, 1 layer normalization layer, 1 GELU activation function, 1 regularization layer and 2 fully connected layers; each LGBlock convolution module is connected by using residual connection.

5. The cross-lingual speech conversion method of claim 4, wherein, The number of channels of each convolution kernel of the one-dimensional deep convolution is equal to 1, each convolution kernel is only responsible for one channel of the input feature, therefore the number of convolution kernels is equal to the number of channels of the input feature, so that the number of channels of the output feature is equal to the number of channels of the input feature; Assume is the height and width of the input feature, is the height and width of the convolution kernel, is the number of channels of the input feature; the computation of the convolution is equal to the height of the convolution kernel the width of the convolution kernel the number of channels of the convolution kernel the number of convolution kernels the height of the input feature the width of the input feature, assuming the stride is equal to 1; The calculation amount of the one-dimensional deep convolution is: 。 6. The cross-lingual speech conversion method of claim 1, wherein the encoder-decoder structure fuses the enhanced coding module and the LGNet network. The training process in step 4 is specifically as follows: Step S4-1 : extracting acoustic features from the training corpus input to the encoder training the encoder extracting content representations and speaker information representations the ability to Step S4-2: In the encoder acoustic features In the encoder After the first one-dimensional convolutional layer, the feature vector After the enhanced encoding module, the feature vector ; Step S4-3: encoder The first instance normalization layer normalizes the feature vector The speaker information representation is extracted as input The speaker information representation is delivered to the decoder through the U-shaped connection ; and the extracted speaker information is normalized to obtain an instance normalized representation , which is input into the last one-dimensional convolutional layer in the encoder to obtain the feature vector ; Step S4-4: further filtering the feature vector using an activation guide function layer as a bottleneck to get pure content representation with speaker information, resulting in a pure content representation ; Step S4-5: Characterize the pure content The input to the LGNet network is further optimized to obtain an optimized content characterization The information in the content characterization is fully expressed. Step S4-6: Optimizing the content representation and speaker information representation to the decoder , training the decoder to reconstruct the acoustic features with the capability; Step S4-7: Characterizing the content Feature vector resulting after the first one-dimensional convolutional layer in the decoder and the encoder Extracted speaker representation Adaptive instance normalization layer input into the decoder to enhance the reconstructed acoustic features in terms of speaker information; Step S4-8: Encoder , LGNet network and decoder Joint training, repeat the process of step S4-1 to step S4-7 by minimizing the reconstruction loss function until a set number of iterations is reached, thereby obtaining a trained conversion model.

7. The cross-lingual speech conversion method of claim 1, wherein the encoder-decoder structure fuses the enhanced coding module and the LGNet network. The reconstruction loss function is expressed as: ； wherein, represents the input acoustic feature of the encoder ; represents the reconstructed acoustic feature of the decoder , represents the reconstructed acoustic feature of the decoder ; represents the loss expectation between the input acoustic feature of the encoder and the reconstructed acoustic feature of the decoder ; represents the L1 norm; represents the corresponding parameter of the encoder ; represents the corresponding parameter of the decoder .

8. The cross-lingual speech conversion method of claim 1, wherein the encoder-decoder structure fuses the enhanced coding module and the LGNet network. The objective function is expressed as: ； wherein, represents an encoder corresponding parameters; represents a decoder corresponding parameters, represents a reconstruction loss function of the model, the hyperparameter represents a weight of the reconstruction loss function.

9. The cross-lingual speech conversion method of claim 1, wherein the encoder-decoder structure fuses the enhanced coding module and the LGNet network. The conversion process in step S6 is specifically as follows: Step S6-1: input the acoustic feature described in step S5 into the trained conversion model, extract the content representation of the source sentence and the speaker information representation of the target sentence using the encoder and ;​​​ Step S6-2: content representation of the source sentence Further extraction and optimization through the LGNet network to obtain the optimized content representation So that the information in the content representation is fully expressed; speaker information representation of the target sentence Directly passed to the decoder through the U-shaped connection ; Step S6-3: decoder characterizing the content of the optimized source sentence and speaker information of the target sentence fully fused in the adaptive instance normalization layer in the decoder, then decoded by the decoding module, and finally the reconstructed converted log-mel spectrum features are obtained characterized as acoustic features .

Images