Singing voice conversion method based on cbam and dynamic convolution decomposition

Abstract

The present application belongs to the technical field of speech conversion, and in particular relates to a singing speech conversion method based on CBAM and dynamic convolution decomposition, which comprises a training stage and a conversion stage, and a model network comprises a generator, a discriminator and a style encoder. First, dynamic convolution decomposition is introduced in the generator, dynamic channel fusion is used to replace the dynamic attention of the channel group, the problem that dynamic convolution in the generator causes the number of convolution weights to increase by K times is solved, the difficulty of joint optimization is reduced, and the model requires fewer parameters without sacrificing accuracy, thereby improving the running performance of the algorithm. Further, CBAM attention modules are introduced in the encoding network and the decoding network of the generator, attention is applied to the channels and the space, the attention and capture of the detailed information in the spectrum are improved, and the quality of the converted singing speech is significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of speech conversion technology, specifically, it is a singing speech conversion method based on CBAM and dynamic convolution decomposition. Background Technology

[0002] Voice conversion (VC) technology is defined as changing the identity characteristics of a source speaker to those of a target speaker while maintaining the content characteristics of the source speaker. The key to voice conversion technology is to effectively change the speaker's timbre while ensuring speech quality.

[0003] Singing speech conversion is a new research direction in speech conversion, transforming the voice of a source singer into that of a target singer while preserving the singing content. Through singing speech conversion, everyone can sing like a professional, overcoming professional limitations, freely controlling the pitch of their voice, and expressing emotions in different ways. The analysis of singing speech focuses more on sustained notes, harmonic / rhythmic structure, and relative pitch—musical values ​​that are absent in ordinary speech. Furthermore, singing speech emphasizes non-periodic aspects such as consonant utterances and rapidly moving spectral envelopes, requiring the processing of a wider range of frequency variations, as well as sharper changes in volume and pitch within singing speech.

[0004] Many methods have emerged in the field of singing speech conversion, such as the statistical singing speech conversion based on direct waveform correction using global variance proposed by Kazuhiro Kobayashi et al. in 2015, and the direct waveform correction singing speech conversion based on spectral difference; or the cascaded vocal synthesis method proposed by Fernando Villavicencio et al. in 2016. They usually require parallel data to train the conversion model.

[0005] To overcome the limitations of relying on parallel training data for singing speech conversion, Facebook AI proposed an unsupervised method using non-parallel data in 2019. This method employs an autoencoder architecture consisting of a WaveNet-like encoder, a WaveNet autoregressive decoder, and a learnable singer embedding vector table. By switching the embedding vectors of different singers, singing speech conversion can be achieved.

[0006] Although the above method can produce singing voice that is perceptually similar to the target singer, the quality of the generated singing is often affected due to the difficulty of learning the joint representation of voice and pitch. In 2020, Deng et al. in Tencent AI Lab proposed a new unsupervised singing voice conversion method, PitchNet, which uses a pitch regression network to separate pitch-related information from the latent space of the encoder, generating a pitch-invariant representation, enabling flexible pitch processing.

[0007] In recent years, GAN-based methods have been widely used in voice conversion and singing voice conversion. They can quickly generate high-quality audio through a small parameter model, which facilitates end-to-end training. However, directly using GAN for singing voice conversion can cause problems such as pitch jitter and high pitch error, which will mainly manifest in the harmony component, which is very important in singing voice and will directly affect the quality of the generated singing voice, thus reducing the auditory experience of the entire song.

[0008] Thanks to the vigorous development of the voice conversion field, the above-mentioned methods have to some extent improved the quality and individual similarity of converted singing voice, but there is still a lack of attention to the details of singing voice, and the processing of pitch, fundamental frequency, and wide frequency is not good enough. Currently, there are still few effective methods in this regard. On the other hand, while improving the quality of converted singing voice, it is also a pressing problem to maintain as low a joint optimization difficulty and training cost as possible. SUMMARY

[0009] To solve the above problems, the present application proposes a singing voice conversion method based on CBAM and dynamic convolution decomposition. This method can improve the network's ability to process singing voice details by applying attention to space and channels, thereby improving the quality of converted singing voice. On the other hand, by re-examining dynamic convolution from the perspective of matrix decomposition and through dynamic channel fusion, the dimensionality of the latent space is reduced, significantly reducing the model's parameter quantity and computational load, significantly reducing the difficulty of joint optimization of dynamic attention and static convolution kernels, and improving the model's running efficiency.

[0010] The specific technical solutions adopted by the present application are as follows:

[0011] A singing voice conversion method based on CBAM and dynamic convolution decomposition includes a training phase and a conversion phase, wherein the training phase includes the following steps:

[0012] (1.1) Obtain training corpus, which consists of singing voices of multiple singers;

[0013] (1.2) Extract the source singing voice spectrum feature x using the WORLDAudio Analysis / Synthesis Models , fundamental frequency feature f 0s and aperiodic feature;

[0014] (1.3) extracting target singing voice style feature s t by style encoder S; t . Generator G includes an encoding network and a decoding network, source singing voice content feature x c is generated by the encoding network, and then source singing voice content feature x c is reconstructed by the decoding network with target singing voice style feature s t , so as to restore singing voice;

[0015] (1.4) continuously adjusting the hyperparameters of the conversion network to minimize the loss function until the ideal number of iterations, so as to obtain the trained conversion network;

[0016] (1.5) constructing a fundamental frequency conversion function from source singing voice fundamental frequency feature f 0s to target singing voice fundamental frequency feature f 0t ;

[0017] The conversion stage includes the following steps:

[0018] (2.1) using the WORLD speech analysis / synthesis model to extract source singing voice spectrum feature x s ′, fundamental frequency feature f 0s ′ and aperiodic feature from the converted corpus;

[0019] (2.2) extracting target singing voice style feature s t ′ by style encoder S, inputting the above-mentioned source singing voice spectrum feature x s ′ and target singing voice style feature s t ′ into the trained conversion network in step (1.4) to reconstruct target singing voice spectrum feature x st ′;

[0020] (2.3) converting the source singing voice fundamental frequency feature f 0s ′ extracted in step (2.1) to target singing voice fundamental frequency feature f 0t ′ by the fundamental frequency conversion function in step (1.5);

[0021] (2.4) synthesizing the aperiodic feature in step (2.1), the reconstructed target singing voice spectrum feature x st ′ obtained in step (2.2) and the target singing voice fundamental frequency feature f 0t ′ obtained in step (2.3) using the WORLD speech analysis / synthesis model to obtain the converted singing voice.

[0022] Further improvement of the present application, the above style encoder S is composed of 5 layers of one-dimensional modules, including one-dimensional convolution module and one-dimensional pooling module, wherein each layer of convolution module is composed of convolution layer and ReLU activation function, each layer of pooling module is composed of average pooling, and the output layer is composed of full connection layer.

[0023] Further improvement of the present application, the training process of (1.3) above comprises the following steps:

[0024] Step 1, the source singing voice spectrum feature x s The encoding network of the input generator G is obtained, and the identity-independent content feature G(x s ) is obtained.

[0025] Step 2, the target singing voice style feature s t is extracted by the style encoder S.

[0026] Step 3, the generated content feature G(x s ) and the target singing voice style feature s t are input into the decoding network of the generator G for training, and the loss function of the generator G is minimized in the training process, so as to obtain the converted target singing voice spectrum feature x st .

[0027] Step 4, the source singing voice spectrum feature x s is input into the style encoder S to obtain the source singing voice style feature s s .

[0028] Step 5, the generated converted target singing voice spectrum feature x st is input into the encoding network of the generator G again to obtain the identity-independent content feature G(x st ).

[0029] Step 6, the generated content feature G(x st ) and the source singing voice style feature s s are input into the decoding network of the generator G for training, and the loss function of the generator G is minimized in the training process, so as to obtain the reconstructed source singing voice spectrum feature

[0030] Step 7, the converted target singing voice spectrum feature x st generated in step 3 is input into the discriminator D for training, and the loss function of the discriminator D is minimized.

[0031] Step 8, the converted target singing voice spectrum feature x st generated in step 3 is input into the style encoder S for training, and the style loss function of the style encoder S is minimized.

[0032] Step 9, return to step 1 and repeat the above steps until a desired number of iterations is reached, thereby obtaining a trained conversion network.

[0033] The singing voice conversion method based on CBAM and dynamic convolution decomposition proposed in the application is characterized in that a generating function of a kernel of the dynamic convolution is expressed as:

[0034]

[0035] where W k is a static convolution kernel, K is the number of static convolution kernels, π k (x) is an attention score, and

[0036]

[0037] The singing voice conversion method based on CBAM and dynamic convolution decomposition proposed in the application is characterized in that a generating function of a kernel of the dynamic convolution decomposition is expressed as:

[0038] W(x) = Λ(x)W0 + Pφ(x)Q T

[0039]

[0040] where Λ(x) is a CxC diagonal matrix, in this way, dynamic channel attention is realized on W0; the dimension of the P matrix is CxL, Q T is LxC, and L<<C, Pφ(x)Q T represents a sparse dynamic residual, which is a diagonal matrix; φ(x) is a full dynamic matrix, used to realize dynamic channel fusion; W0 is a weighted average of K static convolution kernels.

[0041] The singing voice conversion method based on CBAM and dynamic convolution decomposition proposed in the application is characterized in that the dynamic convolution decomposition uses a lightweight dynamic branch to generate coefficients of dynamic channel fusion φ(x) and dynamic channel attention Λ(x). The dynamic branch first applies average pooling to the input vector x, then makes the vector x pass through two fully connected layers and an activation layer in turn, and finally generates the convolution kernel W(x) by weighting. Similar to the static convolution, the dynamic convolution decomposition layer also includes a batch normalization layer and an activation layer.

[0042] The singing voice conversion method based on CBAM and dynamic convolution decomposition proposed in the application is characterized in that the CBAM attention module includes two sub-modules, the first of which is a channel attention module, and the input feature map F is specifically expressed as:

[0043]

[0044]

[0045] M c (F) represents the generated channel attention, sigma represents the activation function Sigmoid, MLP represents a multi-layer perceptron, AvgPool represents global average pooling, MaxPool represents global maximum pooling, W0 and W1 represent weight matrices of the MLP. and respectively represent a global average pooling feature map and a global maximum pooling feature map.

[0046]

[0047] F' represents a feature map output by the channel attention module, which is obtained by multiplying the input feature map F and the generated channel attention M c (F).

[0048] The singing voice conversion method based on CBAM and dynamic convolution decomposition provided in the present application takes the feature map F' output by the channel attention module as the input feature map of the spatial attention module, and the specific function is:

[0049]

[0050] M s (F') represents the generated spatial attention, sigma represents the activation function Sigmoid, f 7×7 represents a 7x7 convolution operation, and respectively represent a global average pooling feature map and a global maximum pooling feature map.

[0051]

[0052] F'' represents a final feature map, which is obtained by multiplying the input feature map F' and the generated spatial attention M s (F').

[0053] The application has the beneficial effects that: compared with the prior art, the application firstly introduces dynamic convolution decomposition in the generator, replaces dynamic attention on the channel group by dynamic channel fusion, solves the problem that dynamic convolution in the generator can cause the number of convolution weights to increase K times, reduces the difficulty of joint optimization, makes the model need fewer parameters without sacrificing accuracy, and can reduce the training time without reducing the quality of singing voice, and improves the operation efficiency of the whole model; the application further introduces the CBAM attention module for wider frequency changes of singing voice, the input feature map will pass through the channel attention module and the spatial attention module in turn, increases the attention to the details in the spectrum through the pooling operation of the shared neuron network, identifies less significant features, effectively improves the representation ability of the network, and thus improves the quality of singing voice. At the same time, since the model can adapt to wide frequency changes, it can convert target non-singing voice, i.e., convert ordinary speaking voice into high-quality singing voice. Therefore, the application is a singing voice conversion method with high synthesized voice quality and individual similarity. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a principle schematic diagram of the model of the application.

[0055] Figure 2 is a network structure diagram of the generator in the model described in the embodiment of the application.

[0056] Figure 3 is a network structure diagram of dynamic convolution decomposition in the model described in the embodiment of the application.

[0057] Figure 4 is a principle schematic diagram of the CBAM attention module in the model described in the embodiment of the application.

[0058] Figure 5 is a principle schematic diagram of the channel attention module in the model described in the embodiment of the application.

[0059] Figure 6 is a principle schematic diagram of the spatial attention module in the model described in the embodiment of the application. DETAILED DESCRIPTION

[0060] In order to deepen the understanding of the application, the application will be further described in detail below in combination with the drawings and embodiments, which are only used to explain the application and do not limit the protection scope of the application.

[0061] Embodiment: as Figure 1As shown, a singing voice conversion method based on CBAM and dynamic convolution decomposition includes a training stage and a conversion stage. The training stage is used to obtain the parameters and conversion functions required for singing voice conversion, and the conversion stage is used to realize the conversion of the source singing voice into the target singing voice.

[0062] Further explanation, the training stage includes the following steps:

[0063] Step 1, obtain the training corpus of non-parallel text, which is composed of singing corpus of multiple singers. In the experiment, the non-parallel corpus dataset in the singing and spoken corpus library NUS-48E developed by the Sound and Music Computing Laboratory of the National University of Singapore is selected. The corpus library is a 169-minute set, with 48 English song lyrics sung and read by 12 singers, and a complete set of transcriptions and duration annotations. Each singer corresponds to a folder, and each folder includes 4 singing and reading wav format files.

[0064] In this experiment, all singing recordings of 8 singers in the corpus library are selected and renamed as SF3, SF4, SM3, SM4, TF1, TF2, TM1, and TM2, where S represents the source singer, T represents the target singer, F represents the female singer, and M represents the male singer. The singing recordings of each singer are processed by cutting, and each singer obtains 80 recordings after processing, each recording lasting 3-4 seconds, and the sampling rate is set to 24 kHz. Among them, 65 are used for training tasks, and 15 are used for testing tasks. In the conversion stage, the above 8 singers will form 16 groups of source-target conversion situations, respectively SF3-TF1, SF3-TF2, SF3-TM1, SF3-TM2, SF4-TF1, SF4-TF2, SF4-TM1, SF4-TM2, SM3-TF1, SM3-TF2, SM3-TM1, SM3-TM2, SM4-TF1, SM4-TF2, SM4-TM1, and SM4-TM2.

[0065] Step 2, use the WOLD speech analysis / synthesis model to extract the source singing voice spectrum features x s , fundamental frequency features f 0s , and aperiodicity features from the training corpus. The fast Fourier transform parameter is set to 1024, the time frame length is set to 256, and the mel cepstrum dimension is set to 36. Therefore, the extracted mel cepstrum dimension is (36, 256), the fundamental frequency feature dimension is (1, 256), and the aperiodicity feature dimension is (1, 256). The batch size is set to 8 during training.

[0066] Step 3, the singing voice conversion method based on CBAM and dynamic convolution decomposition in this embodiment consists of three parts: a generator G that generates real spectrum, a discriminator D that judges whether the input spectrum is a real spectrum feature or a generated spectrum feature, and a style encoder S for extracting target style features.

[0067] The objective function L of the network is:

[0068] L = L G + L D + L s

[0069] Wherein, L G , L D and L s are the loss functions of the generator G, the discriminator D and the style encoder S respectively.

[0070] The loss function L G of the generator is expressed as:

[0071]

[0072] Wherein, λ cyc and λ sty are a set of regularization hyperparameters, representing the weights of the cycle consistency loss and the style reconstruction loss respectively, and represent the adversarial loss, the cycle consistency loss of the generator, and the style reconstruction loss of the style encoder respectively;

[0073] The loss function L D of the discriminator is:

[0074]

[0075] Wherein, is the adversarial loss of the discriminator.

[0076] Step 4, using the WORLd voice analysis / synthesis model to extract the target singing voice spectrum feature x t from the training corpus, and input it into the style encoder S to obtain the target singing voice style feature s t ; The style encoder consists of 5 one-dimensional convolution modules and 5 one-dimensional pooling modules, wherein each one-dimensional convolution module includes convolution and ReLU activation function, each one-dimensional pooling module is composed of average pooling, and the output layer is composed of a fully connected layer.

[0077] Step 5, input the extracted source singing voice spectrum feature x s and the target singing voice style feature s tThe generator is trained together with the input, so that the loss function L of the generator G is as small as possible, and the converted target singing voice spectrum feature x st is obtained.

[0078] As shown in Figure 2 , the generator adopts a two-dimensional convolutional network, and the activation function uses a Mish function. The generator is composed of an encoding network and a decoding network. The encoding network is composed of 7 layers of modules. The first 3 layers are two-dimensional convolutional modules, each of which includes two-dimensional convolution, instance normalization, and a Mish function. The last 4 layers are CBAM attention modules, each of which includes dynamic convolution decomposition, instance normalization, a Mish function, a channel attention module, and a spatial attention module. The decoding network is composed of 6 layers of modules. The first 4 layers are CBAM attention modules, each of which includes dynamic convolution decomposition, instance normalization, a Mish function, a channel attention module, and a spatial attention module. The last 2 layers are two-dimensional transposed convolutional modules, each of which includes transposed dynamic convolution, adaptive instance normalization, and a Mish function. The specific network structure diagram of the dynamic convolution decomposition is shown in Figure 3 , the principle diagram of the CBAM attention module is shown in Figure 4 , Figure 5 , and Figure 6 are the channel attention module and the spatial attention module, respectively.

[0079] Step 6, input the target singing voice spectrum feature x t and the converted target singing voice spectrum feature x st obtained in step 5 into the discriminator, train the discriminator, and make the adversarial loss function L of the discriminator as small as possible. The discriminator is composed of 5 two-dimensional convolutional modules and an output layer. Each two-dimensional convolutional module includes two-dimensional convolution and a LeakyReLU function. The output layer of the discriminator is set to have 1 convolution channel.

[0080] The loss function of the discriminator is:

[0081]

[0082] wherein, is the adversarial loss of the discriminator.

[0083]

[0084] wherein, D(x s ) represents that the discriminator D discriminates the real spectrum feature, s t represents the target singing voice style feature generated by the style encoder S, i.e., S(x t )=s t , and G(xs , s t ) represents the converted target singing voice spectrum feature generated by the generator G, D(G(x s , s t )) represents the generated spectrum feature discriminated by the discriminator, represents the expectation of the probability distribution generated by the generator G, represents the expectation of the real probability distribution.

[0085] The optimization objective is:

[0086]

[0087] Step 7, input the converted target singing voice spectrum feature x st obtained in step 5 into the encoding network of the generator G again to obtain the identity-independent content feature G(x st ), and input the source singing voice spectrum feature x s into the style encoder S to obtain the source singing voice style feature s s , input the obtained content feature G(x st ) and the source singing voice style feature s s into the decoding network of the generator G for training, and minimize the loss function of the generator G in the training process to obtain the reconstructed source singing voice spectrum feature

[0088] The loss function of the generator is minimized in the training process, including the adversarial loss of the generator, the cycle consistency loss, and the style reconstruction loss of the style encoder. Among them, the training cycle consistency loss is to make the reconstructed source singing voice spectrum feature s after passing through the generator G as consistent as possible with x s , and the training style reconstruction loss is to constrain the style encoder to generate a style feature s t more consistent with the target singing voice.

[0089] The loss function of the generator is:

[0090]

[0091] The optimization objective is:

[0092]

[0093] Among them, λ cyc and λ sty are a set of regularization hyperparameters, respectively representing the weights of the cycle consistency loss and the style reconstruction loss.

[0094] ​ The adversarial loss of the generator G:

[0095]

[0096] where, is the expectation of the probability distribution generated by the generator, s t is the target singing voice style feature generated by the style encoder, i.e., S(x t ) = s t , G(x s , s t ) represents the converted target singing voice spectrum feature generated by the generator, and D(G(x s , s t )) represents the discrimination of the discriminator on the real target spectrum feature, which is used to distinguish whether the spectrum input to the discriminator is a real spectrum or a generated spectrum. In the training process is as small as possible, and the generator is continuously optimized until the generated spectrum feature G(x s , s s ) is so realistic that the discriminator cannot distinguish between real and fake.

[0097] The cycle-consistency loss in the generator G:

[0098]

[0099] where, s s represents the source singing voice style feature, i.e., S(x s ) = s s , and G(G(x s , s t ), s s ) is the reconstructed source singing voice spectrum feature generated by the generator, is the loss expectation of the reconstructed source singing voice spectrum and the real source singing voice spectrum, and ||·||1 represents the 1-norm. In the loss of training the generator, is as small as possible, so that the generated target singing voice spectrum feature G(x s , s t ) and the source singing voice style feature s s , when input to the generator again, the reconstructed source singing voice spectrum feature obtained is as similar as possible to x s . Through training , the content features of the singing voice can be effectively ensured not to be lost after being encoded by the generator.

[0100] The style reconstruction loss of the style encoder S is used to optimize the style feature s t :

[0101]

[0102] Among them, s t The style features of the target singing voice generated by the style encoder S are represented as S(x). t ) = s t , G(x s s t ) represents the spectral features of the transformed target singing speech generated by the generator, ||·||1 represents the 1-norm, S(G(x) s s t )) represents the transformed singing speech style features generated by the style encoder S.

[0103] The target singing speech spectral features G(x) will be transformed. s s t The reconstructed style features S(G(x)) are input into the style encoder S to obtain the style features S(G(x)). s s t ), and the target singing voice style features generated by the style encoder. t Find the absolute value during training. The smaller the value of the target singing speech style features s generated by the style encoder S, the better. t It can be fully expressed.

[0104] Step 8: Repeat steps 4 through 7 until the desired number of iterations is reached, thus obtaining the trained network. Due to differences in specific neural network settings and experimental equipment performance, the number of iterations will vary. In this experiment, the number of iterations is set to 250,000.

[0105] Step 9: Construct the fundamental frequency features f of the source singing speech 0s To the fundamental frequency characteristics of the target singing voice f 0t The fundamental frequency conversion function establishes a conversion relationship between the mean and the root mean square error.

[0106] To further explain, the fundamental frequency conversion function is:

[0107]

[0108] Where, μ s and σ s Let μ be the mean and root mean square error of the fundamental frequency features of the source singing speech in the logarithmic domain. t and σ t Let f be the mean and standard deviation of the fundamental frequency features of the target singing speech in the logarithmic domain. 0s For the logarithmic domain fundamental frequency feature of the source singing speech, log f 0t The target singing speech is the logarithmic domain fundamental frequency feature for conversion.

[0109] In the present embodiment, the conversion stage comprises the following steps:

[0110] Step 1, extracting the source singing voice spectral feature x from the to-be-converted corpus by the WORLD voice analysis / synthesis model s ′, the fundamental frequency feature f 0s ′, and the non-periodicity feature;

[0111] Step 2, extracting the target singing voice style feature s by the style encoder S t ′, inputting the source singing voice spectral feature x s ′ and the target singing voice style feature s t ′ in the to-be-converted corpus into the network trained in step 8 of the training stage to reconstruct the target singing voice spectral feature x st ′;

[0112] Step 3, converting the fundamental frequency feature f 0s ′ of the source singing voice extracted in step 1 into the target singing voice fundamental frequency feature f 0t ′ by the fundamental frequency conversion function in step 9 of the training stage;

[0113] Step 4, synthesizing the non-periodicity feature extracted in step 1, the reconstructed target singing voice spectral feature x st ′ obtained in step 2, and the target singing voice fundamental frequency feature f 0t ′ obtained in step 3, etc. using the WORLD voice analysis / synthesis model to obtain the final converted singing voice.

[0114] The singing voice conversion method based on dynamic convolution decomposition and CBAM attention mechanism disclosed in the present embodiment can be run by a computer program, which can be installed on a computer or encapsulated in a mobile storage medium.

[0115] The above describes the exemplary embodiments of the present application, which does not limit the patent protection scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields based on the content of the present application specification and drawings are also included in the patent protection scope of the present application.

Claims

1. A singing voice conversion method based on CBAM and dynamic convolution decomposition, characterized in that, The method comprises a training phase and a conversion phase: The training phase comprises the following steps: Step 1.1, obtaining a training corpus composed of singing voices of multiple singers; Step 1.

2. Extracting source singing voice spectral features x using the WORLD speech analysis / synthesis model s , fundamental frequency features f 0s , and unvoicing features; Step 1.3, extracting target singing voice style features s by a style encoder S t The generator G includes an encoding network and a decoding network, the source singing voice content features x are generated by the encoding network c Then the source singing voice content features x are reconstructed by the decoding network with the target singing voice style features s c And the singing voice is recovered t ​ Step 1.4, continuously adjusting the hyperparameters of the conversion network to minimize the loss function until the ideal number of iterations, thereby obtaining the trained conversion network; Step 1.5, constructing a pitch conversion function from source singing voice pitch features f 0s to target singing voice pitch features f 0t ​ The conversion phase comprises the following steps: Step 2.1, extracting the spectral features x of the source singing voice in the corpus to be converted using the WORLD speech analysis / synthesis model s , fundamental frequency features f 0s , and unvoicing features; Step 2.2, extracting target singing voice style features s by style encoder S t ′, input the above source singing voice spectrum features x s ′ and target singing voice style features s t ′ into the conversion network trained in step 1.4, to reconstruct target singing voice spectrum features x st ′; Step 2.3, convert the pitch features f of the source singing voice extracted in step 2.1 to the pitch features f of the target singing voice by the pitch conversion function in step 1.5 0s Step 2.3, convert the pitch features f of the source singing voice extracted in step 2.1 to the pitch features f of the target singing voice by the pitch conversion function in step 1.5 0t Step 2.3, convert the pitch features f of the source singing voice extracted in step 2.1 to the pitch features f of the target singing voice by the pitch conversion function in step Step 2.

4. Synthesizing the non-periodic features extracted in step 2.1, the reconstructed target singing voice spectral features x obtained in step 2.2 and the target singing voice fundamental frequency features f obtained in step 2.3 using the WORLD voice analysis / synthesis model to obtain converted singing voice. st 0t ′​ In step 1.4, the conversion network is composed of a generator G, a discriminator D and a style encoder S, the generator G is composed of an encoding network and a decoding network, the kernel of dynamic convolution in the generator is decomposed, dynamic channel fusion is used to replace dynamic attention to channel groups, the dimension of the latent space is reduced, and CBAM attention modules are introduced into the last four layers of the encoding network and the first four layers of the decoding network.

2. The CBAM and dynamic convolution decomposition based singing voice conversion method according to claim 1, characterized in that, The generating function of the kernel of the dynamic convolution is represented as: where W k is a static convolution kernel, K is the number of static convolution kernels, π k (x) is the attention score, and 3. The CBAM and dynamic convolution decomposition based singing voice conversion method according to claim 2, characterized in that, The generating function of the kernel of the dynamic convolution decomposition is represented as: W(x) = Λ(x)W0+ Pφ(x)Q T where Λ(x) is a C × C diagonal matrix, in this way the dynamic channel attention is achieved on W0; P matrix has a dimension of C × L, Q T has a dimension of L × C, and L << C, Pφ(x)Q T represents the sparse dynamic residual, which is a diagonal matrix; φ(x) is a full dynamic matrix, which is used to achieve dynamic channel fusion; W0is the weighted average value of K static convolution kernels.

4. The CBAM and dynamic convolution decomposition based singing voice conversion method according to claim 3, characterized in that, The dynamic convolution decomposition uses a lightweight dynamic branch to generate the coefficients of the dynamic channel fusion φ(x) and the dynamic channel attention Λ(x); the dynamic branch first applies average pooling to the input vector x, then makes the vector x pass through two fully connected layers and an activation layer in turn, and finally generates the convolution kernel W(x) by weighting; the dynamic convolution decomposition layer also includes a batch normalization layer and an activation layer.

5. The CBAM and dynamic convolution decomposition based singing voice conversion method according to claim 4, characterized in that, The CBAM attention module includes two sub-modules: a channel attention module and a spatial attention module, input feature map F, CBAM sequentially derives one-dimensional channel attention M c (F) and two-dimensional spatial attention M s (F′).

6. The CBAM and dynamic convolution decomposition based singing voice conversion method according to claim 5, characterized in that, The input of the channel attention module is a feature map F of HxWxC, two 1x1xC feature maps are obtained through channel-based global maximum pooling and channel-based global average pooling respectively; then a neural network shared multi-layer perceptron is used, the feature map output by the shared multi-layer perceptron is weighted, and then a Sigmoid activation operation is performed to generate the final channel attention M c (F); finally, it is multiplied with the input feature map F to generate the input feature map F' required by the spatial attention module.

7. The CBAM and dynamic convolution decomposition based singing voice conversion method according to claim 6, characterized in that, The spatial attention module takes the feature map F' output by the channel attention module as an input feature map of the module; the input feature map F' sequentially passes through channel-based global maximum pooling and channel-based global average pooling to obtain two HxWx1 feature maps, then the two feature maps are subjected to a channel splicing operation; then subjected to a 7x7 convolution operation to reduce the dimension to one channel, i.e. HxWx1, and then subjected to Sigmoid to generate spatial attention M s (F'), and finally subjected to a multiplication operation with the input feature map F' of the module to obtain a final feature map F".

Images