A voice conversion method, apparatus, and device

Abstract

The application discloses a speech conversion method, device and equipment, the method comprises the following steps: obtaining the speech to be converted; inputting the speech to be converted into a conversion model; decoupling the semantic information and the non-semantic information of the speech to be converted by the conversion model, and respectively converting the semantic information and the non-semantic information to generate target style speech; wherein the target style speech is the same as the semantic information of the speech to be converted, and the non-semantic information is different.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a voice conversion method, device and equipment. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, voice conversion (VC) is of great significance to the development of speaker identity conversion and personalized speech synthesis. The corpus required when training a voice conversion model is divided into parallel corpus or non-parallel corpus. Among them, the parallel corpus refers to the same content of the source speaker and the target speaker, while the non-parallel corpus refers to different content of the source speaker and the target speaker. The voice conversion model based on parallel corpus has high collection cost. Therefore, the voice conversion method of non-parallel corpus is increasingly concerned.

[0003] In related technologies, language information is used to assist model training, and a Time-Frequency Adaptive Normalization (TFAN) module is used to realize voice conversion of non-parallel corpus. The introduction of the TFAN module increases the model parameters, which slows down the convergence speed of the model and reduces the conversion speed of the model in use. Moreover, the generated voice after model conversion has a gap with the target voice, resulting in poor voice effect. SUMMARY

[0004] The present application provides a voice conversion method, device and equipment.

[0005] The technical solution of the present application is implemented as follows:

[0006] In a first aspect, the present application provides a voice conversion method, which comprises:

[0007] obtaining a voice to be converted;

[0008] inputting the voice to be converted into a conversion model;

[0009] decoupling and processing semantic information and non-semantic information of the voice to be converted by the conversion model, and respectively converting the semantic information and the non-semantic information to generate a target style voice;

[0010] wherein the target style voice has the same semantic information as the voice to be converted and different non-semantic information.

[0011] In some embodiments, the decoupling and processing semantic information and non-semantic information of the voice to be converted by the conversion model, and respectively converting the semantic information and the non-semantic information to generate a target style voice comprises:

[0012] splitting the speech to be converted to obtain at least one unit word;

[0013] decoupling the unit word containing the stress in the at least one unit word to obtain semantic information and non-semantic information;

[0014] retaining the semantic information and converting the non-semantic information to generate the target style speech.

[0015] In some embodiments, after the inputting the speech to be converted into the conversion model, the method further comprises:

[0016] determining a spectrum graph of the speech to be converted and extracting frequency domain feature information in the spectrum graph;

[0017] performing feature classification on the frequency domain feature information to determine at least one type of frequency domain feature;

[0018] assigning a first type of weight to the frequency domain feature belonging to the semantic information and a second type of weight to the frequency domain feature belonging to the non-semantic information; wherein the first type of weight is less than the second type of weight;

[0019] determining the spectrum graph after the weight assignment as a class activation map.

[0020] In some embodiments, the method further comprises:

[0021] converting the frequency domain feature in the class activation map to obtain time-frequency feature information;

[0022] performing convolution processing on the time-frequency feature information using a self-attention module to generate the target style speech.

[0023] In some embodiments, the method further comprises:

[0024] obtaining a sample set; wherein the sample set includes an original sample set and a target sample set, the original sample set includes at least one original sample speech before conversion, and the target sample set includes at least one target sample speech after conversion;

[0025] training a convolution model according to the at least one original sample speech before conversion and the at least one target sample speech after conversion to obtain the conversion model;

[0026] wherein the convolution model includes a self-attention module and a content style decoupling module.

[0027] In some embodiments, the self-attention module includes an interaction attention module, and the model training according to the at least one original sample voice before conversion and the at least one target sample voice after conversion to obtain the conversion model includes:

[0028] encoding the at least one original sample voice before conversion and the at least one target sample voice after conversion through an encoder to obtain first training feature information corresponding to the original sample voice and the target sample voice;

[0029] performing feature classification on the first training feature information using the interaction attention module, and performing weight distribution on at least one type of training frequency domain feature obtained by the classification to determine a class activation map;

[0030] decoding the class activation map through a decoder to obtain the conversion model.

[0031] In some embodiments, the self-attention module includes a self-attention module, and the decoding the class activation map through a decoder to obtain the conversion model includes:

[0032] converting the frequency domain feature in the class activation map to obtain second training feature information;

[0033] performing convolution model training on the second training feature information using the self-attention module to obtain the conversion model.

[0034] In some embodiments, after obtaining the conversion model, the method further includes:

[0035] inputting the to-be-verified voice into the conversion model to generate a verification conversion voice;

[0036] decoupling the to-be-verified voice and the verification conversion voice using the content style decoupling module to obtain semantic information and non-semantic information;

[0037] performing consistency calculation on the semantic information of the to-be-verified voice and the verification conversion voice to determine a semantic loss value;

[0038] performing perceptual loss judgment on the non-semantic information of the verification conversion voice and the target style voice to determine a style loss value;

[0039] in the case where the semantic loss value and the style loss value converge, determining that the model training is completed.

[0040] In a second aspect, the embodiments of the present application provide a voice conversion device, which includes:

[0041] an acquisition unit configured to acquire a to-be-converted voice;

[0042] The conversion unit is configured to input the to-be-converted voice into a conversion model, and perform decoupling processing on semantic information and non-semantic information of the to-be-converted voice by the conversion model, and perform conversion processing on the semantic information and the non-semantic information respectively, to generate a target style voice; wherein the target style voice is the same as the semantic information of the to-be-converted voice and different from the non-semantic information.

[0043] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor; wherein,

[0044] The memory is configured to store a computer program capable of running on the processor.

[0045] The processor is configured to execute the method according to the first aspect when running the computer program.

[0046] In a fourth aspect, an embodiment of the present application provides a computer storage medium, characterized by storing a computer program, wherein the computer program is executed by at least one processor to implement the method according to the first aspect.

[0047] The present application provides a voice conversion method, device and equipment, by obtaining the to-be-converted voice; inputting the to-be-converted voice into a conversion model; performing decoupling processing on the semantic information and non-semantic information of the to-be-converted voice by the conversion model, and performing conversion processing on the semantic information and non-semantic information respectively, to generate a target style voice; wherein the target style voice is the same as the semantic information of the to-be-converted voice and different from the non-semantic information. In this way, the conversion of the to-be-converted voice by the conversion model not only improves the quality of voice synthesis, but also improves the conversion performance of the model without introducing additional model parameters. In addition, the decoupling of semantic information and non-semantic information can also achieve high-quality conversion. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 A flowchart of a voice conversion method provided by an embodiment of the present application is shown;

[0049] Figure 2 A flowchart of another voice conversion method provided by an embodiment of the present application is shown;

[0050] Figure 3 A network structure diagram of a conversion model generator provided by an embodiment of the present application is shown;

[0051] Figure 4 A structure diagram of an interactive attention module in a conversion model generator provided by an embodiment of the present application is shown;

[0052] Figure 5 A structural diagram of a self-attention module in a conversion model generator is provided for an embodiment of the present application.

[0053] Figure 6 A network structure diagram of a conversion model discriminator is provided for an embodiment of the present application.

[0054] Figure 7 A component structure diagram of a voice conversion device is provided for an embodiment of the present application.

[0055] Figure 8 A specific hardware structure diagram of an electronic device is provided for an embodiment of the present application.

[0056] Figure 9 A component structure diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to enable a person skilled in the art to better understand the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings, which are only used for reference and are not intended to limit the embodiments of the present application.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0059] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict. It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects, and do not represent a specific order of the objects. Understandably, "first\second\third" can be interchanged with a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0060] It can be understood that voice conversion (VC) is of great significance to the development of speaker identity conversion, personalized speech synthesis and other applications. The corpus required when training a voice conversion model is divided into parallel corpus and non-parallel corpus. The parallel corpus refers to the same content of the source speaker and the target speaker, while the non-parallel corpus refers to the different content of the source speaker and the target speaker. The voice conversion model based on the parallel corpus needs to input a set of audio, and the source speaker and the target speaker express the same content, which has high collection cost. Therefore, the voice conversion method based on the non-parallel corpus is increasingly concerned.

[0061] In the related art, language information is used to assist model training to realize voice conversion of non-parallel corpus. Since obtaining such language information requires additional modules and data, it will bring additional cost and limit its application. In recent years, deep generative models have appeared, such as VAE-based methods and GAN-based methods. Such methods can realize voice synthesis of non-parallel corpus using only acoustic data. Among them, CycleGAN-VC and its variants, such as CycleGAN-VC2 and StarGAN-VCs, have been widely accepted as a benchmark method. However, the above-mentioned algorithms are only applicable to mel-cepstrum conversion and cannot capture the time-frequency structure of mel-spectrogram. In order to overcome these limitations, CycleGAN-VC3 proposes a time-frequency adaptive normalization (TFAN) module to realize mel-spectrogram conversion. However, CylceGAN-VC3 has the following shortcomings: the conversion effect of the model is poor, and the generated voice has a gap with the target voice, especially for words with heavy reading in the voice, the conversion effect is poor; the introduction of the TFAN module increases the model parameters, resulting in slow model convergence speed.

[0062] Based on this, the embodiment of the present application proposes a voice conversion method, the basic idea of which is: obtaining a voice to be converted; inputting the voice to be converted into a conversion model; decoupling the semantic information and non-semantic information of the voice to be converted by the conversion model, and converting the semantic information and non-semantic information respectively to generate a target style voice; wherein the target style voice has the same semantic information and different non-semantic information as the voice to be converted. In this way, the conversion of the voice to be converted by the conversion model not only improves the quality of voice synthesis, but also does not need to introduce additional model parameters, and also improves the conversion performance of the model and improves the training efficiency; in addition, the decoupling of the semantic information and the non-semantic information also enables high-quality conversion.

[0063] In an embodiment of the present application, referring to Figure 1 which shows a flowchart of a voice conversion method provided by an embodiment of the present application. As shown in Figure 1 , the method can include:

[0064] S101: obtaining a to-be-converted voice.

[0065] It should be noted that the voice conversion method provided by the embodiments of the present application can be applied to a device that needs to be converted, or an electronic device integrated with the device. Here, the electronic device can be, for example, a computer, a smart phone, a tablet computer, a notebook computer, a palm computer, a personal digital assistant (PDA), a navigation device, a server, etc., and the embodiments of the present application do not make specific limitations.

[0066] It should also be noted that the to-be-converted voice can include semantic information and non-semantic information. In the voice conversion process, the to-be-converted voice can be a word, a phrase or a sentence, or the to-be-converted voice can also be various languages, and the embodiments of the present application do not make specific limitations.

[0067] S102: inputting the to-be-converted voice into a conversion model.

[0068] S103: decoupling and processing the semantic information and non-semantic information of the to-be-converted voice by the conversion model, and respectively converting the semantic information and non-semantic information to generate a target style voice; wherein the target style voice has the same semantic information as the to-be-converted voice and different non-semantic information.

[0069] It should be noted that in the process of inputting the to-be-converted voice into the conversion model, the to-be-converted voice can be limited to a target style, i.e., the non-semantic information of the target style voice after conversion is limited; wherein the non-semantic information can include timbre, pause habit, speaking style, etc., or the to-be-converted voice can not be limited to a target style, and different styles of voice can be generated by conversion for the user to select, and the embodiments of the present application do not make any limitations.

[0070] It should also be noted that in the case of limiting the target style, for different to-be-converted voices, different target styles can be limited in the conversion process, or the same target style can be limited for these input sentences, and the embodiments of the present application do not make any limitations.

[0071] In some embodiments, the decoupling and processing of the semantic information and non-semantic information of the to-be-converted voice by the conversion model, and the conversion of the semantic information and non-semantic information to generate a target style voice can include:

[0072] Split the speech to be converted to obtain at least one unit word;

[0073] In the at least one unit word, the unit word containing the stress is decoupled to obtain semantic information and non-semantic information;

[0074] The semantic information is retained and the non-semantic information is converted to generate the target style speech.

[0075] It should be noted that for the speech to be converted, the whole speech to be converted can be decoupled to obtain semantic information and non-semantic information, and in the conversion process, the semantic information is retained and the non-semantic information is converted to obtain a target style speech with high conversion quality. The speech to be converted can also be split into unit words, and only the unit word containing the stress is decoupled to obtain semantic information and non-semantic information, and in the conversion process, the semantic information is retained and the non-semantic information is converted, and the remaining unit words are directly converted to obtain a target style speech with high conversion quality. In this way, not only is the conversion quality of the speech to be converted high, but also the operation pressure of the model is reduced and the speed of speech conversion is accelerated.

[0076] In some embodiments, after the speech to be converted is input into the conversion model, the method can further include:

[0077] Determine the frequency spectrum of the speech to be converted, and extract the frequency domain feature information in the frequency spectrum;

[0078] Classify the frequency domain feature information to determine at least one type of frequency domain feature;

[0079] Assign a first type of weight to the frequency domain feature belonging to the semantic information, and assign a second type of weight to the frequency domain feature belonging to the non-semantic information; wherein the first type of weight is less than the second type of weight;

[0080] Determine the frequency spectrum after the weight assignment as a class activation map.

[0081] It should be noted that the frequency spectrum is also called a mel frequency spectrum, and the frequency domain feature information extracted from the frequency spectrum is one-dimensional feature information. In the process of obtaining the frequency domain feature information, the conversion model converts the frequency spectrum from two-dimensional to one-dimensional. After classifying the frequency domain feature information extracted from the frequency spectrum, different weights are assigned to different frequency domain features according to the degree of attention required by different types of frequency domain features. The frequency domain features that need to be paid attention to are assigned higher weights, which can improve the conversion efficiency.

[0082] It should be noted that, for the conversion model, non-semantic information needs to be converted, and semantic information needs to be retained, so more attention needs to be paid to non-semantic information in the conversion process of the conversion model, and therefore, in the conversion model, the weight of non-semantic information is higher, and the weight of semantic information is lower.

[0083] It should be noted that, after the weight allocation of the frequency domain features, a class activation map is obtained, and the class activation map includes a source domain and a target domain. The source domain can include frequency domain features with lower weights, i.e., semantic information includes frequency domain features; and the target domain can include frequency domain features with higher weights, i.e., non-semantic information includes frequency domain features.

[0084] In some embodiments, the method can further include:

[0085] Converting the frequency domain features in the class activation map to obtain time-frequency feature information;

[0086] Using a self-attention module to perform convolution processing on the time-frequency feature information to generate the target style speech.

[0087] It should be noted that, when the frequency domain features in the class activation map are converted to obtain time-frequency feature information, one-dimensional frequency domain features can be converted into two-dimensional time-frequency features, and then a self-attention module is used for convolution training on the time-frequency features, thereby increasing the relevance of time domain information and frequency domain information, improving the connection between frames of the target style speech after the conversion of the to-be-converted speech, and further improving the coherence of the target style speech.

[0088] In the embodiments of the present application, the conversion model can be divided into a conversion model generator and a conversion model discriminator. The conversion model generator can include a sentence encoder, a self-attention module and a decoder; and the conversion model discriminator can include a content style decoupling module.

[0089] In some embodiments, referring to Figure 2 which shows a flowchart of another voice conversion method provided by the embodiments of the present application. Specifically, the training process of the conversion model, as shown in Figure 2 , the method can include:

[0090] S201: obtaining a sample set; wherein the sample set includes an original sample set and a target sample set, the original sample set includes at least one original sample speech before conversion, and the target sample set includes at least one target sample speech after conversion.

[0091] It should be noted that the at least one original sample voice before conversion and the at least one target sample voice after conversion are two independent training sets, that is, non-parallel corpus is used for training, specifically, the non-parallel corpus refers to that in the conversion process, the speaking content of the original sample voice and the target sample voice is different, and the two sample training sets have no association.

[0092] S202: training a convolution model according to at least one original sample voice before conversion and at least one target sample voice after conversion to obtain a conversion model; wherein the convolution model can include a self-attention module and a content style decoupling module.

[0093] It should be noted that the self-attention module is used to adjust the model to pay more attention to important features, speed up the adjustment of parameters in the model, improve the conversion performance of the model, and improve the training efficiency, and the content style decoupling module is used to convert non-semantic information from the original sample signal to the converted sample signal. On the other hand, the self-attention module also needs to retain the semantics of the source voice signal to strengthen the conversion quality of voice conversion.

[0094] In some embodiments, the self-attention module can include an interactive attention module, and the model training according to the at least one original sample voice before conversion and the at least one target sample voice after conversion to obtain the conversion model can include:

[0095] The at least one original sample voice before conversion and the at least one target sample voice after conversion are encoded by an encoder to obtain first training feature information corresponding to the original sample voice and the target sample voice;

[0096] The first training feature information is classified using the interactive attention module, and at least one type of training frequency domain feature obtained by the classification is weighted to determine a class activation map;

[0097] The class activation map is decoded by a decoder to obtain the conversion model.

[0098] It should be noted that the at least one original sample voice before conversion and the at least one target sample voice after conversion are encoded by an encoder to obtain first training feature information, and the first training feature information is one-dimensional feature information. In the encoding process, the frequency spectrum of the original sample voice and the target sample voice is converted from two-dimensional to one-dimensional. After the first training feature information is classified, different weights are assigned to different frequency domain features according to the degree of attention required by different types of frequency domain features, and higher weights are assigned to the frequency domain features that need to be paid attention to, which can improve the conversion efficiency.

[0099] In some embodiments, the self-attention module can include a self-attention module, and the decoding processing of the class activation map by the decoder can include:

[0100] Converting the frequency domain features in the class activation map to obtain second training feature information;

[0101] Using the self-attention module to perform convolution model training on the second training feature information to obtain the conversion model.

[0102] It should be noted that when the frequency domain features in the class activation map are converted to obtain the second training feature information, the one-dimensional first training feature information is converted into two-dimensional second training feature information, and the self-attention module is used for convolution training on the second training feature information, thereby increasing the relevance of the time domain information and the frequency domain information, improving the connection between the target style speech frames after the conversion of the to-be-converted speech, and improving the coherence of the target style speech.

[0103] In some embodiments, after obtaining the conversion model, the method can further include:

[0104] Inputting the to-be-verified speech into the conversion model to generate a verification converted speech;

[0105] Using the content style decoupling module to perform decoupling processing on the to-be-verified speech and the verification converted speech to obtain semantic information and non-semantic information;

[0106] Performing consistency calculation on the semantic information of the to-be-verified speech and the verification converted speech to determine a semantic loss value;

[0107] Performing perceptual loss judgment on the non-semantic information of the verification converted speech and the target style speech to determine a style loss value;

[0108] In the case where the semantic loss value and the style loss value converge, it is determined that the model training is completed.

[0109] It should be noted that compared with the conventional discriminator which only judges whether the semantic information conversion is accurate or not, the discriminator of the embodiment of the present application increases a module for calculating the style loss, and in the discrimination process, the style loss value is calculated at the same time as the semantic loss value is determined, and then the conversion quality of the model is judged in combination with the semantic loss value and the style loss value. In the case where the semantic loss value and the style loss value converge, it is determined that the model training is completed.

[0110] In this way, the convolution model is trained according to at least one original sample voice before conversion and at least one target sample voice after conversion, to obtain a final conversion model; and the conversion model has less model parameters, fast convergence speed and good synthesis effect.

[0111] The embodiment provides a voice conversion method, which comprises the following steps: obtaining a voice to be converted; inputting the voice to be converted into a conversion model; decoupling semantic information and non-semantic information of the voice to be converted by the conversion model, and converting the semantic information and the non-semantic information respectively to generate a target style voice; wherein the target style voice is the same as the voice to be converted in semantic information and different in non-semantic information. In this way, the conversion of the voice to be converted by the conversion model can not only improve the voice synthesis quality, but also improve the conversion performance of the model without introducing additional model parameters; in addition, the decoupling of the semantic information and the non-semantic information can also realize high-quality conversion.

[0112] In another embodiment of the present application, based on the voice conversion method of the foregoing embodiment, referring to Figure 3 which shows a network structure schematic diagram of a conversion model generator provided by the application embodiment. As shown in Figure 3 , the generator of the conversion model can include an encoding module 301, an attention module and a decoding module 302. The attention module can include an interaction attention module and a self-attention module.

[0113] It should be noted that the generator network structure in the end-to-end unsupervised voice synthesis system (U-GAT-VC, i.e. the conversion model in the foregoing embodiment). Wherein h, w and c represent the height, width and channel number of the feature respectively. In each convolution layer, k, c and s represent the kernel number, channel number and step length respectively.

[0114] Specifically, on the one hand, referring to Figure 4 which shows a structure schematic diagram of an interaction attention module in a conversion model generator provided by the application embodiment. As shown in Figure 4As shown, for the inter-attention module 40, when performing voice conversion, the timbre of the voice to be converted is the voice feature of the speaker, which is reflected by the formant frequency. In the mel-spectrogram, the formant corresponds to the significant frequency component of the spectral envelope. We embed an auxiliary classifier in the generator, and by using the class activation maps (CAM) derived by the auxiliary classifier, the inter-attention module 40 is used to assign different weights to different categories of different categories, and the class activation maps are used as attention maps (weights of feature maps) of the mel-spectrogram to distinguish the source domain and the target domain. Among them, the features in the source domain are assigned a lower weight, and the features in the target domain are assigned a higher weight, so as to guide the model to pay attention to more important areas of the mel-spectrogram, and improve the conversion efficiency.

[0115] On the other hand, referring to Figure 5 , which shows a structure diagram of a self-attention module in a conversion model generator provided by an embodiment of the present application. As Figure 5 shown, for the intra-attention / self-attention module 50, the mel-spectrogram is a two-dimensional time-frequency representation of the voice signal. By adding the self-attention mechanism, the time-frequency information of the mel-spectrogram can be captured, the relationship between the time domain information and the frequency domain information is increased, the fluency of the target style voice after the conversion of the voice to be converted is improved, the coherence of the synthesis model is also improved, and the model training efficiency is also improved.

[0116] Referring to Figure 6 , which shows a network structure diagram of a conversion model discriminator provided by an embodiment of the present application. As Figure 6 shown, the conversion model discriminator includes a content style disentangled module 60. For the content style disentangled module 60, the module is mainly a disentangled perceptual loss of content style. The module needs to complete two tasks: on the one hand, it needs to transmit non-semantic information (i.e. timbre) from the source voice signal to the target signal. On the other hand, it needs to preserve the semantics (i.e. content) of the source voice signal. Therefore, in the network training process, in addition to the discrimination of semantic content loss, an additional disentangled perceptual loss of content style is added to the discriminator to realize the discrimination of the conversion effect of non-semantic information.

[0117] In some embodiments, the embodiments of the present application provide an end-to-end unsupervised speech synthesis system U-GAT-VC (Unsupervised Generative Attentional Networks for Non-parallel Voice Conversion). The system comprises at least the following two modules:

[0118] Module 1: Attention module (inter-attention and intra-attention / self-attention).

[0119] The attention module is to guide the model to quickly capture important information of the mel-spectrogram, improve the conversion performance of the model, and improve the training efficiency, specifically as follows:

[0120] (1) Inter-attention: timbre is a speaker's voice feature, which is reflected by the formant frequency. In the mel-spectrogram, the formant corresponds to the significant frequency component of the spectral envelope. We embed an auxiliary classifier in the generator, and use the class activation map (CAM) derived by the auxiliary classifier as the attention map (i.e. the weight of the feature map) of the mel-spectrogram to distinguish the source domain and the target domain, so as to guide the model to pay attention to the more important area of the mel-spectrogram and improve the conversion efficiency.

[0121] (2) Intra-attention / Self-attention: the mel-spectrogram is a two-dimensional time-frequency representation of the speech signal. By adding the self-attention mechanism, the time-frequency information of the mel-spectrogram can be captured, the coherence of the synthesis model is improved, and the training efficiency of the model is also improved.

[0122] The attention module is to guide the model to quickly capture important information of the mel-spectrogram, improve the conversion performance of the model, and improve the training efficiency, specifically as follows:

[0123] (1) Inter-attention: we embed an auxiliary classifier in the generator, and use the class activation map (CAM) derived by the auxiliary classifier as the attention map (i.e. the weight of the feature map) of the mel-spectrogram to distinguish the source domain and the target domain, so as to guide the model to pay attention to the more important area of the mel-spectrogram and improve the conversion efficiency.

[0124] Specifically, the U-GAT-VC conversion model is composed of an encoder, a decoder and an auxiliary classifier, wherein the auxiliary classifier ηx(x) represents the probability that the current signal comes from the source domain X. By training the auxiliary classifier, and with the help of the average pooling layer (GAP) and the maximum pooling layer (GMP), the weight of the kth feature map is trained. The attention feature map can be obtained by the following formula:

[0125]

[0126] Wherein, X represents the source domain, x represents a single sample signal in the source domain, ω represents the weight of the feature map, and n represents the number of feature maps.

[0127] And the CAM loss L cam The CAM loss L can be obtained by the following formula:

[0128]

[0129]

[0130] Wherein, Y represents the target domain, y represents a single sample signal in the target domain, And represents the expected value of the current signal from the source domain.

[0131] (2) Intra-attention / Self-attention: the current feature layer x is passed through f, g and h to generate variables query, key and value, and then the attention matrix is calculated by the following formula:

[0132]

[0133] Wherein, i, j represent the coordinates of the attention matrix, N represents the size of the matrix, T represents the transposed matrix, f, g and h represent a convolution kernel respectively.

[0134] Here, the attention map S with weight is calculated:

[0135] S = v(x)(h(x)A T )

[0136] Wherein, v represents the weight function, h represents the value function, and S multiplied by a proportional parameter can obtain the final output of the self-attention module.

[0137] Module two: content style disentangled perceptual loss.

[0138] In the embodiments of the present application, the content style disentangled perceptual loss is introduced to optimize the generator, which can improve the quality of speech synthesis; especially for the words containing heavy reading in the speech, the semantic and timbre decoupling is realized to achieve high-quality conversion.

[0139] The voice conversion model needs to complete two tasks: on the one hand, it is necessary to transfer the non-semantic information (including timbre, style, etc.) from the source voice signal to the target signal. On the other hand, it is necessary to retain the semantics (i.e. content) of the source voice signal. Therefore, in the network training process, an additional content style decoupling perceptual loss is added in the discriminator to achieve this goal.

[0140] Specifically, the perceptual loss L perc can be divided into two sub-losses: content loss (i.e. semantic loss in the foregoing embodiment) L cont and style loss L sty The calculation formula of the perceptual loss is as follows:

[0141] L perc = L cont + L sty

[0142] Here, given a content target (source speaker voice) x, a style target (target speaker voice) y, and a model synthesized voice y', the content loss is calculated by calculating the Euclidean distance of the third layer feature map of the perceptual network, and the specific calculation is as follows:

[0143]

[0144] The style loss is obtained by calculating the distance of the Gram matrix of the fourth layer feature map of the perceptual network, and the specific calculation is as follows:

[0145]

[0146] Where, Euclidean distance, C: channel, H: high, W: wide. Gram matrix, the Euclidean Frobenius distance of the Gram matrix of the output and the target voice signal Mel spectrogram

[0147] The foregoing embodiment is described in detail through the foregoing embodiment. As can be seen from the foregoing embodiment, through the technical solution of the foregoing embodiment, on the one hand, the conversion model provided by the embodiment only needs a small amount of non-parallel corpus to train the model, the model has fewer parameters, faster convergence speed, and better synthesis effect (in the training set VCC2018, 256 source domain and target domain corpora with a frequency of 22.05 kHz and a window length of 1024 are used for training, the model parameters are reduced by 34.69%, the training time is reduced by 34.33%, and the subjective and objective evaluations are improved); on the other hand, the model conversion effect is good, solving the problem of poor conversion effect of words containing a stress accent; on the other hand, the effect of the embodiment of the present application is detectable, and currently many voice synthesis algorithms cannot properly retain the stress accent position during conversion, while the embodiment of the present application can accurately convert unit words containing a stress accent during voice conversion, greatly improving the quality of the converted voice.

[0148] In yet another embodiment of the present application, based on the same inventive concept as the foregoing embodiments, see Figure 7 which shows a schematic diagram of the composition structure of a voice conversion device 70 provided by an embodiment of the present application. As shown in Figure 7 The voice conversion device 70 can include an acquisition unit 701 and a conversion unit 702; wherein

[0149] The acquisition unit 701 is configured to acquire a voice to be converted.

[0150] The conversion unit 702 is configured to input the voice to be converted into a conversion model, and perform decoupling processing on semantic information and non-semantic information of the voice to be converted by the conversion model, and perform conversion processing on the semantic information and the non-semantic information respectively, to generate a target style voice; wherein the target style voice is the same as the semantic information of the voice to be converted, and different from the non-semantic information.

[0151] In some embodiments, the conversion unit 702 is specifically configured to split the voice to be converted to obtain at least one unit word; and perform decoupling processing on a unit word containing a stress sound in the at least one unit word to obtain semantic information and non-semantic information; and retain the semantic information and convert the non-semantic information to generate the target style voice.

[0152] In some embodiments, the conversion unit 702 is further configured to determine a spectrum graph of the voice to be converted, and extract frequency domain feature information in the spectrum graph; and perform feature classification on the frequency domain feature information to determine at least one type of frequency domain feature; and assign a first type of weight to the frequency domain feature belonging to the semantic information; assign a second type of weight to the frequency domain feature belonging to the non-semantic information; wherein the first type of weight is less than the second type of weight; and determine the spectrum graph after the weight assignment as a class activation map.

[0153] In some embodiments, the conversion unit 702 is further configured to convert the frequency domain features in the class activation map to obtain time-frequency feature information; and use a self-attention module to perform convolution processing on the time-frequency feature information to generate the target style voice.

[0154] In the above scheme, see Figure 7The voice conversion device 70 further includes a training unit 703 configured to obtain a sample set; wherein the sample set includes an original sample set and a target sample set, the original sample set includes at least one original sample voice before conversion, and the target sample set includes at least one target sample voice after conversion; and train a convolution model according to the at least one original sample voice before conversion and the at least one target sample voice after conversion to obtain the conversion model; wherein the convolution model includes a self-attention module and a content style decoupling module.

[0155] In some embodiments, the training unit 703 is specifically configured to encode the at least one original sample voice before conversion and the at least one target sample voice after conversion through an encoder to obtain first training feature information corresponding to the original sample voice and the target sample voice; and use the interaction attention module to classify the first training feature information, and assign weights to at least one type of training frequency domain feature obtained by classification to determine a class activation map; and decode the class activation map through a decoder to obtain the conversion model.

[0156] In some embodiments, the training unit 703 is specifically configured to convert the frequency domain features in the class activation map to obtain second training feature information; and use the self-attention module to train the convolution model according to the second training feature information to obtain the conversion model.

[0157] In some embodiments, the training unit 703 is further configured to input the to-be-verified voice into the conversion model to generate a verification conversion voice; and use the content style decoupling module to decouple the to-be-verified voice and the verification conversion voice to obtain semantic information and non-semantic information; and perform consistency calculation on the semantic information of the to-be-verified voice and the verification conversion voice to determine a semantic loss value; and perform perceptual loss judgment on the non-semantic information of the verification conversion voice and the target style voice to determine a style loss value; and in the case that the semantic loss value and the style loss value converge, determine that the model training is completed.

[0158] It can be understood that in the present embodiment, the "unit" can be a part of circuit, a part of processor, a part of program or software, etc., and of course can be a module, and can also be non-modular. Moreover, the components in the present embodiment can be integrated in one processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function module.

[0159] The integrated unit, if implemented in the form of a software function module and not sold or used as an independent product, can be stored in a computer readable storage medium based on such understanding. The technical solutions of the embodiments essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in the embodiments. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0160] Therefore, the embodiments provide a computer storage medium storing a speech conversion program, which, when executed by at least one processor, implements the steps of the method of any one of the preceding embodiments.

[0161] Based on the composition of the speech conversion device 70 and the computer storage medium, refer to Figure 8 , which shows a specific hardware structure schematic diagram of an electronic device provided by the embodiments of the present application. As Figure 8 indicated, it can include a communication interface 801, a memory 802 and a processor 803; various components are coupled together through a bus system 804. It can be understood that the bus system 804 is used to realize the connection communication between the components. In addition to including a data bus, the bus system 804 also includes a power bus, a control bus and a status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 804 in Figure 8 . Among them, the communication interface 801 is used for receiving and sending signals in the process of transceiving information with other external network elements;

[0162] The memory 802 is used for storing computer programs capable of running on the processor 803;

[0163] The processor 803 is used for executing the following steps when running the computer program:

[0164] Obtaining the speech to be converted;

[0165] Inputting the speech to be converted into a conversion model;

[0166] Decoupling the semantic information and the non-semantic information of the speech to be converted by the conversion model, and respectively converting the semantic information and the non-semantic information to generate the target style speech;

[0167] The target style voice is the same in semantic information and different in non-semantic information as the to-be-converted voice.

[0168] It can be understood that the memory 802 in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM) or a flash memory. The volatile memory can be a random access memory (Random Access Memory, RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Synchronous link DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DRRAM). The memory 802 of the system and method described herein is intended to include, but not limited to, these and any other suitable types of memory.

[0169] The processor 803 can be an integrated circuit chip including a processing unit that is configured to process signals. In implementation, the steps of the above-described method can be completed by the integrated logic circuit of the processor 803 or the instructions in the form of software. The processor 803 described above can be a general-purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed by the processor 803. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly embodied as hardware code of the processor or a combination of hardware and software modules in the processor. The software module can reside in the storage media of the memory 802, the flash memory, the read-only memory (ROM), the programmable read-only memory (PROM), the electrically programmable read-only memory (EPROM), the electrically erasable programmable read-only memory (EEPROM), the register, or other forms of the storage media in the art. The storage media is located in the memory 802, and the processor 803 reads information in the memory 802 and combines the hardware to complete the steps of the above-described method.

[0170] It can be understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing units can be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSP Devices), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions described herein, or a combination thereof.

[0171] For software implementation, the techniques described herein can be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. The software codes can be stored in the memory and executed by the processor. The memory can be implemented within the processor or external to the processor.

[0172] Optionally, as another embodiment, the processor 803 is further configured to execute the steps of the method of any one of the preceding embodiments when running the computer program.

[0173] In some embodiments, referring to Figure 9 , a schematic diagram of a composition structure of an electronic device 80 is shown. As Figure 9 indicated, the electronic device 80 at least includes the voice conversion apparatus 70 of any one of the preceding embodiments.

[0174] In the embodiments of the present application, for the electronic device 80, by obtaining the to-be-converted voice; inputting the to-be-converted voice into the conversion model; performing decoupling processing on the semantic information and the non-semantic information of the to-be-converted voice by the conversion model, and respectively performing conversion processing on the semantic information and the non-semantic information, a target style voice is generated; wherein the target style voice is the same as the semantic information of the to-be-converted voice and different from the non-semantic information. In this way, by using the conversion model to convert the to-be-converted voice, not only the quality of the synthesized voice can be improved, but also the conversion performance of the model can be improved without introducing additional model parameters; in addition, the decoupling of the semantic information and the non-semantic information can also achieve high-quality conversion.

[0175] It should be noted that in the present application, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0176] The serial numbers of the embodiments of the present application described above are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0177] The methods disclosed in the several method embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0178] The features disclosed in the several product embodiments provided by the present application can be combined arbitrarily without conflict to obtain new product embodiments.

[0179] The features disclosed in the several method or device embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

[0180] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A speech conversion method, the method comprising: Get the speech to be converted; The speech to be converted is input into the conversion model; The speech to be converted is split into at least one unit word using the conversion model. In the at least one unit word, the unit word containing stressed syllables is decoupled through the conversion model to obtain semantic information and non-semantic information; The semantic information is preserved and the non-semantic information is transformed using the transformation model to generate the target style speech; The target style speech and the speech to be converted have the same semantic information but different non-semantic information.

2. The method according to claim 1, further comprising, after inputting the speech to be converted into the conversion model: Determine the spectrogram of the speech to be converted, and extract the frequency domain feature information from the spectrogram; The frequency domain feature information is classified to determine at least one type of frequency domain feature; A first type of weight is assigned to the frequency domain features belonging to the semantic information; a second type of weight is assigned to the frequency domain features belonging to the non-semantic information; wherein the first type of weight is less than the second type of weight; The spectrum graph after weight allocation is determined as a class activation graph.

3. The method according to claim 2, further comprising: The frequency domain features in the activation graph are transformed to obtain time-frequency feature information; The time-frequency feature information is convolved using a self-attention module to generate the target style speech.

4. The method according to claim 1, further comprising: Obtain a sample set; wherein the sample set includes an original sample set and a target sample set, the original sample set includes at least one original sample speech before conversion, and the target sample set includes at least one target sample speech after conversion; The convolutional model is trained based on the at least one original sample speech before conversion and the at least one target sample speech after conversion to obtain the conversion model; The convolutional model includes a self-attention module and a content-style decoupling module.

5. The method according to claim 4, wherein the self-attention module includes an interactive attention module, and the step of training the model based on the at least one original sample speech before conversion and the at least one target sample speech after conversion to obtain the conversion model includes: The original sample speech before conversion and the target sample speech after conversion are encoded by an encoder to obtain the first training feature information corresponding to the original sample speech and the target sample speech. The interactive attention module is used to classify the first training feature information, and weights are assigned to at least one class of training frequency domain features obtained by the classification to determine the class activation map. The class activation graph is decoded by a decoder to obtain the transformation model.

6. The method according to claim 5, wherein the self-attention module comprises a self-attention module, and the step of decoding the class activation graph through a decoder to obtain the transformation model comprises: The frequency domain features in the activation graph are transformed to obtain the second training feature information; The self-attention module is used to train the second training feature information into a convolutional model to obtain the transformation model.

7. The method according to any one of claims 4 to 6, wherein after obtaining the conversion model, the method further comprises: The speech to be verified is input into the conversion model to generate the verified converted speech; The content style decoupling module is used to decouple the speech to be verified and the verified converted speech to obtain semantic and non-semantic information. The semantic information of the speech to be verified and the verified converted speech are calculated to determine the semantic loss value; Perceptual loss is assessed on the non-semantic information of the verification converted speech and the target style speech to determine the style loss value; The model training is considered complete when the semantic loss value and style loss value converge.

8. A speech conversion device, comprising: The acquisition unit is configured to acquire the speech to be converted; A conversion unit is configured to input the speech to be converted into a conversion model; The conversion model is specifically used to split the speech to be converted into at least one unit word; among the at least one unit word, the unit word containing stressed syllables is decoupled to obtain semantic information and non-semantic information; the semantic information is preserved and the non-semantic information is converted to generate the target style speech; The target style speech and the speech to be converted have the same semantic information but different non-semantic information.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor; wherein... The memory is used to store computer programs that can run on the processor; The processor is configured to perform the method as described in any one of claims 1 to 7 when running the computer program.

Images