Sound changing method and device based on model distillation, computer equipment and storage medium
Through model distillation technology, a real-time voice change model suitable for resource-constrained devices was constructed, which solved the problems of low computing efficiency and high inference delay of AI voice change models on low-end devices, and achieved high-quality real-time voice change, suitable for a variety of Terminal equipment.
Patent Information
- Application Number
- CN202410974405.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-08
AI Technical Summary
The existing AI voice change model is inefficient in computing and has high inference latency on devices with resource-constrained resources, especially in mobile devices and low-end CPU devices. The traditional voice change method has a high hearing delay when changing voice in real time, and the graphics card resource occupies Too much will lead to a decrease in the quality of voice change.
Using a voice change method based on model distillation, the voice data and template audio are distilled through the teacher model to obtain the distillation data set, and then used to train the student model to build a real-time voice change model to reduce the model parameter amount and calculation overhead, which is suitable for More system platforms.
It realizes end-to-end real-time sound change with low latency, reduces the listening delay, significantly reduces the model's calculation complexity and storage space requirements, and enables voice change technology to be widely used in more terminal devices and real-time applications, and has high voice change quality.
Smart Images

Figure CN118918908A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer software technology, and in particular to a voice changing method, device, computer equipment and storage medium based on model distillation. Background Art
[0002] With the rapid development of artificial intelligence and deep learning technologies, AI voice changing technology has made significant progress in many fields. However, although the existing AI voice changing models perform well in generating high-quality, natural target voices, the structures of these AI voice changing models are very complex, so they require a lot of computing resources and storage space, which limits their application on resource-constrained devices, that is, their scope of application is not wide enough. Especially in mobile devices, Internet of Things (IoT) devices, and low-end CPU and integrated graphics devices, the computational efficiency and inference latency of the model become particularly critical. In addition, in desktop voice changing software, since most user scenarios are game scenarios, graphics card resources are already greatly occupied. At this time, if the existing AI voice changing model is used, it will occupy the user's graphics card resources. Therefore, the graphics card performance required to achieve real-time voice changing will also be higher, resulting in increased difficulty in achieving real-time voice changing.
[0003] Traditional voice changing methods use Gaussian mixture models, while current voice changing methods use artificial neural networks. Voice changing architectures typically include variational autoencoders (VAE) and generative adversarial networks (GANs), which are usually trained on non-parallel datasets, i.e., datasets where speakers do not need to perform the same utterances. Although they can achieve real-time voice changing on mainstream consumer graphics cards such as NVIDIA RTX3060, their model architectures have not been trained to process low-latency streaming audio, which results in higher auditory delays (e.g., audio input length 200ms, model inference time 100ms, auditory delay excluding system delay is around 300ms). In addition, simply converting short continuous audio clips can also result in a decrease in perceived output quality. Summary of the invention
[0004] The embodiments of the present invention provide a voice changing method, apparatus, computer equipment and storage medium based on model distillation, aiming to ensure the quality of voice changing while improving the applicability of real-time voice changing.
[0005] In a first aspect, an embodiment of the present invention provides a voice changing method based on model distillation, comprising:
[0006] Obtain a basic training set containing multiple voice data, and input the voice data in the basic training set and a preset template audio into a teacher model for distillation training to obtain a distilled data set containing distilled data;
[0007] Inputting the distilled data in the distilled data set into the student model for voice change training to obtain voice change audio, thereby constructing a real-time voice change model;
[0008] The real-time voice changing model is used to perform voice changing processing on the specified audio data.
[0009] In a second aspect, an embodiment of the present invention provides a voice changing device based on model distillation, comprising:
[0010] A distillation training unit, used to obtain a basic training set containing multiple voice data, and input the voice data in the basic training set and a preset template audio into a teacher model for distillation training to obtain a distillation data set containing distilled data;
[0011] A voice change training unit, used for inputting the distilled data in the distilled data set into the student model for voice change training to obtain voice change audio, thereby constructing a real-time voice change model;
[0012] The real-time voice changing unit is used to perform voice changing processing on the specified audio data using the real-time voice changing model.
[0013] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the voice changing method based on model distillation as described in the first aspect is implemented.
[0014] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the voice changing method based on model distillation as described in the first aspect is implemented.
[0015] The embodiment of the present invention provides a voice changing method, device, computer equipment and storage medium based on model distillation, the method comprising: obtaining a basic training set containing multiple voice data, and inputting the voice data in the basic training set and a preset template audio into a teacher model for distillation training to obtain a distilled data set containing distilled data; inputting the distilled data in the distilled data set into a student model for voice changing training to obtain voice changing audio, thereby constructing a real-time voice changing model; and using the real-time voice changing model to perform voice changing processing on the specified audio data. The embodiment of the present invention first performs distillation training on the teacher model through voice data and template audio to obtain a distilled data set, and then uses the distilled data set to perform voice changing training on the student model, so that not only can some performance advantages of the teacher model be retained, but also the amount of model parameters and calculation overhead can be significantly reduced, so that while ensuring the quality of voice changing, the voice changing method provided in this embodiment can be applied to more system platforms without being limited by the graphics card resources of the system platform. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.
[0017] Figure 1 A schematic diagram of a flow chart of a voice changing method based on model distillation provided in an embodiment of the present invention;
[0018] Figure 2 A diagram of the teacher model architecture in a voice changing method based on model distillation provided in an embodiment of the present invention;
[0019] Figure 3 A diagram of the student model architecture in a voice changing method based on model distillation provided in an embodiment of the present invention;
[0020] Figure 4 A causal convolution preprocessing layer architecture diagram in a voice changing method based on model distillation provided in an embodiment of the present invention;
[0021] Figure 5 A diagram of the architecture of a dilated causal convolutional encoder in a voice changing method based on model distillation provided in an embodiment of the present invention;
[0022] Figure 6 A Transformer decoder architecture diagram in a voice changing method based on model distillation provided in an embodiment of the present invention;
[0023] Figure 7 A schematic block diagram of a voice changing device based on model distillation provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0025] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0026] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0027] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0028] See below Figure 1 , Figure 1 A flow chart of a voice changing method based on model distillation provided in an embodiment of the present invention, the method specifically comprises: steps S101 to S103.
[0029] S101, obtaining a basic training set containing a plurality of speech data, and inputting the speech data in the basic training set and a preset template audio into a teacher model for distillation training, to obtain a distillation data set containing distilled data;
[0030] S102, inputting the distilled data in the distilled data set into the student model for voice change training to obtain voice change audio, thereby constructing a real-time voice change model;
[0031] S103: Perform voice change processing on the specified audio data using the real-time voice change model.
[0032] In this embodiment, the teacher model is firstly distilled and trained using speech data and template audio to obtain a distilled data set, and then the student model is trained to change its voice using the distilled data set. This not only retains some performance advantages of the teacher model, but also significantly reduces the number of model parameters and computational overhead. This ensures the quality of the voice change while enabling the voice change method provided in this embodiment to be applicable to more system platforms without being limited by the graphics card resources of the system platform.
[0033] The voice changing method provided in this embodiment is based on model distillation, which can achieve low-latency end-to-end real-time voice changing, so that the auditory delay excluding system delay can be reduced to about 50ms on consumer-grade CPUs such as Inter i5-12400F. And through knowledge distillation, the knowledge in a large-scale, high-performance AI voice changing teacher model can be effectively transferred to a more concise AI voice changing student model. This not only significantly reduces the computational complexity of the model, but also realizes real-time voice changing on the CPU and integrated graphics and reduces storage space requirements, so that voice changing technology can be more widely used in various terminal devices and real-time applications.
[0034] Preferably, this embodiment can also introduce adversarial learning ideas, and the discriminator adopts the multi-period discriminator architecture of VITS, where the discriminator period is [2, 3, 5, 7, 11, 17, 23, 37], so as to further improve the voice changing quality.
[0035] In actual application scenarios, the voice changing method provided in this embodiment can achieve good results after training on multiple template character audios, and the audio results have a high degree of similarity with the template character's timbre, and the generated audio has no mechanical sound, noise, etc., and is very natural, that is, the voice changing quality is high. At the same time, the model volume is controlled within 15M, and real-time voice changing can be achieved on Intel i512400F with a delay of about 50ms.
[0036] In a specific embodiment, the LibriTTS dataset is selected as the basic training set. LibriTTS is a multi-speaker English corpus containing about 585 hours of English speech with a sampling rate of 24kHz. The LibriTTS corpus is designed for TTS research. It is derived from the original material of the LibriSpeech corpus. This embodiment can use two subsets, train-clean-360.tar.gz and dev-clean.tar.gz, which together total 120,000 voice data as the basic training set.
[0037] In one embodiment, the step S101 includes:
[0038] Extracting content features unrelated to the speaker's timbre from the speech data by a content encoder, and extracting pitch information from the speech data by a pitch extractor;
[0039] Inputting the content features and pitch information into a text encoder to encode the content information, and performing attention weighted summation on the results of the content information encoding to obtain a priori latent variables;
[0040] Obtaining a character number corresponding to the model audio, and inputting the character number into an embedding layer to obtain a speaker embedding vector;
[0041] Inputting the prior latent variable and the speaker embedding vector into a flow model to obtain a latent variable;
[0042] The latent variable and the speaker embedding vector are input into a vocoder for decoding to obtain audio waveform data, and the audio waveform data is used as a label of the speech data, and then the speech data and its label are used as the distilled data.
[0043] Combination Figure 2In this embodiment, the teacher model uses the existing real-time voice change model based on the content encoder Contentvec, the text encoder TextEncoder, the flow model Flow, the vocoder NSF-Hifigan and the feature fusion method, and uses the basic training set and the template audio to train the teacher model, so that the teacher model can convert any human voice audio into the character voice audio corresponding to the template audio. Specifically, the training stage of the teacher model only requires two parameters, namely the voice data audio and the character number sid corresponding to the template audio. For these two parameters, the content encoder Contentvec is first used to extract the 768-dimensional content feature F1 that is unrelated to the speaker's timbre from the voice data, and then the PitchExtractor (pitch enhancer) is used to extract the audio pitch information P1 from the voice data. In order to achieve the real-time reasoning here, the Pitch Extractor adopts the traditional pitch extraction algorithm pm. Then P1 and F1 are input into the text encoder TextEncoder for content information encoding, and after the weighted summation of the attention weight, the prior hidden variable z_p is obtained in the form of residual. At the same time, the character number sid of the template audio is input into the Embedding layer of the pytorch framework to obtain the speaker embedding vector Speaker Embedding, and then z_p and Speaker Embedding are input into the flow model Flow to obtain the latent variable z. The latent variable z and the speaker embedding vector Speaker Embedding are finally decoded into audio waveform data by the vocoder NSF-Hifigan.
[0044] Furthermore, in actual application scenarios, the audio reconstruction loss can be calculated for the decoded audio waveform data, so as to back-propagate and update the parameters to realize model training. In addition, the pitch guidance parameter pitch, feature index degree index_rate and Top1 retrieval operation are not required in the training phase.
[0045] In one embodiment, the step S101 further includes:
[0046] Obtaining the fundamental frequency of the template audio;
[0047] Down-sampling the voice data, and scaling the fundamental frequency of the down-sampled voice data according to the fundamental frequency of the template audio, so that the fundamental frequency of the voice data is consistent with the fundamental frequency of the template audio;
[0048] The speech data and template data with consistent fundamental frequency are input into the teacher model for distillation training.
[0049] In this embodiment, first, for the acquired template audio (according to product requirements, the speech audio of the specific template character is collected and processed), the fundamental frequency f0 is calculated, and then for the speech data of the basic training set, it is first downsampled to 16khz, and then the fundamental frequency of the speech data is calculated, and it is enlarged or reduced to f0, that is, the fundamental frequency of the template audio is consistent with the fundamental frequency of the speech data. The template data and the speech data are then input into the teacher model for voice change. In actual application scenarios, since the teacher model is trained with 48khz data, the voice change result needs to be downsampled to 16khz in the end, so that a pair of data that can be used for user distillation training can be obtained, namely the distilled data, which contains speech data and voice change results (labels of speech data).
[0050] In one embodiment, the step S102 includes:
[0051] Performing feature extraction on the distilled data through a causal convolution preprocessing layer to obtain corresponding audio features;
[0052] Feature encoding the audio features by using a dilated causal convolutional encoder to obtain a corresponding encoded representation;
[0053] Decoding and adjusting the encoded representation through a Transformer decoder to obtain a corresponding mask;
[0054] The mask is decoded through a one-dimensional transposed convolution layer and a Tanh activation function layer to obtain a speech waveform corresponding to the distilled data, and the speech waveform is used as the voice-changed audio.
[0055] Combination Figure 3 The student model described in this embodiment adopts an encoder-decoder architecture. Specifically, the audio features of the distilled data are first extracted through a causal convolution preprocessing layer, and then the extracted audio features are encoded through an expanded causal convolution encoder. The encoding is adjusted through a Transformer decoder to obtain a corresponding mask, and finally decoded through a one-dimensional transposed convolution layer and a Tanh activation function layer to obtain a speech waveform that can finally be used as a voice-changed audio.
[0056] This embodiment modifies the input audio by constructing a mask that is added to the input audio signal to isolate a sound present in the training set, such as an acoustic guitar, a cough, a gunshot, etc. It should be noted here that while the initial convolution of the encoder provides access to a small amount of context, the dilated causal convolution (DCC) in the encoder and the mask converter that only focuses on the current and past tokens in the decoder ensure that the model's reasoning is mainly based on past data. This makes the architecture well suited for streaming settings, where the need for future context introduces additional latency. In addition, the causal nature of the encoder and decoder allows intermediate calculations to be cached for future reasoning passes, which gives the network access to past context without the need for the entire context to run through every part of the network, thereby increasing the speed of reasoning.
[0057] In one embodiment, extracting features from the distilled data through a causal convolution preprocessing layer to obtain corresponding audio features includes:
[0058] The distilled data is subjected to feature extraction through continuous multi-layer residual modules, and the output result of the last layer of residual modules is input into the first one-dimensional convolution layer to obtain the audio feature; wherein each layer of the residual module includes two second one-dimensional convolution layers, and one of the second one-dimensional convolution layers is connected to the Tanh activation function layer, and the other second one-dimensional convolution layer is connected to the Sigmoid activation function layer, and the product of the output results of the Tanh activation function layer and the Sigmoid activation function layer is the output result of the residual module of the current layer.
[0059] Combination Figure 4 The causal convolution preprocessing layer (Causal Convolution Prenet) described in this embodiment is composed of multiple layers (for example, 12 layers) of residual blocks (Residual Blocks*12), each residual block includes two one-dimensional convolution layers, and at the same time, the two one-dimensional convolution layers are respectively activated by Tanh and Sigmoid, and the output results of the two activation functions are multiplied and output through the Dropout layer, which is used to extract the feature f from the current frame and the previous frame data, and f is added to the input x and then input to the next residual module. The result after n cycles (12 times) is input into a one-dimensional convolution Conv1D, thereby obtaining the audio feature y of the input audio. k , which is the input y of the subsequent dilated causal convolutional encoder k .
[0060] In one embodiment, the step of encoding the audio features by using a dilated causal convolutional encoder to obtain a corresponding encoded representation includes:
[0061] The audio features are spatially encoded through continuous multiple layers of depth-wise separable convolution, and the spatial encoding result of the last layer of depth-wise separable convolution is used as the encoded representation of the audio features; wherein each layer of depth-wise separable convolution is provided with a context buffer, and each of the depth-wise separable convolutions fills and expands its own output result through the context buffer.
[0062] Combination Figure 5 The dilated causal convolutional encoder (DCC Encoder) described in this embodiment is composed of multiple layers (e.g., 8 layers) of dilated depthwise separable convolutions, which are used to encode time domain audio input into latent space. In the depthwise separable convolution, for block-based streaming reasoning, the encoder maintains a context buffer for each DCC layer. In addition, each DCC layer inputs the output block of the previous layer, such as Figure 5 As shown, Figure 5 The left side of yk is padded with a context length twice the dilation of the layer. After encoding a block, the context is updated with the rightmost element of the padded input so that it can be used to encode the next block. For each input block yk, the DCC encoder computes an encoded representation ek.
[0063] In a specific embodiment, the length of the context buffer of each layer is calculated as follows:
[0064] (kerner_size-1)*2^i for iin range(num_layers);
[0065] Where kernel_size is the volume or kernel size of the depthwise separable convolution of DCC, which defaults to 3, num_layers is the number of depthwise separable convolution layers of DCC, which defaults to 8, and the expansion factor is {2^0, 2^1, 2^2, ..., 2^(8-1)}. Therefore, the length of the first-layer context buffer of DCC is 2, and the feature size of the entire buffer is [1, 512, 510], where 512 is determined by the output dimension of a Conv1D before DCC, and the calculation method of 510 is:
[0066] (kerner_size-1)*(2^num_layers-1) is (3-1)*(2^8-1)=510.
[0067] In one embodiment, the decoding and adjusting the encoded representation by a Transformer decoder to obtain a corresponding mask includes:
[0068] Obtaining an embedding tag of the distilled data, and embedding the embedding tag into the coded representation to obtain an embedded coded representation;
[0069] Projecting the coded representation and the embedded coded representation to the Transformer decoder dimension respectively using 1*1 convolution, and obtaining a projected coded representation and a projected embedded coded representation respectively;
[0070] Inputting the projected coded representation to a cross-attention layer of the Transformer decoder, and inputting the projected embedded coded representation to a self-attention layer of the Transformer decoder;
[0071] Inputting the output result of the self-attention layer into the cross-attention layer through the first feedforward neural network, and inputting the output result of the cross-attention layer into the second feedforward neural network, and using the output result of the second feedforward neural network as the initial mask;
[0072] Projecting the initial mask to the dimension of the dilated causal convolutional encoder using a 1*1 convolution to obtain a projected mask;
[0073] A skip connection is performed on the projection mask and the projection embedded coding representation, and a result of the skip connection is used as the mask.
[0074] Combination Figure 6 In order to obtain the mask mask, the encoded representation calculated by the DCC Encoder in this embodiment must be conditioned on the query q, and for this purpose, an embedding layer is used to implement the embedding label corresponding to q (with 1-dimensional input and 512-dimensional output). The Transformer decoder uses the query embedding label to adjust the encoded representation ek, and derives the mask mask from it. Specifically, a multiplication is first performed to calculate the conditional representation: ek0'=ek*label, where ek0' represents the encoded representation with speaker information, that is, the embedded encoded representation. At the same time, since the computational cost of the Transformer is very high, and the higher the dimension, the greater the computational cost, a 1*1 convolution is used to project the encoded representations ek0' and ek to the decoder dimension to obtain the projected encoded representation p_ek and the embedded projected encoded representation p_ek'. The decoded representation is then calculated by passing p_ek and p_ek' to the self-attention and cross-attention blocks of the Transformer decoder layer respectively to obtain the initial mask p_mk in the projected decoder space. Another 1*1 convolutional layer is then used to project it back to the encoder dimension to obtain the projected mask mask'. Since the bottleneck caused by the projection layer may affect the flow of gradients, a skip connection is used to the output of the projection layer immediately after the multiplication to calculate the final mask, i.e., mask = mask'+ek0'.
[0075] Figure 7 A schematic block diagram of a voice changing device 700 based on model distillation provided in an embodiment of the present invention, the device 700 includes:
[0076] A distillation training unit 701 is used to obtain a basic training set containing multiple speech data, and input the speech data in the basic training set and a preset template audio into a teacher model for distillation training to obtain a distillation data set containing distilled data;
[0077] A voice change training unit 702 is used to input the distilled data in the distilled data set into the student model for voice change training to obtain voice change audio, thereby constructing a real-time voice change model;
[0078] The real-time voice changing unit 703 is used to perform voice changing processing on the specified audio data using the real-time voice changing model.
[0079] In one embodiment, the distillation training unit 701 includes:
[0080] A content extraction unit, which extracts content features unrelated to the speaker's timbre from the speech data through a content encoder, and extracts pitch information from the speech data through a pitch extractor;
[0081] A content encoding unit, used for inputting the content features and pitch information into a text encoder to encode the content information, and performing attention weighted summation on the results of the content information encoding to obtain a priori latent variables;
[0082] A number embedding unit, used to obtain a character number corresponding to the model audio, and input the character number into an embedding layer to obtain a speaker embedding vector;
[0083] A stream model input unit, used for inputting the prior latent variable and the speaker embedding vector into the stream model to obtain a latent variable;
[0084] A label setting unit is used to input the latent variable and the speaker embedding vector into a vocoder for decoding to obtain audio waveform data, and use the audio waveform data as a label of the speech data, and then use the speech data and its label as the distilled data.
[0085] In one embodiment, the distillation training unit 701 further includes:
[0086] A fundamental frequency acquisition unit, used to acquire the fundamental frequency of the template audio;
[0087] A fundamental frequency scaling unit, used for down-sampling the voice data, and scaling the fundamental frequency of the down-sampled voice data according to the fundamental frequency of the template audio, so that the fundamental frequency of the voice data is consistent with the fundamental frequency of the template audio;
[0088] The data input unit is used to input speech data and template data with consistent fundamental frequency into the teacher model for distillation training.
[0089] In one embodiment, the voice changing training unit 702 includes:
[0090] A feature extraction unit, configured to extract features from the distilled data through a causal convolution preprocessing layer to obtain corresponding audio features;
[0091] A feature encoding unit, used for encoding the audio feature by a dilated causal convolutional encoder to obtain a corresponding encoding representation;
[0092] A decoding adjustment unit, configured to perform decoding adjustment on the encoded representation through a Transformer decoder to obtain a corresponding mask;
[0093] The mask decoding unit is used to decode the mask through a one-dimensional transposed convolution layer and a Tanh activation function layer to obtain a speech waveform corresponding to the distilled data, and use the speech waveform as the voice-changed audio.
[0094] In one embodiment, the feature extraction unit includes:
[0095] A residual extraction unit is used to extract features from the distilled data through continuous multi-layer residual modules, and input the output results of the last layer of residual modules to the first one-dimensional convolution layer to obtain the audio features; wherein each layer of the residual module includes two second one-dimensional convolution layers, and one of the second one-dimensional convolution layers is connected to the Tanh activation function layer, and the other second one-dimensional convolution layer is connected to the Sigmoid activation function layer, and the product of the output results of the Tanh activation function layer and the Sigmoid activation function layer is the output result of the residual module of the current layer.
[0096] In one embodiment, the feature encoding unit includes:
[0097] A spatial encoding unit is used to spatially encode the audio features through multiple layers of continuous depth-separable convolution, and use the spatial encoding result of the last layer of depth-separable convolution as the encoded representation of the audio features; wherein each layer of depth-separable convolution is provided with a context buffer, and each of the depth-separable convolutions fills and expands its own output result through the context buffer.
[0098] In one embodiment, the decoding adjustment unit includes:
[0099] a label embedding unit, configured to obtain an embedding label of the distilled data, and embed the embedding label into the coded representation to obtain an embedded coded representation;
[0100] A coding projection unit, used to project the coding representation and the embedded coding representation to the Transformer decoder dimension respectively by using 1*1 convolution, and obtain a projected coding representation and a projected embedded coding representation respectively;
[0101] A decoding input unit, configured to input the projected coded representation into a cross-attention layer of the Transformer decoder, and to input the projected embedded coded representation into a self-attention layer of the Transformer decoder;
[0102] A feedforward processing unit, configured to input the output result of the self-attention layer into the cross-attention layer through a first feedforward neural network, and input the output result of the cross-attention layer into a second feedforward neural network, and use the output result of the second feedforward neural network as an initial mask;
[0103] A mask projection unit, configured to project the initial mask to the dimension of the dilated causal convolutional encoder using a 1*1 convolution to obtain a projected mask;
[0104] A jump connection unit is used to perform a jump connection on the projection mask and the projection embedded coding representation, and use the jump connection result as the mask.
[0105] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.
[0106] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.
[0107] The embodiment of the present invention also provides a computer device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps provided in the above embodiment may be implemented. Of course, the computer device may also include various network interfaces, power supplies and other components.
[0108] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.
[0109] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
Claims
1. A voice changing method based on model distillation, characterized in that: include: Obtain a basic training set containing multiple voice data, and input the voice data in the basic training set and a preset template audio into a teacher model for distillation training to obtain a distilled data set containing distilled data; Inputting the distilled data in the distilled data set into the student model for voice change training to obtain voice change audio, thereby constructing a real-time voice change model; The real-time voice changing model is used to perform voice changing processing on the specified audio data.
2. The voice changing method based on model distillation according to claim 1, characterized in that: The speech data in the basic training set and the preset template audio are input into the teacher model for distillation training to obtain a distillation data set containing distilled data, including: Extracting content features unrelated to the speaker's timbre from the speech data by a content encoder, and extracting pitch information from the speech data by a pitch extractor; Inputting the content features and pitch information into a text encoder to encode the content information, and performing attention weighted summation on the results of the content information encoding to obtain a priori latent variables; Obtaining a character number corresponding to the model audio, and inputting the character number into an embedding layer to obtain a speaker embedding vector; Inputting the prior latent variable and the speaker embedding vector into a flow model to obtain a latent variable; The latent variable and the speaker embedding vector are input into a vocoder for decoding to obtain audio waveform data, and the audio waveform data is used as a label of the speech data, and then the speech data and its label are used as the distilled data.
3. The voice changing method based on model distillation according to claim 1, characterized in that: The method of inputting the speech data in the basic training set and the preset template audio into the teacher model for distillation training to obtain a distillation data set containing distilled data also includes: Obtaining the fundamental frequency of the template audio; Down-sampling the voice data, and scaling the fundamental frequency of the down-sampled voice data according to the fundamental frequency of the template audio, so that the fundamental frequency of the voice data is consistent with the fundamental frequency of the template audio; The speech data and template data with consistent fundamental frequency are input into the teacher model for distillation training.
4. The voice changing method based on model distillation according to claim 1, characterized in that: The step of inputting the distilled data in the distilled data set into the student model for voice change training to obtain voice change audio includes: Performing feature extraction on the distilled data through a causal convolution preprocessing layer to obtain corresponding audio features; Feature encoding the audio features by using a dilated causal convolutional encoder to obtain a corresponding encoded representation; Decoding and adjusting the encoded representation through a Transformer decoder to obtain a corresponding mask; The mask is decoded through a one-dimensional transposed convolution layer and a Tanh activation function layer to obtain a speech waveform corresponding to the distilled data, and the speech waveform is used as the voice-changed audio.
5. The voice changing method based on model distillation according to claim 4, characterized in that: The extracting features of the distilled data through the causal convolution preprocessing layer to obtain corresponding audio features includes: The distilled data is subjected to feature extraction through continuous multi-layer residual modules, and the output result of the last layer of residual modules is input into the first one-dimensional convolution layer to obtain the audio feature; wherein each layer of the residual module includes two second one-dimensional convolution layers, and one of the second one-dimensional convolution layers is connected to the Tanh activation function layer, and the other second one-dimensional convolution layer is connected to the Sigmoid activation function layer, and the product of the output results of the Tanh activation function layer and the Sigmoid activation function layer is the output result of the residual module of the current layer.
6. The voice changing method based on model distillation according to claim 4, characterized in that: The step of encoding the audio features by using the dilated causal convolutional encoder to obtain a corresponding encoding representation includes: The audio features are spatially encoded through continuous multiple layers of depth-wise separable convolution, and the spatial encoding result of the last layer of depth-wise separable convolution is used as the encoded representation of the audio features; wherein each layer of depth-wise separable convolution is provided with a context buffer, and each of the depth-wise separable convolutions fills and expands its own output result through the context buffer.
7. The voice changing method based on model distillation according to claim 4, characterized in that: The decoding and adjusting the encoded representation by the Transformer decoder to obtain a corresponding mask includes: Obtaining an embedding tag of the distilled data, and embedding the embedding tag into the coded representation to obtain an embedded coded representation; Projecting the coded representation and the embedded coded representation to the Transformer decoder dimension respectively using 1*1 convolution to obtain a projected coded representation and a projected embedded coded representation respectively; Inputting the projected coded representation to a cross-attention layer of the Transformer decoder, and inputting the projected embedded coded representation to a self-attention layer of the Transformer decoder; Inputting the output result of the self-attention layer into the cross-attention layer through the first feedforward neural network, and inputting the output result of the cross-attention layer into the second feedforward neural network, and using the output result of the second feedforward neural network as the initial mask; Projecting the initial mask to the dimension of the dilated causal convolutional encoder using a 1*1 convolution to obtain a projected mask; A skip connection is performed on the projection mask and the projection embedded coding representation, and a result of the skip connection is used as the mask.
8. A voice changing device based on model distillation, characterized in that: include: A distillation training unit, used to obtain a basic training set containing multiple voice data, and input the voice data in the basic training set and a preset template audio into a teacher model for distillation training to obtain a distillation data set containing distilled data; A voice change training unit, used for inputting the distilled data in the distilled data set into the student model for voice change training to obtain voice change audio, thereby constructing a real-time voice change model; The real-time voice changing unit is used to perform voice changing processing on the specified audio data using the real-time voice changing model.
9. A computer device, characterized in that: It comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the voice changing method based on model distillation as described in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the voice changing method based on model distillation as described in any one of claims 1 to 7 is implemented.
