Speech conversion method, device, equipment and storage medium based on quantization coding

Abstract

The present application discloses a speech conversion method, device, equipment and storage medium based on quantization coding, which belongs to the field of artificial intelligence technology. The present application extracts a first content feature, a first identity feature and a first acoustic feature from the first speech data, and then imports the above features into a pre-trained speech conversion model respectively, and encodes the above features in sequence through the encoder of the speech conversion model to obtain a feature vector, and vector quantizes the first content feature vector therein to obtain a first vector vector, and performs feature mapping on the first vector vector and the feature vector to generate a first Mel spectrogram, and decodes the first Mel spectrogram through a decoder to obtain second speech data. The present application also relates to the field of blockchain technology, and the above speech data can be stored in a blockchain network. The present application realizes non-parallel speech conversion processing by vector quantizing the content feature vector, thereby improving the accuracy of speech conversion.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a speech conversion method, device, equipment and storage medium based on quantization coding. Background Art

[0002] Voice conversion (VC) is to convert the speech of a source speaker into the speech of a target speaker. However, the content of the speech cannot be changed during the conversion process. In order to imitate the target speaker, the VC system achieves voice conversion by modifying the tone, accent, and voice of the source speaker.

[0003] Traditionally, VC systems focus on using parallel data for one-to-one or many-to-one speech conversion and regard it as a statistical problem. However, the above method requires a large amount of parallel data, that is, collecting a large amount of speech data of source speakers and target speakers, and requiring the speech data of the source speakers to be completely consistent with the speech data of the target speakers, and such parallel data is difficult to collect. Many scholars have tried to implement VC tasks based on non-parallel data, for example, the currently popular deep style transfer algorithm, generative adversarial network (GAN) and conditional variational autoencoder (CVAE), etc. However, the current non-parallel VC task processing method also has certain problems, that is, it tends to ignore the correlation between different speech representations during the training phase, which leads to conversion errors such as skipping words, missing words, and repetitions during speech conversion. The accuracy of speech conversion is not high, which will reduce the performance of the VC system. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a speech conversion method, device, computer equipment and storage medium based on quantization coding, so as to solve the technical problem that the existing speech conversion schemes are prone to conversion errors such as word skipping, word missing, repetition, etc., resulting in low accuracy of speech conversion.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a voice conversion method based on quantization coding, which adopts the following technical solution:

[0006] A speech conversion method based on quantization coding, comprising:

[0007] Acquire first voice data, and perform feature extraction on the first voice data to obtain a first content feature, a first identity feature, and a first acoustic feature;

[0008] Importing the first content feature, the first identity feature, and the first acoustic feature into a pre-trained speech conversion model, respectively, wherein the speech conversion model includes an encoder, a feature mapper, and a decoder;

[0009] The first content feature, the first identity feature and the first acoustic feature are respectively encoded by an encoder to obtain a first content feature vector, a first identity feature vector and a first acoustic feature vector;

[0010] Performing vector quantization on the first content feature vector to obtain a first vector;

[0011] Performing feature mapping on the first vector vector, the first identity feature vector, and the first acoustic feature vector in a feature mapper to generate a first mel-spectrogram;

[0012] The first mel-spectrogram is decoded by a decoder to obtain second speech data.

[0013] Furthermore, the step of performing vector quantization on the first content feature vector to obtain a first vector vector specifically includes:

[0014] Dividing the first content feature vector to obtain first scalar data;

[0015] Combining the first scalar data to obtain a first initial vector vector;

[0016] The first initial vector vector is mapped to the vector space to generate a first vector vector.

[0017] Furthermore, before the step of respectively importing the first content feature, the first identity feature and the first acoustic feature into the pre-trained speech conversion model, the method further includes:

[0018] Acquiring third voice data, and performing feature extraction on the third voice data to obtain a third content feature, a third identity feature, and a third acoustic feature;

[0019] importing the third content feature, the third identity feature and the third acoustic feature into the initial conversion model respectively, wherein the initial conversion model includes an encoding layer, a feature mapping layer and a decoding layer;

[0020] The third content feature, the third identity feature and the third acoustic feature are respectively encoded by the encoding layer to obtain a third content feature vector, a third identity feature vector and a third acoustic feature vector;

[0021] Performing vector quantization on the third content feature vector to obtain a second vector;

[0022] Performing feature mapping on the second vector, the third identity feature vector, and the third acoustic feature vector in a feature mapping layer to generate a second mel-spectrogram;

[0023] The initial conversion model is iteratively updated based on the second mel-spectrogram to obtain a trained speech conversion model.

[0024] Further, the initial conversion model is iteratively updated based on the second mel-spectrogram to obtain a trained speech conversion model, specifically comprising:

[0025] Build the loss function of the initial conversion model;

[0026] Calculate the loss value of the initial conversion model based on the loss function and the second mel-spectrogram;

[0027] Based on the loss value, the back propagation algorithm is used to iteratively update the initial conversion model to obtain a trained speech conversion model.

[0028] Further, the step of calculating the loss value of the initial conversion model based on the loss function and the second mel-spectrogram specifically includes:

[0029] Performing feature mapping on the third content feature vector, the third identity feature vector, and the third acoustic feature vector in a feature mapping layer to generate a third mel-spectrogram;

[0030] The loss error between the second mel-spectrogram and the third mel-spectrogram is calculated based on the loss function to obtain the loss value of the initial conversion model.

[0031] Further, based on the loss value, the initial conversion model is iteratively updated using a back propagation algorithm to obtain a trained speech conversion model, specifically including:

[0032] Compare the loss value with the preset loss threshold;

[0033] When the loss value is greater than the preset loss threshold, the initial conversion model is iteratively updated through the back propagation algorithm until the loss value is less than or equal to the preset loss threshold, thereby obtaining a trained speech conversion model.

[0034] Furthermore, the step of performing vector quantization on the third content feature vector to obtain a second vector specifically includes:

[0035] Dividing the second content feature vector to obtain second scalar data;

[0036] Combining the second scalar data to obtain a second initial vector vector;

[0037] The second initial vector vector is mapped to the vector space to generate a second vector vector.

[0038] In order to solve the above technical problems, the embodiment of the present application further provides a speech conversion device based on quantization coding, which adopts the following technical solution:

[0039] A speech conversion device based on quantization coding, comprising:

[0040] A first feature extraction module, used to obtain first voice data and perform feature extraction on the first voice data to obtain a first content feature, a first identity feature and a first acoustic feature;

[0041] A first feature import module, used to import the first content feature, the first identity feature and the first acoustic feature into a pre-trained speech conversion model, wherein the speech conversion model includes an encoder, a feature mapper and a decoder;

[0042] A first feature encoding module, used to encode the first content feature, the first identity feature and the first acoustic feature respectively through an encoder to obtain a first content feature vector, a first identity feature vector and a first acoustic feature vector;

[0043] A first vector quantization module, used for performing vector quantization on the first content feature vector to obtain a first vector;

[0044] A first feature mapping module, configured to perform feature mapping on the first vector, the first identity feature vector and the first acoustic feature vector in a feature mapper to generate a first mel-spectrogram;

[0045] The spectrum decoding module is used to decode the first mel-spectrogram through a decoder to obtain second speech data.

[0046] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0047] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the speech conversion method based on quantization coding as described in any one of the above are implemented.

[0048] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0049] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech conversion method based on quantization coding as described in any one of the above are implemented.

[0050] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0051] The present application discloses a speech conversion method, device, equipment and storage medium based on quantization coding, which belongs to the field of artificial intelligence technology. The present application extracts a first content feature, a first identity feature and a first acoustic feature from the first speech data, and then imports the above features into a pre-trained speech conversion model respectively, and encodes the above features in sequence through the encoder of the speech conversion model to obtain a feature vector, and vector quantizes the first content feature vector to obtain a first vector vector, and performs feature mapping on the first vector vector, the first identity feature vector and the first acoustic feature vector to generate a first Mel spectrogram, and decodes the first Mel spectrogram through a decoder to obtain the second speech data. The present application realizes non-parallel speech conversion processing by vector quantizing the content feature vector generated by encoding, while ensuring the correlation between different speech representations, reducing the occurrence of skipped words, missing words, repetitions, etc. during speech conversion, and improving the accuracy of speech conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 An exemplary system architecture diagram is shown in which the present application can be applied;

[0054] Figure 2 A flowchart of an embodiment of a voice conversion method based on quantization coding according to the present application is shown;

[0055] Figure 3 A schematic structural diagram of an embodiment of a speech conversion device based on quantization coding according to the present application is shown;

[0056] Figure 4 A schematic structural diagram of an embodiment of a computer device according to the present application is shown. DETAILED DESCRIPTION

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0058] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0059] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0060] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0061] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0062] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0063] Server 105 can be a server that provides various services, such as a background server that provides support for pages displayed on terminal devices 101, 102, and 103. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0064] It should be noted that the voice conversion method based on quantization coding provided in the embodiment of the present application is generally executed by a server, and accordingly, the voice conversion device based on quantization coding is generally arranged in the server.

[0065] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0066] Continue to refer Figure 2 , shows a flow chart of an embodiment of a speech conversion method based on quantization coding according to the present application. The embodiment of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0067] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The speech conversion method based on quantization coding includes the following steps:

[0068] S201, obtaining first voice data, and performing feature extraction on the first voice data to obtain a first content feature, a first identity feature, and a first acoustic feature.

[0069] Specifically, the first voice data is the voice of the source speaker. After receiving the voice conversion instruction, the server obtains the first voice data and performs feature extraction on the first voice data to obtain a first content feature, a first identity feature and a first acoustic feature, wherein the first content feature represents the voice content, the first identity feature represents the identity of the source speaker, and the first acoustic feature represents the sound parameters in the voice, such as intonation, speaking speed, etc.

[0070] In the above embodiment, by performing feature extraction on the first speech data respectively, the first content feature, the first identity feature and the first acoustic feature in the first speech data are obtained, so as to facilitate subsequent specific processing operations on different features, for example, vector quantization processing of the first content feature.

[0071] In this embodiment, the electronic device (eg Figure 1 The server shown in the figure) can receive the voice conversion instruction through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0072] S202, importing the first content feature, the first identity feature and the first acoustic feature into a pre-trained speech conversion model respectively, wherein the speech conversion model includes an encoder, a feature mapper and a decoder.

[0073] Specifically, the server will import the first content feature, the first identity feature and the first acoustic feature into a pre-trained speech conversion model respectively, wherein the speech conversion model includes an encoder, a feature mapper and a decoder, the encoder is used to encode the input features to obtain a feature vector, and the feature mapper maps the feature vector to a specific space to form a Mel-spectrogram, intercepts the Mel-spectrogram, and puts the Mel-spectrogram into the decoder for voice-to-code conversion, so that the speech can be restored.

[0074] It should be noted that the encoder can be composed of a stack of LSTM networks or a combination of LSTM layers and fully connected layers. The decoder has the opposite structure to the encoder. The feature mapper includes an attention layer and a mapping layer. The attention layer of the feature mapper is used to classify the input feature vector, and the mapping layer maps the classified feature vector to a specific space. It is understandable that the encoder, decoder and feature mapper can also be composed of other convolutional neural networks, recurrent convolutional neural networks, etc., which are not specifically limited in this embodiment.

[0075] S203 , respectively encode the first content feature, the first identity feature, and the first acoustic feature through an encoder to obtain a first content feature vector, a first identity feature vector, and a first acoustic feature vector.

[0076] Specifically, the encoder in the specific embodiment of the present application includes at least three components, namely a content encoding unit, an identity encoding unit and an acoustic encoding unit. The content encoding unit is used to encode content features, the identity encoding unit is used to encode identity features, and the acoustic encoding unit encodes acoustic features. The server encodes the first content feature through the content encoding unit to obtain a first content feature vector, encodes the first identity feature through the identity encoding unit to obtain a first identity feature vector, and encodes the first acoustic feature through the acoustic encoding unit to obtain a first acoustic feature vector.

[0077] S204: Perform vector quantization on the first content feature vector to obtain a first vector.

[0078] Among them, vector quantization (VQ) is a data compression technology. Its basic idea is to group several scalar data into a vector, and then quantize it as a whole in the vector space, thereby compressing the data without losing much information. Vector quantization is the mapping from N-dimensional real space RN to L discrete vectors in RN, which can also be called group quantization. Scalar quantization is a special case of vector quantization when the dimension is 1. Vector quantization coding is also a new type of quantization coding method that has been studied more in image and speech signal coding technology. Its appearance is not only proposed as a quantizer design, but more of a compression coding method. In traditional prediction and transform coding, the signal is first transformed into a sequence of numbers through a certain mapping transformation, and then scalar quantization coding is performed one by one. In vector quantization coding, the input data is divided into many groups in groups, and quantized and encoded in groups, that is, these numbers are regarded as a k-dimensional vector, and then quantized one by one in units of vectors.

[0079] Specifically, in order to realize non-parallel speech conversion processing, it is necessary to vector quantize the content feature vector. In a specific embodiment of the present application, the content encoding unit uses a VQ vector network (VQCPC) including two networks, h-net and g-net, and after encoding the content features, performs a vector quantization operation, striving to extract the content information of the language bottom layer from the speech data, and extracts local features (such as phonemes) across multiple time steps to be encoded into a content representation for accurately reconstructing the language content, wherein the content encoding unit outputs E k , after VQ vector quantization, the output is

[0080] Furthermore, the step of performing vector quantization on the first content feature vector to obtain a first vector vector specifically includes:

[0081] Dividing the first content feature vector to obtain first scalar data;

[0082] Combining the first scalar data to obtain a first initial vector vector;

[0083] The first initial vector vector is mapped to the vector space to generate a first vector vector.

[0084] Specifically, the server divides the first content feature vector based on a preset data division principle to obtain a number of first scalar data, and then combines the first scalar data to obtain multiple k-dimensional first initial vector vectors, maps the multiple k-dimensional first initial vector vectors to the same vector space, and generates a first vector vector.

[0085] In the above embodiment, a k-dimensional first initial vector vector is constructed by a plurality of scalar data groups, and then the k-dimensional first initial vector vector is quantized as a whole in the vector space, thereby compressing the data without losing much information.

[0086] S205 , performing feature mapping on the first vector, the first identity feature vector, and the first acoustic feature vector in a feature mapper to generate a first mel-spectrogram.

[0087] Specifically, the feature mapper maps the first vector, the first identity feature vector and the first acoustic feature vector to the same feature space to generate a first mel-spectrogram. It should be noted that the feature vectors are feature reconstructed in the feature space, and a new mel-spectrogram is formed after the mapping is completed. By decoding the mel-spectrogram, the speech can be restored.

[0088] S206: Decode the first mel-spectrogram through a decoder to obtain second speech data.

[0089] Specifically, the first mel-spectrogram is decoded by a decoder to obtain second voice data, which is the voice of the target speaker. At this point, the server completes the voice conversion from the first voice data to the second voice data, that is, converts the voice of the source speaker into the voice of the target speaker.

[0090] In the above embodiment, the present application discloses a speech conversion method based on quantization coding, which belongs to the field of artificial intelligence technology. The present application extracts a first content feature, a first identity feature and a first acoustic feature from the first speech data, and then imports the above features into a pre-trained speech conversion model respectively, and encodes the above features in sequence through the encoder of the speech conversion model to obtain a feature vector, and vector quantizes the first content feature vector to obtain a first vector vector, and performs feature mapping on the first vector vector, the first identity feature vector and the first acoustic feature vector to generate a first Mel spectrogram, and decodes the first Mel spectrogram through a decoder to obtain the second speech data. The present application realizes non-parallel speech conversion processing by vector quantizing the content feature vector generated by encoding, while ensuring the correlation between different speech representations, reducing the occurrence of skipped words, missing words, repetitions, etc. during speech conversion, and improving the accuracy of speech conversion.

[0091] Furthermore, before the first content feature, the first identity feature, and the first acoustic feature are respectively imported into the pre-trained speech conversion model to complete the speech conversion step, the speech conversion model needs to be pre-trained. The speech conversion model is specifically trained by the following steps:

[0092] Acquiring third voice data, and performing feature extraction on the third voice data to obtain a third content feature, a third identity feature, and a third acoustic feature;

[0093] importing the third content feature, the third identity feature and the third acoustic feature into the initial conversion model respectively, wherein the initial conversion model includes an encoding layer, a feature mapping layer and a decoding layer;

[0094] The third content feature, the third identity feature and the third acoustic feature are respectively encoded by the encoding layer to obtain a third content feature vector, a third identity feature vector and a third acoustic feature vector;

[0095] Performing vector quantization on the third content feature vector to obtain a second vector;

[0096] Performing feature mapping on the second vector, the third identity feature vector, and the third acoustic feature vector in a feature mapping layer to generate a second mel-spectrogram;

[0097] The initial conversion model is iteratively updated based on the second mel-spectrogram to obtain a trained speech conversion model.

[0098] Among them, the third voice data is the historical voice data collected in advance, and the third voice data is used to train the voice conversion model. The server first constructs an initial conversion model, wherein the initial conversion model can be composed of a convolutional neural network, a recurrent convolutional neural network or other machine learning models, which is not specifically limited in this embodiment. The initial conversion model includes a coding layer, a feature mapping layer and a decoding layer, which correspond to the encoder, feature mapper and decoder of the voice conversion model respectively. The server encodes the features of the third voice data through the coding layer to obtain a third content feature vector, a third identity feature vector and a third acoustic feature vector, further performs vector quantization on the third content feature vector to obtain a second vector vector, and performs feature mapping on the second vector vector, the third identity feature vector and the third acoustic feature vector through the feature mapping layer to generate a second mel spectrogram, which is used to determine whether the initial conversion model is fitted.

[0099] Further, the initial conversion model is iteratively updated based on the second mel-spectrogram to obtain a trained speech conversion model, specifically comprising:

[0100] Build the loss function of the initial conversion model;

[0101] Calculate the loss value of the initial conversion model based on the loss function and the second mel-spectrogram;

[0102] Based on the loss value, the back propagation algorithm is used to iteratively update the initial conversion model to obtain a trained speech conversion model.

[0103] In a specific embodiment of the present application, the content feature vector, identity feature vector and acoustic feature vector can be constrained by cosine similarity to avoid distortion of the training results. The server first constructs the initial conversion model loss function, which includes a cosine loss function, a reconstruction loss function and a vector quantization loss function. The specific expression is as follows:

[0104] L=L VQ +L REC +λL COS

[0105] Where L is the loss function of the initial conversion model, L VQ is the vector quantization loss function of the content encoder, L REC To reconstruct the loss function, L COS Cosine loss function, λ is the control L COS The weight of the loss function L, where:

[0106]

[0107] In the formula, K is the total amount of speech data used for model training, T is the number of frames of any speech data, is the predicted output value, x t is the actual output value.

[0108]

[0109] In the formula, E k,t is the output result of the speech data of frame number t after passing through the content coding unit, For E k The output result after VQ vector quantization, sg(*) is the stop gradient calculation symbol to stop the quantized Gradient calculation of .

[0110]

[0111] Where V is a vector vector Identity feature vector s k and acoustic eigenvector p k The feature vector collection is a and b are any two eigenvectors in V.

[0112] Further, the step of calculating the loss value of the initial conversion model based on the loss function and the second mel-spectrogram specifically includes:

[0113] Performing feature mapping on the third content feature vector, the third identity feature vector, and the third acoustic feature vector in a feature mapping layer to generate a third mel-spectrogram;

[0114] The loss error between the second mel-spectrogram and the third mel-spectrogram is calculated based on the loss function to obtain the loss value of the initial conversion model.

[0115] Specifically, before performing loss calculation, the server first performs feature mapping on the third content feature vector, the third identity feature vector and the third acoustic feature vector in the feature mapping layer to generate a Mel-spectrogram of the third speech data, i.e., the third Mel-spectrogram, and calculates the loss error between the second Mel-spectrogram and the third Mel-spectrogram based on the loss function to obtain the loss value of the initial conversion model.

[0116] In a specific embodiment of the present application, assuming that there are K utterances, we use the Mel spectrogram as the acoustic feature and randomly select T frames from each utterance for model training. The initial Mel spectrogram is represented as:

[0117] X k =X k,1 ,X k,2 ,...,X k,T

[0118] After vector quantization and feature reconstruction, the new Mel spectrogram is represented as:

[0119]

[0120] The loss function is used to calculate the loss error between the initial mel-spectrogram and the new mel-spectrogram to obtain the loss value of the initial conversion model. The initial conversion model is iterated using the loss value until the model is fitted to obtain a trained speech conversion model. The trained speech conversion model can be directly used for speech conversion tasks.

[0121] Further, based on the loss value, the initial conversion model is iteratively updated using a back propagation algorithm to obtain a trained speech conversion model, specifically including:

[0122] Compare the loss value with the preset loss threshold;

[0123] When the loss value is greater than the preset loss threshold, the initial conversion model is iteratively updated through the back propagation algorithm until the loss value is less than or equal to the preset loss threshold, thereby obtaining a trained speech conversion model.

[0124] Among them, the BP algorithm (i.e., the error back propagation algorithm) is a learning algorithm suitable for multi-layer neural networks, which is based on the gradient descent method. The input-output relationship of the BP network is essentially a mapping relationship: the function performed by a BP neural network with n inputs and m outputs is a continuous mapping from n-dimensional Euclidean space to a finite field in m-dimensional Euclidean space, and this mapping is highly nonlinear. Its information processing ability comes from the multiple compounding of simple nonlinear functions, so it has a strong function reproduction ability. The learning process of the BP algorithm consists of a forward propagation process and a backward propagation process. In the forward propagation process, the input information passes through the input layer and the hidden layer, and is processed layer by layer and transmitted to the output layer. If the expected output value is not obtained in the output layer, the sum of the squares of the error between the output and the expected value is taken as the objective function, and the back propagation is turned on. The partial derivatives of the objective function to each neuron weight are obtained layer by layer, and the gradient of the objective function to the weight vector is formed as the basis for modifying the weight. The learning of the network is completed in the weight modification process. When the error reaches the expected value, the network learning ends.

[0125] Specifically, the server compares the loss value with the preset loss threshold. When the loss value is greater than the preset loss threshold, the server iteratively updates the initial conversion model through the back propagation algorithm, and continuously adjusts the network parameters of the initial conversion model until the loss value is less than or equal to the preset loss threshold. At this time, the initial conversion model approaches fitting and obtains a trained speech conversion model.

[0126] Furthermore, the step of performing vector quantization on the third content feature vector to obtain a second vector specifically includes:

[0127] Dividing the second content feature vector to obtain second scalar data;

[0128] Combining the second scalar data to obtain a second initial vector vector;

[0129] The second initial vector vector is mapped to the vector space to generate a second vector vector.

[0130] It should be noted that in order to achieve non-parallel speech conversion processing, it is necessary to vector quantize the content vector of the training sample when training the model. In a specific embodiment of the present application, the content encoding unit uses a VQ vector network (VQCPC) including two networks, h-net and g-net. After encoding the content features, it performs a vector quantization operation to extract the content information at the bottom of the language from the speech data. The local features (such as phonemes) extracted across multiple time steps are encoded into the content representation for accurately reconstructing the language content, where the content encoding unit outputs E k , after VQ vector quantization, the output is

[0131] Specifically, the server divides the second content feature vector based on a preset data division principle to obtain a number of second scalar data, and then combines the second scalar data to obtain multiple k-dimensional second initial vector vectors, and maps the multiple k-dimensional second initial vector vectors to the same two vector spaces to generate a second vector vector.

[0132] In the above embodiment, scalar data is obtained by dividing the content feature vectors of the training samples, and a k-dimensional initial vector is formed by the scalar data group. Then, the k-dimensional initial vector is quantized as a whole in the vector space, thereby compressing the data without losing much information.

[0133] In the above embodiment, before the voice conversion step, it is also necessary to pre-train a voice conversion model, obtain third voice data as a training sample, and perform feature extraction on the third voice data to obtain third content features, third identity features and third acoustic features, and import the third content features, third identity features and third acoustic features into the initial conversion model respectively, wherein the initial conversion model includes a coding layer, a feature mapping layer and a decoding layer, and the third content features, third identity features and third acoustic features are encoded by the coding layer respectively to obtain a third content feature vector, a third identity feature vector and a third acoustic feature vector, and the third content feature vector is vector quantized to obtain a second vector vector, and feature mapping is performed on the second vector vector, the third identity feature vector and the third acoustic feature vector in the feature mapping layer to generate a second Mel-spectrogram, and the initial conversion model is iteratively updated based on the second Mel-spectrogram to obtain a trained voice conversion model, and the trained voice conversion model can be directly used for non-parallel voice conversion tasks, while ensuring the correlation between different voice representations, reducing word skipping, word missing, repetition and the like during voice conversion, and improving the accuracy of voice conversion.

[0134] It should be emphasized that in order to further ensure the privacy and security of the above voice data, the above voice data can also be stored in a blockchain node.

[0135] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0136] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0137] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0138] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a speech conversion device based on quantization coding, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0139] like Figure 3 As shown, the speech conversion device based on quantization coding described in this embodiment includes:

[0140] A first feature extraction module 301 is used to obtain first speech data and perform feature extraction on the first speech data to obtain a first content feature, a first identity feature and a first acoustic feature;

[0141] A first feature importing module 302, used to import the first content feature, the first identity feature and the first acoustic feature into a pre-trained speech conversion model, wherein the speech conversion model includes an encoder, a feature mapper and a decoder;

[0142] A first feature encoding module 303, configured to encode the first content feature, the first identity feature and the first acoustic feature respectively through an encoder to obtain a first content feature vector, a first identity feature vector and a first acoustic feature vector;

[0143] A first vector quantization module 304, configured to perform vector quantization on the first content feature vector to obtain a first vector;

[0144] A first feature mapping module 305 is used to perform feature mapping on the first vector vector, the first identity feature vector and the first acoustic feature vector in a feature mapper to generate a first mel-spectrogram;

[0145] The spectrogram decoding module 306 is used to decode the first mel-spectrogram through a decoder to obtain second speech data.

[0146] Furthermore, the first vector quantization module 304 specifically includes:

[0147] A first vector division unit, used for dividing the first content feature vector to obtain first scalar data;

[0148] A first scalar combining unit, used for combining the first scalar data to obtain a first initial vector vector;

[0149] The first vector mapping unit is used to map the first initial vector vector to the vector space to generate a first vector vector.

[0150] Furthermore, the speech conversion device based on quantization coding also includes:

[0151] A second feature extraction module, used to obtain third voice data and perform feature extraction on the third voice data to obtain a third content feature, a third identity feature and a third acoustic feature;

[0152] A second feature importing module, used to import the third content feature, the third identity feature and the third acoustic feature into the initial conversion model respectively, wherein the initial conversion model includes a coding layer, a feature mapping layer and a decoding layer;

[0153] A second feature encoding module, used to encode the third content feature, the third identity feature and the third acoustic feature respectively through the encoding layer to obtain a third content feature vector, a third identity feature vector and a third acoustic feature vector;

[0154] A second vector quantization module, used for performing vector quantization on the third content feature vector to obtain a second vector;

[0155] A second feature mapping module, used for performing feature mapping on the second vector, the third identity feature vector and the third acoustic feature vector in a feature mapping layer to generate a second mel-spectrogram;

[0156] The model iteration module is used to iteratively update the initial conversion model based on the second mel-spectrogram to obtain a trained speech conversion model.

[0157] Furthermore, the model iteration module specifically includes:

[0158] A loss function building unit, used to build the loss function of the initial conversion model;

[0159] A loss calculation unit, used for calculating a loss value of the initial conversion model based on the loss function and the second mel-spectrogram;

[0160] The model iteration unit is used to iteratively update the initial conversion model based on the loss value by using the back propagation algorithm to obtain a trained speech conversion model.

[0161] Furthermore, the loss calculation unit specifically includes:

[0162] A third feature mapping subunit is used to perform feature mapping on the third content feature vector, the third identity feature vector and the third acoustic feature vector in the feature mapping layer to generate a third mel-spectrogram;

[0163] The loss calculation subunit is used to calculate the loss error between the second mel-spectrogram and the third mel-spectrogram based on the loss function to obtain the loss value of the initial conversion model.

[0164] Furthermore, the model iteration unit specifically includes:

[0165] A loss comparison subunit, used to compare the loss value with a preset loss threshold;

[0166] The model iteration subunit is used to iteratively update the initial conversion model through the back propagation algorithm when the loss value is greater than the preset loss threshold, until the loss value is less than or equal to the preset loss threshold, so as to obtain a trained speech conversion model.

[0167] Furthermore, the second vector quantization module specifically includes:

[0168] A second vector division unit, used for dividing the second content feature vector to obtain second scalar data;

[0169] A second scalar combining unit, used for combining the second scalar data to obtain a second initial vector vector;

[0170] The second vector mapping unit is used to map the second initial vector vector to the vector space to generate a second vector vector.

[0171] The present application discloses a speech conversion device based on quantization coding, which belongs to the field of artificial intelligence technology. The present application extracts a first content feature, a first identity feature and a first acoustic feature from the first speech data, and then imports the above features into a pre-trained speech conversion model respectively, and encodes the above features in sequence through the encoder of the speech conversion model to obtain a feature vector, and vector quantizes the first content feature vector to obtain a first vector vector, and performs feature mapping on the first vector vector, the first identity feature vector and the first acoustic feature vector to generate a first Mel spectrogram, and decodes the first Mel spectrogram through a decoder to obtain second speech data. The present application realizes non-parallel speech conversion processing by vector quantizing the content feature vector generated by encoding, while ensuring the correlation between different speech representations, reducing the occurrence of skipped words, missing words, repetitions, etc. during speech conversion, and improving the accuracy of speech conversion.

[0172] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0173] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0174] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0175] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a speech conversion method based on quantization coding, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0176] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the speech conversion method based on quantization coding.

[0177] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0178] The present application discloses a computer device, which belongs to the field of artificial intelligence technology. The present application extracts a first content feature, a first identity feature and a first acoustic feature from the first voice data, and then imports the above features into a pre-trained voice conversion model respectively, and encodes the above features in sequence through the encoder of the voice conversion model to obtain a feature vector, and vector quantizes the first content feature vector to obtain a first vector vector, and performs feature mapping on the first vector vector, the first identity feature vector and the first acoustic feature vector to generate a first Mel spectrogram, and decodes the first Mel spectrogram through a decoder to obtain second voice data. The present application realizes non-parallel voice conversion processing by vector quantizing the content feature vector generated by encoding, while ensuring the correlation between different voice representations, reducing the occurrence of skipped words, missing words, repetitions, etc. during voice conversion, and improving the accuracy of voice conversion.

[0179] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the speech conversion method based on quantization coding as described above.

[0180] The present application discloses a storage medium, which belongs to the field of artificial intelligence technology. The present application extracts a first content feature, a first identity feature, and a first acoustic feature from the first voice data, and then imports the above features into a pre-trained voice conversion model, and encodes the above features in sequence through the encoder of the voice conversion model to obtain a feature vector, and vector quantizes the first content feature vector to obtain a first vector vector, and performs feature mapping on the first vector vector, the first identity feature vector, and the first acoustic feature vector to generate a first Mel spectrogram, and decodes the first Mel spectrogram through a decoder to obtain the second voice data. The present application realizes non-parallel voice conversion processing by vector quantizing the content feature vector generated by encoding, while ensuring the correlation between different voice representations, reducing the occurrence of skipped words, missing words, repetitions, etc. during voice conversion, and improving the accuracy of voice conversion.

[0181] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0182] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0183] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A speech conversion method based on quantization coding, characterized in that: include: Acquire first voice data, and perform feature extraction on the first voice data to obtain a first content feature, a first identity feature, and a first acoustic feature; Importing the first content feature, the first identity feature, and the first acoustic feature into a pre-trained speech conversion model, respectively, wherein the speech conversion model includes an encoder, a feature mapper, and a decoder, the encoder is formed by stacking LSTM networks or by combining LSTM layers and fully connected layers, the decoder has the opposite structure to the encoder, the feature mapper includes an attention layer and a mapping layer, the attention layer of the feature mapper is used to classify the input feature vector, and the mapping layer maps the classified feature vector to a specific space; Encoding the first content feature, the first identity feature and the first acoustic feature respectively by the encoder to obtain a first content feature vector, a first identity feature vector and a first acoustic feature vector; Performing vector quantization on the first content feature vector to obtain a first vector; Performing feature mapping on the first vector, the first identity feature vector, and the first acoustic feature vector in the feature mapper to generate a first mel-spectrogram; The first mel-spectrogram is decoded by the decoder to obtain second speech data.

2. The voice conversion method based on quantization coding as claimed in claim 1, characterized in that: The step of performing vector quantization on the first content feature vector to obtain a first vector specifically includes: Dividing the first content feature vector to obtain first scalar data; Combining the first scalar data to obtain a first initial vector vector; The first initial vector vector is mapped to a vector space to generate the first vector vector.

3. The voice conversion method based on quantization coding as claimed in claim 1, characterized in that: Before the step of respectively importing the first content feature, the first identity feature and the first acoustic feature into the pre-trained speech conversion model, the method further includes: Acquiring third voice data, and performing feature extraction on the third voice data to obtain a third content feature, a third identity feature, and a third acoustic feature; importing the third content feature, the third identity feature and the third acoustic feature into an initial conversion model respectively, wherein the initial conversion model comprises an encoding layer, a feature mapping layer and a decoding layer; The third content feature, the third identity feature and the third acoustic feature are respectively encoded by the encoding layer to obtain a third content feature vector, a third identity feature vector and a third acoustic feature vector; Performing vector quantization on the third content feature vector to obtain a second vector; Performing feature mapping on the second vector, the third identity feature vector and the third acoustic feature vector in the feature mapping layer to generate a second mel-spectrogram; The initial conversion model is iteratively updated based on the second mel-spectrogram to obtain the trained speech conversion model.

4. The voice conversion method based on quantization coding as claimed in claim 3, characterized in that: The step of iteratively updating the initial conversion model based on the second mel-spectrogram to obtain the trained speech conversion model specifically includes: Constructing a loss function of the initial conversion model; Calculating a loss value of the initial conversion model based on the loss function and the second mel-spectrogram; Based on the loss value, the initial conversion model is iteratively updated using a back-propagation algorithm to obtain the trained speech conversion model.

5. The voice conversion method based on quantization coding as claimed in claim 4, characterized in that: The step of calculating the loss value of the initial conversion model based on the loss function and the second mel-spectrogram specifically includes: Performing feature mapping on the third content feature vector, the third identity feature vector, and the third acoustic feature vector in the feature mapping layer to generate a third mel-spectrogram; The loss error between the second mel-spectrogram and the third mel-spectrogram is calculated based on the loss function to obtain a loss value of the initial conversion model.

6. The voice conversion method based on quantization coding as claimed in claim 4, characterized in that: The step of iteratively updating the initial conversion model based on the loss value using a back propagation algorithm to obtain the trained speech conversion model specifically includes: Comparing the loss value with a preset loss threshold; When the loss value is greater than the preset loss threshold, the initial conversion model is iteratively updated through the back propagation algorithm until the loss value is less than or equal to the preset loss threshold, thereby obtaining the trained speech conversion model.

7. The voice conversion method based on quantization coding as claimed in claim 3, characterized in that: The step of performing vector quantization on the third content feature vector to obtain a second vector specifically includes: Dividing the third content feature vector to obtain second scalar data; Combining the second scalar data to obtain a second initial vector vector; The second initial vector vector is mapped to a vector space to generate the second vector vector.

8. A speech conversion device based on quantization coding, characterized in that: include: A first feature extraction module, used to obtain first voice data, and perform feature extraction on the first voice data to obtain a first content feature, a first identity feature, and a first acoustic feature; A first feature import module, used to import the first content feature, the first identity feature and the first acoustic feature into a pre-trained speech conversion model, respectively, wherein the speech conversion model includes an encoder, a feature mapper and a decoder, the encoder is formed by stacking LSTM networks or by combining LSTM layers and fully connected layers, the decoder has the opposite structure to the encoder, the feature mapper includes an attention layer and a mapping layer, the attention layer of the feature mapper is used to classify the input feature vector, and the mapping layer maps the classified feature vector to a specific space; a first feature encoding module, configured to encode the first content feature, the first identity feature, and the first acoustic feature respectively through the encoder to obtain a first content feature vector, a first identity feature vector, and a first acoustic feature vector; A first vector quantization module, configured to perform vector quantization on the first content feature vector to obtain a first vector; a first feature mapping module, configured to perform feature mapping on the first vector, the first identity feature vector and the first acoustic feature vector in the feature mapper to generate a first mel-spectrogram; The spectrum decoding module is used to decode the first mel-spectrogram through the decoder to obtain second speech data.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech conversion method based on quantization coding as claimed in any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech conversion method based on quantization coding as described in any one of claims 1 to 7 are implemented.

Images