Speech conversion method, device, equipment and storage medium

Abstract

The present application relates to artificial intelligence technology, and provides a method, device, equipment and computer-readable storage medium for speech conversion, the method comprising: obtaining a first speech; inputting the first speech into a first encoder of a trained speech conversion model to obtain a language expression feature vector output by the first encoder; determining a target expression feature vector based on the language expression feature vector based on a language expression feature library; inputting the first speech into a speech recognition submodel of the speech conversion model to obtain a text vector corresponding to the first speech output by the speech recognition submodel; vector integration of the target expression feature vector and the text vector to obtain a first integrated speech vector; inputting the first integrated speech vector into a speech generation submodel of the speech conversion model to obtain a second speech output by the speech generation submodel. It is possible to achieve speech conversion and improve the conversion rate. The present application also relates to blockchain technology, and the speech conversion model can be stored in the blockchain.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a speech conversion method, device, equipment and computer-readable storage medium. Background Art

[0002] At present, through speech synthesis models and voice cloning technology, the speech output by the speech synthesis model can be converted into the sound of a specific speaker to meet the requirements of personalized speech output, thereby achieving the purpose of simulating a person's speech. However, when the model's corpus training data is relatively small, the speech output by the speech synthesis model is not very similar to the real speech. In addition, the speech synthesis model needs to input text data into the model to generate specific audio based on the text. It is impossible to convert the speaker's speech into a specific audio that simulates another speaker in real time, and the audio conversion efficiency is slow. Summary of the invention

[0003] The main purpose of this application is to provide a speech conversion method, device, equipment and computer-readable storage medium, aiming to improve the speech conversion efficiency of the speech conversion model, and enhance the naturalness and authenticity of the speech generated by the speech conversion model.

[0004] In a first aspect, the present application provides a speech conversion method, the speech conversion method comprising the following steps: obtaining a first speech;

[0005] Inputting the first speech into a first encoder of a trained speech conversion model to obtain a speech expression feature vector output by the first encoder;

[0006] Based on the language expression feature library, determining a target expression feature vector according to the language expression feature vector;

[0007] Inputting the first speech into the speech recognition submodel of the speech conversion model to obtain a text vector corresponding to the first speech output by the speech recognition submodel;

[0008] Performing vector integration on the target expression feature vector and the text vector to obtain a first integrated speech vector;

[0009] The first integrated speech vector is input into the speech generation sub-model of the speech conversion model to obtain a second speech output by the speech generation sub-model.

[0010] In a second aspect, the present application further provides a speech conversion device, the speech conversion device comprising:

[0011] A first voice acquisition module, used to acquire a first voice;

[0012] A speech expression feature vector acquisition module, used for inputting the first speech into a first encoder of a trained speech conversion model to obtain a speech expression feature vector output by the first encoder;

[0013] A target expression feature vector determination module, used to determine a target expression feature vector according to the language expression feature vector based on a language expression feature library;

[0014] A text vector acquisition module, used for inputting the first speech into the speech recognition submodel of the speech conversion model to obtain a text vector corresponding to the first speech output by the speech recognition submodel;

[0015] A vector integration module, used for performing vector integration on the target expression feature vector and the text vector to obtain a first integrated speech vector;

[0016] The second speech generation module is used to input the first integrated speech vector into the speech generation sub-model of the speech conversion model to obtain the second speech output by the speech generation sub-model.

[0017] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the speech conversion method as described above are implemented.

[0018] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the speech conversion method as described above are implemented.

[0019] The present application provides a speech conversion method, apparatus, device and computer-readable storage medium. The present application obtains a first speech; inputs the first speech into a first encoder of a trained speech conversion model to obtain a speech expression feature vector output by the first encoder; determines a target expression feature vector based on the language expression feature vector based on a language expression feature library; inputs the first speech into a speech recognition submodel of the speech conversion model to obtain a text vector corresponding to the first speech output by the speech recognition submodel; performs vector integration on the target expression feature vector and the text vector to obtain a first integrated speech vector; inputs the first integrated speech vector into a speech generation submodel of the speech conversion model to obtain a second speech output by the speech generation submodel, and the first speech can be converted into a second speech different from the first speech expression mode through the first encoder, speech recognition submodel and speech generation submodel of the speech conversion model, so as to realize real-time speech conversion and effectively improve the speech conversion rate; and the language expression feature library can store expression features synthesized from expression features corresponding to multiple historical speech, which can effectively improve the naturalness and authenticity of speech generated by the speech conversion model in the case of less corpus training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A flowchart of a speech conversion method provided in an embodiment of the present application;

[0022] Figure 2 A schematic block diagram of a speech conversion model provided in one embodiment of the present application;

[0023] Figure 3 A schematic block diagram of a speech conversion model provided in another embodiment of the present application;

[0024] Figure 4 A schematic block diagram of a speech conversion device provided in an embodiment of the present application;

[0025] Figure 5 The present invention is a block diagram showing the structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0028] The embodiments of the present application provide a method, device, computer equipment and computer-readable storage medium for voice conversion. The voice conversion method can be applied to a terminal device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, etc. It can also be applied to a server, which can be a separate server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0029] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0030] Please refer to Figure 1 , Figure 1 A flowchart of a speech conversion method provided in an embodiment of the present application.

[0031] like Figure 1 As shown, the speech conversion method includes steps S101 to S106.

[0032] Step S101: Acquire a first voice.

[0033] Exemplarily, the first voice may be a voice obtained based on a recording of the speech of the first speaker, and the voiceprint, timbre, intonation, and sentence punctuation of the first speaker's voice may be adjusted to achieve conversion into the sound emitted by a specific speaker or expression based on the language expression habits of the specific speaker, thereby meeting the requirements of personalized voice output. In some usage scenarios, the privacy of the first speaker can also be effectively protected.

[0034] Exemplarily, the first voice can be acquired by a voice acquisition device. It can be understood that the voice acquisition device is communicatively connected to the terminal device or the server. After the voice acquisition device acquires the first voice, the first voice is sent to the terminal device or the server, and the first voice is processed in the terminal device or the server to adjust at least one of the intonation, tone, speaking speed, sentence punctuation position, and timbre of the first voice.

[0035] For example, the first voice can also be obtained through recording information. It can be understood that the recording information is the voice pre-recorded by the speaker, and the recording information can be processed to change the voice expression of the speaker in the recording information, so as to achieve the purpose of changing the voice expression and realize voice conversion. In some usage scenarios where voice samples need to be provided, the privacy of the speaker providing the voice sample can be effectively protected.

[0036] Step S102: input the first speech into a first encoder of a trained speech conversion model to obtain a speech expression feature vector output by the first encoder.

[0037] like Figure 2 As shown, Figure 2 A structural block diagram of a speech conversion model provided for one embodiment of the present application, wherein the speech conversion model includes a first encoder, a language expression feature library, a speech recognition sub-model and a speech generation sub-model.

[0038] Exemplarily, the speech conversion model can realize the conversion of speech, and the trained speech conversion model can be stored in the blockchain so that the terminal in need can call it.

[0039] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0040] Exemplarily, the first speech is input into a first encoder of a speech conversion model to extract a language expression feature vector from the first speech.

[0041] Exemplarily, the first encoder can encode the first speech to extract a language expression feature vector from the first speech. It can be understood that the language expression feature vector is used to indicate at least one of the speaker's tone, voice, speech speed, sentence punctuation position, and timbre when speaking the first speech. For example, the sentence punctuation position in the first speech is: "I / am a student" " / " indicates the position of the sentence punctuation during expression. After being processed by the speech conversion model, the output second speech can be "I am / a / student" to achieve the purpose of changing the language expression of the first speech.

[0042] Exemplarily, when performing encoding processing, the first encoder performs different encodings on different tones and different sentence positions in the first speech. For example, if the text has a second tone, it can be encoded as "2", and the symbol "-" can be added to the sentence position. The timbre can correspond to the letter encoding, for example, a. After obtaining the first speech, the language expression feature vector corresponding to the first speech is obtained.

[0043] Exemplarily, by processing the language expression feature vector corresponding to the first voice, the first voice can be changed, thereby achieving the purpose of voice conversion. For example, the language expression feature vector can be modified or the language expression feature vector corresponding to the first voice can be replaced to change the first voice.

[0044] Step S103: Based on the language expression feature library, determine the target expression feature vector according to the language expression feature vector.

[0045] Exemplarily, the target expression feature vector can be determined based on the language expression feature vector according to the language expression similarity set by the user. For example, if the similarity is set to not more than 20%, the language expression feature vector can be used to screen the language expression feature vector in the language expression feature library for expression feature vectors whose similarity with the language expression feature vector is not more than 20%, and the target expression feature vector can be determined from the screened language expression feature vectors.

[0046] Exemplarily, the similarity may be calculated as a ratio of a modulus of a vector difference between two vectors to a modulus of a language expression feature vector extracted from the first speech.

[0047] It is understandable that the language expression feature library contains expression features corresponding to multiple speakers or different expression features of the same speaker. The expression features may be language expression features obtained through historical speech and stored in the language expression feature library; or they may be language expression features synthesized from language expression features obtained from at least two historical speech. It is understandable that if the user wishes to convert the first speech into the second speech of a specified person, the expression features corresponding to the specified person may be searched in the language expression feature library to process the first speech; if the user only needs to protect the information of the speaker corresponding to the first speech, the synthesized language expression features may be searched in the language expression feature library to process the first speech.

[0048] Exemplarily, by synthesizing language expression features obtained from at least two historical speech samples, more language expression features can be effectively obtained when there are fewer speech samples, and the authenticity of the obtained language expression features is higher, thereby improving the similarity and naturalness of the speech obtained after speech synthesis and achieving the purpose of reducing dependence on speech samples.

[0049] In some embodiments, the language expression feature library includes multiple expression feature vectors, and determining the target expression feature vector based on the language expression feature library and the language expression feature vector includes: calculating the vector difference between the language expression feature vector and each of the expression feature vectors in the language expression feature library; determining a vector difference range in response to an input instruction; and determining an expression feature vector whose vector difference size is within the vector difference range as a target expression feature vector.

[0050] Exemplarily, all expression feature vectors in the language expression feature library are traversed through the language expression feature vector, wherein during the traversal, vector differences are calculated between the language expression feature vector and the expression feature vector to obtain the vector difference between the language expression feature vector and each expression feature vector in the language expression feature library.

[0051] Exemplarily, in response to an input instruction, a vector difference range is determined, and an expression feature vector whose vector difference size is within the vector difference range is determined as a target expression feature vector.

[0052] It can be understood that the vector difference is used to represent the degree of difference between the expression characteristics of the first speech and the expression characteristics of the second speech. The greater the difference between the first speech and the second speech, the greater the vector difference.

[0053] Exemplarily, the language expression feature vector also includes multiple phoneme vector elements. When the user needs to adjust the first speech more accurately, the target expression vector can be determined by adjusting each phoneme vector element of the language expression feature vector in the first speech, that is, determining the vector difference between each phoneme vector element in the language expression feature vector and each corresponding phoneme vector element in the expression feature vector, thereby achieving the purpose of accurately adjusting the first speech.

[0054] For example, in response to a speech rate adjustment instruction input by a user, the phoneme vector element corresponding to the speech rate in the expression feature vector of the first voice is determined, and the target expression feature vector is determined based on the language expression feature vector extracted from the first voice and the expression feature vector in the language expression feature library. For example, if the speech rate of the first voice is 240 syllables / minute, and it is expected to reach 300 syllables / minute after adjustment, in response to the input of 300 syllables / minute, the vector difference range of the phoneme vector element corresponding to the speech rate is determined to be greater than 60, and the vector difference with the speech rate phoneme vector element in the language expression feature vector is calculated in the language expression feature library, and the expression feature vector with a vector difference greater than 60 is determined as the target expression feature vector. It can be understood that the phoneme vector elements corresponding to other language expression features, such as tone, timbre, etc., can be implemented according to the above implementation process, which will not be described here. The above adjustment of the speech rate is an exemplary example, and the specific implementation method of the adjustment of the speech rate is not limited.

[0055] Step S104: input the first speech into the speech recognition sub-model of the speech conversion model to obtain a text vector corresponding to the first speech output by the speech recognition sub-model.

[0056] Exemplarily, the first speech is input into the speech recognition sub-model in the speech conversion model to perform speech recognition on the first speech to obtain a text vector corresponding to the first speech. It can be understood that the text vector is used to indicate the words spoken in the first speech.

[0057] In some embodiments, the speech recognition sub-model includes a convolution layer and a time series classification layer, and the step of inputting the first speech into the speech sub-model of the speech conversion model to obtain a text vector corresponding to the first speech output by the speech recognition sub-model includes: inputting the first speech into the convolution layer for feature extraction processing to obtain a speech vector of the first speech; and inputting the speech vector of the first speech into the time series classification layer for conversion processing to obtain a text vector corresponding to the first speech.

[0058] Exemplarily, the artificial intelligence-based network recognizes the speech to obtain a text vector corresponding to the first speech.

[0059] Exemplarily, the speech recognition submodel includes multiple batch-normalized convolutional layers, PW (pointwise) convolutional layers and connectionist temporal classification layers. The input first speech is first converted into a high-dimensional vector through a convolutional layer with batch normalization, wherein each convolutional layer is connected to a batch normalization layer and a linear rectifier. After the first speech enters the convolutional layer, vector feature extraction is performed to obtain a speech vector of the first speech, and then the data is batch normalized through a batch normalization layer connected to the convolutional layer. Specifically, the mean and variance of the speech vector are calculated in the batch normalization layer, the vector is normalized according to the mean and variance, the value of the input speech vector is updated according to the parameters of the batch normalization layer and the vector after normalization, thereby obtaining an updated speech vector, avoiding the problems of gradient vanishing and gradient exploding, the updated speech vector is input into a linear rectifier function (ReLU), the speech vector is output after linear rectification, and the speech vector output by the batch normalization convolutional layer is input into the PW (pointwise) convolutional layer.

[0060] Exemplarily, the size of the convolution kernel in the PW (pointwise) convolution layer is 1*1*M, where M is the number of channels of the previous layer. In the PW (pointwise) convolution layer, the speech vectors output from the convolution layer with batch normalization are weighted combined in the depth direction to obtain the output of the PW (pointwise) convolution layer. The number of PW (pointwise) convolution layers can be more than that of conventional convolution layers, and the computing speed is faster.

[0061] Exemplarily, after being processed by the PW (pointwise) convolution layer, it enters the connectionist temporal classification layer to generate the text corresponding to the first speech. The CTC processing, for example, defines the input time step as T, and the feature dimension at each time step as m, then x = (x1, x2, ..., xT), xt = (xt1, xt2, ..., xtm). Similarly, the time step of each output is T, which corresponds to the input one by one. The feature dimension of each time step output is recorded as n, and we can get y = (y1, y2, ..., yT), yt = (yt1, yt2, ..., ytn). Assuming that the first voice is English, there may be no characters in some positions in the text vector. Define "-" as a blank character and add it to the character set. Then the output dimension of each time step is 27, that is, the probability of 27 characters being output at this time step. According to these probabilities, select one element at each time step to get the output sequence. Define a B transformation to transform the output sequence into the real output (such as state in the following example), delete consecutive identical characters to 1, and delete the blank characters to get the text vector. For example, the output sequence is π = (--stta-t---e), and the word state is obtained after B transformation.

[0062] For example, for one of the real outputs, there are multiple output sequences that can be obtained through B transformation. If they are traversed one by one, there will be n to the power of T possibilities. In order to increase the calculation speed, the connectionist temporal classification layer (Connectionist Temporal Classification) can be calculated using a forward-backward algorithm.

[0063] The first speech is recognized by the speech recognition sub-model in the speech processing model to obtain the corresponding text vector, so that the second speech is generated through the target expression feature vector and the text vector, and through the temporal classification layer of the speech recognition sub-model, each syllable / word in the first speech can be effectively recognized, so that the target expression feature vector can integrate each syllable / word in the text, and the obtained second speech is closer to the real speech.

[0064] S105. Perform vector integration on the target expression feature vector and the text vector to obtain a first integrated speech vector.

[0065] Exemplarily, by performing vector integration on the target expression feature vector and the text vector, a first integrated vector can be obtained. The first integrated vector can be a text vector carrying language expression features. Speech synthesis is performed on the first integrated speech vector to generate a second speech.

[0066] In some embodiments, the speech conversion model also includes a first decoder, and the vector integration of the target expression feature vector and the text vector to obtain a first integrated speech vector includes: inputting the target expression feature vector and the text vector into the first decoder for vector alignment to obtain the aligned target expression feature vector and text vector; merging the aligned target expression feature vector and text vector to obtain the speech vector of the second speech.

[0067] Exemplarily, the first decoder is connected to the speech recognition sub-model and the language expression feature library. The first decoder obtains the text vector corresponding to the first speech output by the speech recognition sub-model, and obtains the target expression feature vector output by the language expression feature library, and performs vector integration on the text vector and the target expression feature vector. The vector integration can be, for example, vector merging to obtain a first integrated speech vector. It can be understood that the first integrated speech vector can be a text vector including language expression features. The first integrated speech vector is subjected to speech synthesis by an artificial intelligence-based network to generate a second speech.

[0068] Exemplarily, before vector integration of the target expression feature vector and the text vector, the target expression feature vector and the text vector are aligned. It can be understood that the alignment process is that each vector element in the text vector has a corresponding target expression feature vector, thereby improving the authenticity of the generated second speech.

[0069] S106. Input the first integrated speech vector into the speech generation sub-model of the speech conversion model to obtain a second speech output by the speech generation sub-model.

[0070] Exemplarily, the speech generation sub-model is connected to the first decoder, and is used to obtain the first integrated speech vector output by the first decoder, and perform speech synthesis on the first integrated speech vector to obtain the second speech and output it.

[0071] It is understandable that the second voice can be output to the target terminal, or played through a voice playback device. If the second voice is played through a voice playback device, the speaker's first voice can be converted in real time so that the audience can hear the second voice obtained after the conversion, which can effectively protect the speaker's privacy.

[0072] In some embodiments, the speech generation submodel includes a spectrogram prediction network and a vocoder, and the step of inputting the speech vector of the second speech into the speech generation submodel of the speech conversion model to obtain the second speech comprises: inputting the speech vector of the second speech into the spectrogram prediction network, performing feature prediction to obtain the Mel spectrum of the speech vector of the second speech; and inputting the Mel spectrum of the speech vector of the second speech into the vocoder, performing convolution processing to obtain the second speech.

[0073] Exemplarily, the spectrogram prediction network can predict the Mel spectrum of the second speech through the first integrated speech vector. It can be understood that the first integrated speech vector can obtain a spectrogram through a short-time Fourier transform (STFT), and the Mel spectrum is obtained after the spectrogram is processed by a Mel filter. Generally, the human ear is more sensitive to low-frequency tones, and is very dull at high frequencies. The Mel spectrum can analyze the details of the low frequency of the sound wave, which just corresponds to the objective law that the higher the frequency, the duller the human ear is. It is more suitable than the spectrogram size, which can effectively improve the processing speed of the computer.

[0074] Exemplarily, the spectrogram prediction network also includes a prenet layer, an LSTM layer, a projection layer and a post-net layer. The speech vector of the second voice is input into the second decoder. The decoding process starts with inputting the output speech vector of the previous step, and the output of PreNet and the context vector (context vector) calculated using the output of the previous decoding step are concatenated and passed through a 2-layer unidirectional LSTM layer to calculate a new context vector. The new context vector (context vector) is then concatenated with the output of the unidirectional LSTM layer and sent to the projection layer (linear projection) to predict the output, and the Mel spectrum corresponding to the speech vector of the second voice is obtained through linear transformation.

[0075] The output Mel spectrum is passed through a 5-layer convolutional post-net. The post-net predicts the residual and adds it to the prediction result to improve the overall effect. For example, each post-net is composed of 512 filters.

[0076] The Mel spectrum after post-net processing enters the vocoder, which is a wave network that can process the Mel spectrum through dilated convolution and finally output the second speech.

[0077] In some embodiments, the method further includes: acquiring training data, the training data including multiple third voices and texts corresponding to each of the third voices; training the voice conversion model according to the training data to obtain a trained voice conversion model.

[0078] Exemplarily, the speech conversion model can be trained through training data to adjust the parameters in the speech conversion model. It can be understood that the speech synthesized and output by the adjusted speech conversion model has higher authenticity, thereby improving the audience's auditory experience.

[0079] Exemplarily, multiple third voices and texts corresponding to each third voice may be obtained. The third voice may be, for example, a voice that has been historically input into the voice conversion model. After obtaining storage permission, the voice input into the voice conversion model is stored. The third voice may also be obtained from a server or a blockchain. It is understandable that when the third voice is obtained, the text of the third voice is obtained accordingly, so that the voice conversion model is trained through the third voice and the text corresponding to the third voice.

[0080] In some embodiments, the method also includes: inputting the third speech into the convolution layer of the speech recognition submodel to obtain a speech vector of the third speech output by the convolution layer; dividing the speech vector of the third speech into a preset number of frames, wherein each frame of speech vector includes multiple phoneme vector units, and the preset number is a natural number greater than 0; clustering the multiple phoneme vector units in each frame of speech vector to obtain multiple cluster centers; calculating the distance between each frame of speech vector and each of the cluster centers; and determining the weight of each frame of speech vector based on the distance from each frame of speech vector to each of the cluster centers.

[0081] Exemplarily, by inputting the third speech into the convolutional layer of the speech recognition sub-model, a speech vector of the third speech can be obtained, wherein the speech vector includes multiple phoneme vector elements. It can be understood that the phoneme vector elements can be used to indicate the expression characteristics of each word in the speech, such as tone, timbre, sound speed, and whether there is a pause.

[0082] Exemplarily, the speech vector of the third speech is segmented into a preset number of frames, for example, the speech vector of the third speech is segmented according to a preset time period. It can be understood that each frame of the speech vector obtained after the segmentation includes multiple phoneme vector units.

[0083] Exemplarily, multiple phoneme vector units of each frame of speech vector are clustered to obtain multiple cluster centers, for example, using x t represents a frame of speech vector, then x t =(x1, x2, ...x T ), x1 represents a phoneme vector element, T represents a sequence length, and is used to indicate the number of frames obtained by segmenting the speech vector of the third speech. Multiple clustering centers are obtained by clustering multiple phoneme vector elements.

[0084] Exemplarily, the distance from each frame speech vector to each cluster center is calculated, and the weight corresponding to each frame speech vector is determined by the distance from each cluster center. For example, a target cluster center can be set, and the distance from each frame speech vector to the target cluster center and the remaining cluster centers is calculated. The weight of the current frame speech vector is determined by the ratio of the distance from the current frame speech vector to the remaining cluster centers and the distance from the current frame speech vector to the target cluster center. For example, the distance from the A frame speech vector to the target cluster center is 10, and the sum of the distances to the remaining cluster centers is 5, and the weight of the A frame speech vector is determined to be 50%. It can be understood that the above method for determining the weight of the speech vector is an exemplary example, and does not limit the determination of the weight of each frame speech vector.

[0085] Exemplarily, after determining the weight of each frame of speech vector, the speech conversion model is trained based on the weight of each frame of speech vector, the speech vector of the third speech and the text, so that more attention is paid to the phoneme vector elements contained in a frame of speech vector with a high weight during training, thereby achieving better training effect.

[0086] In some embodiments, the speech conversion model also includes a second encoder, and the speech conversion model is trained according to the weight of the speech vector of each frame, the speech vector of the third speech and the text, including: inputting the text into the second encoder to obtain a text sequence vector; determining a target expression feature vector, and performing vector integration on the text sequence vector and the target expression feature vector to obtain a second integrated speech vector; inputting the second integrated speech vector, the speech vector of the third speech, and the weight corresponding to each frame in the speech vector of the third speech into the second decoder, and calculating the deviation value through a preset formula; adjusting the parameters of the speech conversion model according to the deviation value to obtain a trained speech conversion model.

[0087] For example, Figure 3 As shown, Figure 3 A schematic block diagram of a speech conversion model provided for another embodiment of the present application, wherein the speech conversion model also includes a second encoder, which is connected to the decoder for vectorizing the text. It is understandable that the figure does not show a structure for calculating the corresponding weight of each frame in the speech vector of the third speech. In the speech conversion model, there should be a structure that can calculate the corresponding weight of each frame in the speech vector of the third speech.

[0088] Exemplarily, a target expression feature vector is determined in a language expression feature library, and the target expression feature vector and the text sequence vector are integrated in a first decoder to obtain a second integrated speech vector. For example, the target expression feature vector can be determined in a language expression feature library by using the expected speech output result.

[0089] Exemplarily, the voice vector of the third voice is updated by applying the weight corresponding to each frame in the voice vector of the third voice to obtain an updated voice vector. The deviation value can be calculated by using the second integrated voice vector and the updated voice vector. If the deviation value is less than a preset difference, it indicates that the training of the voice conversion model is completed. If the deviation value is not less than the preset difference, the parameters of the voice conversion model are adjusted.

[0090] Exemplarily, the preset formula may be as follows:

[0091]

[0092] Among them, L(x,y) is the deviation value, w T is the weight corresponding to the speech vector of the Tth frame, y i is the second integrated speech vector, X i is the speech vector of the third speech, and n is the number of phoneme vector elements in the speech vector of the third speech.

[0093] After the deviation value is obtained by a preset formula, it is determined whether the deviation value is less than a preset difference value, wherein the preset difference value can be determined based on a speech expression feature vector extracted from the third speech and a target expression feature vector.

[0094] It can be understood that by calculating the deviation value and adjusting the speech conversion model, the authenticity and naturalness of the speech obtained after the speech conversion by the speech conversion model can be improved, and the speech can be closer to the expression habits of a specific speaker.

[0095] The speech conversion method provided in the above embodiment can convert a first speech into a second speech that is different from the first speech expression through the first encoder, speech recognition sub-model and speech generation sub-model of the speech conversion model, can realize the speech conversion in real time, and improve the speech conversion efficiency, and the language expression feature library can store expression features synthesized from expression features corresponding to multiple historical speeches, thereby ensuring the naturalness and authenticity of the speech generated by the speech conversion model with less corpus training data.

[0096] See also Figure 4 , Figure 4 It is a schematic diagram of a speech conversion device provided in one embodiment of the present application. The speech conversion device can be configured in a server or a terminal to execute the aforementioned speech conversion method.

[0097] like Figure 4As shown, the speech conversion device includes: a first speech acquisition module 110, a speech expression feature vector acquisition module 120, a target expression feature vector determination module 130, a text vector acquisition module 140, a vector integration module 150, and a second speech generation module 160.

[0098] The first voice acquisition module 110 is used to acquire a first voice.

[0099] The speech expression feature vector acquisition module 120 is used to input the first speech into the first encoder of the trained speech conversion model to obtain the speech expression feature vector output by the first encoder.

[0100] The target expression feature vector determination module 130 is used to determine the target expression feature vector according to the language expression feature vector based on the language expression feature library.

[0101] The text vector acquisition module 140 is used to input the first speech into the speech recognition sub-model of the speech conversion model to obtain the text vector corresponding to the first speech output by the speech recognition sub-model.

[0102] The vector integration module 150 is used to perform vector integration on the target expression feature vector and the text vector to obtain a first integrated speech vector.

[0103] The second speech generation module 160 is used to input the first integrated speech vector into the speech generation sub-model of the speech conversion model to obtain the second speech output by the speech generation sub-model.

[0104] Exemplarily, the target expression feature vector determination module 130 further includes a vector difference calculation submodule and a vector difference range determination submodule.

[0105] The vector difference calculation submodule is used to calculate the vector difference between the language expression feature vector and each of the expression feature vectors in the language expression feature library.

[0106] The vector difference range determination submodule is used to determine the vector difference range in response to an input instruction.

[0107] The target expression feature vector determination module 130 is further configured to determine an expression feature vector whose vector difference is equal to a preset vector difference threshold as a target expression feature vector.

[0108] Exemplarily, the vector integration module 150 further includes a decoder submodule and a vector merging submodule.

[0109] The decoder submodule is used to input the target expression feature vector and the text vector into a first decoder for vector alignment to obtain the aligned target expression feature vector and text vector.

[0110] The vector merging submodule is used to merge the aligned target expression feature vector and the text vector to obtain the speech vector of the second speech.

[0111] Exemplarily, the text vector acquisition module 140 also includes a convolution layer submodule and a temporal classification layer submodule.

[0112] The convolutional layer submodule is used to input the first speech into the convolutional layer for feature extraction processing to obtain a speech vector of the first speech.

[0113] The time series classification layer submodule is used to input the speech vector of the first speech into the time series classification layer for conversion processing to obtain a text vector corresponding to the first speech.

[0114] Exemplarily, the speech conversion device also includes a training data acquisition module and a model training module.

[0115] The training data acquisition module is used to acquire training data, wherein the training data includes a plurality of third voices and texts corresponding to each of the third voices.

[0116] The model training module is used to train the speech conversion model according to the training data to obtain a trained speech conversion model.

[0117] Exemplarily, the speech conversion device also includes a speech vector determination module for the third speech, a speech vector segmentation module, a cluster center determination module, a cluster center distance determination module, and a weight determination module.

[0118] The speech vector determination module of the third speech is used to input the third speech into the convolution layer of the speech recognition sub-model to obtain the speech vector of the third speech output by the convolution layer.

[0119] A speech vector segmentation module is used to segment the speech vector of the third speech into a preset number of frames, wherein each frame of the speech vector includes a plurality of phoneme vector units, and the preset number is a natural number greater than 0.

[0120] The cluster center determination module is used to cluster multiple phoneme vector units in each frame of speech vector to obtain multiple cluster centers.

[0121] The cluster center distance determination module is used to calculate the distance between each frame speech vector and each cluster center.

[0122] The weight determination module is used to determine the weight of each frame of speech vector according to the distance from each frame of speech vector to each cluster center.

[0123] The model training module is also used to train the speech conversion model according to the weight of each frame of speech vector, the speech vector of the third speech and the text.

[0124] Exemplarily, the model training module also includes a second encoder submodule, a second integrated speech vector determination submodule, and a model parameter adjustment submodule.

[0125] The second encoder submodule is used to input the text into the second encoder to obtain a text sequence vector.

[0126] The second integrated speech vector determination submodule is used to determine a target expression feature vector, perform vector integration on the text sequence vector and the target expression feature vector, and obtain a second integrated speech vector.

[0127] The decoder submodule is also used to input the second integrated speech vector, the speech vector of the third speech and the weight corresponding to each frame in the speech vector of the third speech into the second decoder, and calculate the deviation value through a preset formula.

[0128] The model parameter adjustment submodule is used to adjust the parameters of the speech conversion model according to the deviation value to obtain a trained speech conversion model.

[0129] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0130] The method of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0131] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in Figure 5 Runs on the computer device shown.

[0132] See also Figure 5 , Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a server or a terminal.

[0133] like Figure 5 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and an internal memory.

[0134] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any speech conversion method.

[0135] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0136] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any speech conversion method.

[0137] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0138] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0139] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0140] Get the first voice;

[0141] Inputting the first speech into a first encoder of a trained speech-to-speech conversion model to obtain a language expression feature vector output by the first encoder;

[0142] Based on the language expression feature library, determining a target expression feature vector according to the language expression feature vector;

[0143] Inputting the first speech into the speech recognition submodel of the speech conversion model to obtain a text vector corresponding to the first speech output by the speech recognition submodel;

[0144] Performing vector integration on the target expression feature vector and the text vector to obtain a first integrated speech vector;

[0145] The first integrated speech vector is input into the speech generation sub-model of the speech conversion model to obtain a second speech output by the speech generation sub-model.

[0146] In one embodiment, when the processor determines the target expression feature vector according to the language expression feature vector based on the language expression feature library, the processor is used to implement:

[0147] Calculating a vector difference between the language expression feature vector and each of the expression feature vectors in the language expression feature library;

[0148] In response to an input instruction, determining a vector difference range;

[0149] An expression feature vector whose vector difference size is within the vector difference range is determined as a target expression feature vector.

[0150] In one embodiment, when the processor integrates the target expression feature vector and the text vector to obtain the first integrated speech vector, it is used to implement:

[0151] Inputting the target expression feature vector and the text vector into the decoder of the speech conversion model for vector alignment to obtain the aligned target expression feature vector and text vector;

[0152] The aligned target expression feature vector and the text vector are merged to obtain the speech vector of the second speech.

[0153] In one embodiment, when the processor implements inputting the first speech into the speech recognition submodel of the speech conversion model to obtain the text vector corresponding to the first speech output by the speech recognition submodel, it is used to implement:

[0154] Inputting the first speech into the convolution layer of the speech recognition sub-model for feature extraction processing to obtain a speech vector of the first speech;

[0155] The speech vector of the first speech is input into the temporal classification layer of the speech recognition sub-model for conversion processing to obtain a text vector corresponding to the first speech.

[0156] In one embodiment, when implementing the speech conversion method, the processor is used to implement:

[0157] Acquire training data, the training data comprising a plurality of third voices and texts corresponding to the third voices;

[0158] The speech conversion model is trained according to the training data to obtain a trained speech conversion model.

[0159] In one embodiment, when implementing the speech conversion method, the processor is used to implement:

[0160] Inputting the third speech into the convolution layer of the speech recognition sub-model to obtain a speech vector of the third speech output by the convolution layer;

[0161] Dividing the speech vector of the third speech into a preset number of frames, wherein each frame of the speech vector includes a plurality of phoneme vector units, and the preset number is a natural number greater than 0;

[0162] Clustering multiple phoneme vector units in each frame of speech vector to obtain multiple cluster centers;

[0163] Calculating the distance between each frame speech vector and each cluster center;

[0164] Determine the weight of each frame of speech vector according to the distance from each frame of speech vector to each of the cluster centers;

[0165] When the processor trains the speech conversion model according to the training data to obtain the trained speech conversion model, the processor is used to implement:

[0166] The speech conversion model is trained according to the weight of each frame of speech vector, the speech vector of the third speech and the text.

[0167] In one embodiment, when the processor implements training the speech conversion model according to the weight of each frame speech vector, the speech vector of the third speech and the text, it is used to implement:

[0168] Inputting the text into a second encoder of the speech conversion model to obtain a text sequence vector;

[0169] Determine a target expression feature vector, and perform vector integration on the text sequence vector and the target expression feature vector to obtain a second integrated speech vector;

[0170] Input the second integrated speech vector, the speech vector of the third speech, and the weight corresponding to each frame in the speech vector of the third speech into the decoder of the speech conversion model, and calculate the deviation value by a preset formula;

[0171] The parameters of the speech conversion model are adjusted according to the deviation value to obtain a trained speech conversion model.

[0172] It should be noted that technical personnel in the relevant field can clearly understand that for the convenience and brevity of description, the specific working process of the voice conversion described above can refer to the corresponding process in the aforementioned voice conversion control method embodiment, and will not be repeated here.

[0173] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the speech conversion method of the present application.

[0174] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0175] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0176] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0177] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A speech conversion method, characterized in that: include: Get the first voice; Inputting the first speech into a first encoder of a trained speech-to-speech conversion model, obtaining a language expression feature vector output by the first encoder, wherein the language expression feature vector is used to indicate at least one of the tone, speech speed, sentence segmentation position, and timbre of the speaker when speaking the first speech; Based on the language expression feature library, determining a target expression feature vector according to the language expression feature vector; Inputting the first speech into the speech recognition submodel of the speech conversion model to obtain a text vector corresponding to the first speech output by the speech recognition submodel; Performing vector integration on the target expression feature vector and the text vector to obtain a first integrated speech vector; Inputting the first integrated speech vector into the speech generation sub-model of the speech conversion model to obtain a second speech output by the speech generation sub-model; Wherein, the method further comprises: Acquire training data, the training data comprising a plurality of third voices and texts corresponding to the third voices; Inputting the third speech into the convolution layer of the speech recognition sub-model to obtain a speech vector of the third speech output by the convolution layer; Dividing the speech vector of the third speech into a preset number of frames, wherein each frame of the speech vector includes a plurality of phoneme vector units, and the preset number is a natural number greater than 0; Clustering multiple phoneme vector units in each frame of speech vector to obtain multiple cluster centers; Calculating the distance between each frame speech vector and each cluster center; Determine the weight of each frame of speech vector according to the distance from each frame of speech vector to each of the cluster centers; Inputting the text into a second encoder of the speech conversion model to obtain a text sequence vector; Determine a target expression feature vector, and perform vector integration on the text sequence vector and the target expression feature vector to obtain a second integrated speech vector; Input the second integrated speech vector, the speech vector of the third speech, and the weight corresponding to each frame in the speech vector of the third speech into the decoder of the speech conversion model, and calculate the deviation value by a preset formula; The parameters of the speech conversion model are adjusted according to the deviation value to obtain a trained speech conversion model.

2. The speech conversion method according to claim 1, characterized in that: The language expression feature library includes a plurality of expression feature vectors, and the step of determining a target expression feature vector based on the language expression feature library and the language expression feature vectors includes: Calculating a vector difference between the language expression feature vector and each of the expression feature vectors in the language expression feature library; In response to an input instruction, determining a vector difference range; An expression feature vector whose vector difference size is within the vector difference range is determined as a target expression feature vector.

3. The speech conversion method according to claim 1, wherein: The step of performing vector integration on the target expression feature vector and the text vector to obtain a first integrated speech vector includes: Inputting the target expression feature vector and the text vector into the decoder of the speech conversion model for vector alignment to obtain the aligned target expression feature vector and text vector; The aligned target expression feature vector and the text vector are merged to obtain the speech vector of the second speech.

4. The speech conversion method according to any one of claims 1 to 3, characterized in that: The step of inputting the first speech into the speech recognition submodel of the speech conversion model to obtain a text vector corresponding to the first speech output by the speech recognition submodel includes: Inputting the first speech into the convolution layer of the speech recognition sub-model for feature extraction processing to obtain a speech vector of the first speech; The speech vector of the first speech is input into the temporal classification layer of the speech recognition sub-model for conversion processing to obtain a text vector corresponding to the first speech.

5. A speech conversion device, characterized in that: The speech conversion device comprises: A first voice acquisition module, used to acquire a first voice; a language expression feature vector acquisition module, configured to input the first speech into a first encoder of a trained speech conversion model, and obtain a language expression feature vector output by the first encoder, wherein the language expression feature vector is used to indicate at least one of the tone, speech speed, sentence segmentation position and timbre of the speaker when speaking the first speech; A target expression feature vector determination module, used to determine a target expression feature vector according to the language expression feature vector based on a language expression feature library; A text vector acquisition module, used for inputting the first speech into the speech recognition submodel of the speech conversion model to obtain a text vector corresponding to the first speech output by the speech recognition submodel; A vector integration module, used for performing vector integration on the target expression feature vector and the text vector to obtain a first integrated speech vector; A second speech generation module, used for inputting the first integrated speech vector into the speech generation sub-model of the speech conversion model to obtain a second speech output by the speech generation sub-model; The speech conversion device also includes a training data acquisition module, a speech vector determination module for the third speech, a speech vector segmentation module, a cluster center determination module, a cluster center distance determination module, a weight determination module, a second encoder submodule, a second integrated speech vector determination submodule, a decoder submodule and a model parameter adjustment submodule; The training data acquisition module is used to acquire training data, wherein the training data includes a plurality of third voices and texts corresponding to each of the third voices; The speech vector determination module of the third speech is used to input the third speech into the convolution layer of the speech recognition sub-model to obtain the speech vector of the third speech output by the convolution layer; The speech vector segmentation module is used to segment the speech vector of the third speech into a preset number of frames, wherein each frame of the speech vector includes a plurality of phoneme vector units, and the preset number is a natural number greater than 0; The cluster center determination module is used to cluster multiple phoneme vector units in each frame of speech vector to obtain multiple cluster centers; The cluster center distance determination module is used to calculate the distance between each frame speech vector and each cluster center; The weight determination module is used to determine the weight of each frame of speech vector according to the distance from each frame of speech vector to each cluster center; The second encoder submodule is used to input the text into the second encoder to obtain a text sequence vector; The second integrated speech vector determination submodule is used to determine a target expression feature vector, and perform vector integration on the text sequence vector and the target expression feature vector to obtain a second integrated speech vector; The decoder submodule is further used to input the second integrated speech vector, the speech vector of the third speech, and the weight corresponding to each frame in the speech vector of the third speech into the decoder of the speech conversion model, and calculate the deviation value by a preset formula; The model parameter adjustment submodule is used to adjust the parameters of the speech conversion model according to the deviation value to obtain a trained speech conversion model.

6. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the speech conversion method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the speech conversion method according to any one of claims 1 to 4 are implemented.

Images