Voice style conversion method and device, equipment and storage medium

Abstract

The invention discloses a voice style conversion method, and the method comprises the steps: obtaining a voice data set which comprises a source voice and a target voice; performing feature extraction on the source voice and the target voice to obtain a first Mel spectrum and a second Mel spectrum; performing content feature extraction on the first Mel spectrum through a preset content encoder to obtain content features; performing down-sampling processing on the phoneme sequence aligned with the content features to obtain a hidden state sequence; performing style feature extraction on the second Mel spectrum through a preset style encoder to obtain style features; performing alignment embedding on the style features and the hidden state sequence to obtain target style features; and determining a voice conversion result according to the hidden state sequence and the target style feature. According to the method, the phoneme sequence aligned with the content features is subjected to down-sampling, so that the problem that noise frames exist is solved. By performing alignment embedding on the style characteristics and the hidden state sequence, other styles except timbre are effectively transferred.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a speech style conversion method, device, equipment and storage medium. Background Art

[0002] With the continuous development of science and technology, voice conversion technology has gradually been applied in different fields. For example, in voice changing systems or voice chats. Voice conversion refers to converting one person's voice into another person's voice without changing the content of the speech. In other words, given the audio of a certain speaker, a voice with the characteristics of the speaker is generated for any text sequence.

[0003] In the prior art, speech conversion is often achieved by decoupling and reorganizing speech components. Decoupling is to split speech into audio components of multiple dimensions, but pitch, content and style features are often still mixed, and speech cannot be completely decoupled. Reorganization means that given any style features, the pitch content features can be converted into speech of any style. This method generally uses a time aggregation method to convert frame-level features into a feature vector of a fixed length, and then uses a fully connected layer to map the speech-level features to speaker embedding. However, style conversion at the frame level is easily affected by noise frames, resulting in poor speech style conversion results. Summary of the invention

[0004] The embodiments of the present invention provide a speech style conversion method, apparatus, device and storage medium to solve the problem of poor effect of speech style conversion in the prior art.

[0005] A speech style conversion method, comprising: Acquire a speech data set, wherein the speech data set includes at least one source speech and a target speech corresponding to the source speech; Performing spectrum feature extraction on all the source speech and all the target speech to obtain a first Mel spectrum corresponding to the source speech and a second Mel spectrum corresponding to the target speech; Extracting content features from the first Mel-spectrogram using a preset content encoder to obtain content features corresponding to the source speech; Downsampling the phoneme sequence aligned with the content feature to obtain a hidden state sequence; Extracting style features from the second Mel-spectrogram using a preset style encoder to obtain style features corresponding to the target speech; Aligning and embedding the style feature and the hidden state sequence to obtain a target style feature; A speech conversion result is determined according to the hidden state sequence and the target style feature.

[0006] A speech style conversion device, comprising: A speech acquisition module, used to acquire a speech data set, wherein the speech data set includes at least one source speech and a target speech corresponding to the source speech; A spectrum extraction module, used for extracting spectrum features of all the source speech and all the target speech to obtain a first Mel spectrum corresponding to the source speech and a second Mel spectrum corresponding to the target speech; A content feature module, configured to extract content features from the first Mel-spectrogram through a preset content encoder to obtain content features corresponding to the source speech; A downsampling module, used for downsampling the phoneme sequence aligned with the content feature to obtain a hidden state sequence; A style feature module, configured to extract style features from the second Mel spectrum using a preset style encoder to obtain style features corresponding to the target speech; An alignment embedding module, used for aligning and embedding the style feature and the hidden state sequence to obtain a target style feature; The conversion result module is used to determine the speech conversion result according to the hidden state sequence and the target style feature.

[0007] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned speech style conversion method when executing the computer program.

[0008] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for converting speech style is implemented.

[0009] The present invention provides a speech style conversion method, device, equipment and storage medium. The method realizes the acquisition of a first Mel spectrum of the source speech and the acquisition of a second Mel spectrum of the target speech by performing spectrum conversion on all the acquired source speech and all the target speech. The content feature of the first Mel spectrum is extracted by a preset content encoder, so as to realize the extraction of the content feature in the source speech. Then, by downsampling the phoneme sequence aligned with the content feature, the conversion of the phoneme sequence in the content feature is realized. The style feature of the second Mel spectrum is extracted by a preset style encoder, so as to realize the extraction of the style feature of the target speech. The style feature and the hidden state sequence are aligned and embedded, so as to realize the alignment of the local features in the target speech. According to the hidden state sequence and the target style feature, the style conversion of the source speech is realized, so as to realize the acquisition of the speech conversion result. Furthermore, by downsampling the phoneme sequence aligned with the content feature, the hidden state extraction representation of each frame of the phoneme sequence is realized, so as to solve the problem of the existence of noise frames. By aligning and embedding the style feature and the hidden state sequence, the other styles except the timbre are effectively transferred, so as to reduce the influence of the noise frame and the timbre, and improve the conversion effect of the speech style. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0011] Figure 1 is a schematic diagram of an application environment of a method for voice style conversion in one embodiment of the present invention; Figure 2 is a flow chart of a method for voice style conversion in one embodiment of the present invention; Figure 3 is a flowchart of step S40 of the method for voice style conversion in one embodiment of the present invention; Figure 4 is a flow chart of step S60 of a method for voice style conversion in one embodiment of the present invention; Figure 5 is a principle block diagram of a speech style conversion device in one embodiment of the present invention; Figure 6 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] The speech style conversion method provided by the embodiment of the present invention can be applied as follows: Figure 1 Specifically, the speech style conversion method is applied in a speech style conversion device, and the speech style conversion device includes Figure 1 The client and server shown in the figure communicate with each other through the network to solve the problem of poor effect of speech style conversion in the prior art. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The client, also known as the user end, refers to a program that corresponds to the server and provides classification services to customers. The client can be installed on, but not limited to, various computers, laptops, smartphones, tablets, and portable wearable devices.

[0014] In one embodiment, if Figure 2 As shown, a speech style conversion method is provided, and the method is applied in Figure 1 The server in the example is used as an example to illustrate the process, including the following steps: S10: Acquire a speech data set, wherein the speech data set includes at least one source speech and a target speech corresponding to the source speech.

[0015] Understandably, the source voice is voice data whose style is changed but whose content remains unchanged. The target voice is voice data whose style is extracted and whose style is migrated. For example, in the application scenario of a voice changer, a pop-style music is changed into a rock-style music or a hip-hop-style music. The source voice and the target voice can be the voices of two completely different people, that is, the voices of two people, or the voices of the same person with different voice styles. Furthermore, a voice data set is constructed based on all source voices and all target voices corresponding to the source voices. Furthermore, the voice data set can be obtained by collecting from different databases, or it can be a voice data set sent from the client to the server.

[0016] S20: Extracting spectrum features of all the source speech and all the target speech to obtain a first Mel spectrum corresponding to the source speech and a second Mel spectrum corresponding to the target speech.

[0017] It can be understood that the first mel spectrum is a mel spectrum diagram of the source speech, and the second mel spectrum is a mel spectrum diagram of the target speech.

[0018] Specifically, after the source speech and the target speech are obtained, the spectrum features of all the source speech and all the target speech are extracted, that is, the one-dimensional time series signal that is difficult to process is converted into two-dimensional frequency domain data that is easy to process and has richer information, and the linear natural spectrum is converted into a Mel spectrum that reflects the characteristics of human hearing, so that the first Mel spectrum corresponding to the source speech and the second Mel spectrum corresponding to the target speech can be obtained. The spectral features can be extracted by using Mel cepstral coefficients or by functions in the librosa library, or other Mel spectrum extraction methods.

[0019] S30, extracting content features from the first mel-spectrogram using a preset content encoder to obtain content features corresponding to the source speech.

[0020] S40: Down-sampling the phoneme sequence aligned with the content feature to obtain a hidden state sequence.

[0021] It can be understood that the content feature is obtained by extracting the content of the source speech based on the first Mel spectrum. The hidden state sequence is obtained by downsampling the hidden state of the phoneme sequence in the content feature. The preset content encoder is constructed based on the encoder in autoVC.

[0022] Specifically, after obtaining the first Mel spectrum, the preset content encoder is called, the first Mel spectrum is input into the preset content encoder, and the content features of the first Mel spectrum are extracted by the preset content encoder, that is, the first Mel spectrum is decoupled by the preset content encoder, and the content features of the source speech are decoupled from other features, thereby extracting the content features in the first Mel spectrum, and the content features of the source speech can be obtained. Further, the MFA (Montreal Forced Aligner) alignment tool is used to force the alignment of the content features and the phoneme sequence corresponding to the content features to obtain the alignment features. The hidden state of the alignment features is converted to obtain a hidden state vector, and then the hidden state sequence of the phoneme sequence can be obtained by downsampling the hidden state vector.

[0023] S50: extracting style features from the second Mel-spectrogram using a preset style encoder to obtain style features corresponding to the target speech.

[0024] S60: Align and embed the style feature and the hidden state sequence to obtain a target style feature.

[0025] Understandably, the preset style encoder is an encoder for extracting the speaking style in speech. The style feature is obtained by extracting the style of the target speech based on the second Mel spectrum. The target style feature is obtained by aligning and embedding the local prosody embedding and the hidden state sequence of the style feature.

[0026] Specifically, after obtaining the second Mel spectrum, the preset style encoder is called, the second Mel spectrum is input into the preset style encoder, and the style features of the second Mel spectrum are extracted by the preset style encoder, that is, the second Mel spectrum is decoupled by the preset style encoder, and the style and other features of the target speech are decomposed, thereby extracting the style features in the second Mel spectrum, and the style features of the target speech can be obtained. Further, the global style embedding and local prosody embedding in the style features are extracted, and the hidden state sequence obtained by downsampling the source speech and the local prosody embedding in the style features are aligned and embedded, and the result after the aligned embedding is fused with the global style embedding again, and the target style features can be obtained.

[0027] S70: Determine a speech conversion result according to the hidden state sequence and the target style feature.

[0028] It can be understood that the speech conversion result is obtained by converting the target style features of the target speech into the content features of the source speech.

[0029] Specifically, after obtaining the target style feature, the hidden state sequence extracted from the source speech content feature and the target style feature are first concatenated or fused to obtain the fused feature. The fused feature is input into a preset decoder, and the fused feature is decoded by the preset decoder to obtain the target spectrogram, and then the target spectrogram is inversely transformed by the preset vocoder to convert the target spectrogram into speech data, and the obtained speech data is the speech conversion result.

[0030] In an embodiment of the present invention, a speech style conversion method is provided. The method realizes the acquisition of a first Mel spectrum of the source speech and the acquisition of a second Mel spectrum of the target speech by performing spectrum conversion on all the acquired source speech and all the target speech. The content feature of the source speech is extracted by a preset content encoder. Then, the conversion of the phoneme sequence in the content feature is realized by downsampling the phoneme sequence aligned with the content feature. The style feature of the target speech is extracted by a preset style encoder. The style feature and the hidden state sequence are aligned and embedded to realize the alignment of the local features in the target speech. According to the hidden state sequence and the target style feature, the style conversion of the source speech is realized, and then the speech conversion result is obtained. Furthermore, by downsampling the phoneme sequence aligned with the content feature, the hidden state extraction representation of each frame of the phoneme sequence is realized, thereby solving the problem of the existence of noise frames. By aligning and embedding the style feature and the hidden state sequence, the effective transfer of other styles except timbre is realized, thereby reducing the influence of noise frames and timbre, and improving the conversion effect of speech style.

[0031] In one embodiment, in step S20, spectral features are extracted from all the source speech and all the target speech to obtain a first Mel spectrum corresponding to the source speech and a second Mel spectrum corresponding to the target speech, including: S201, performing audio processing on all the source speech and all the target speech to obtain a first Fourier spectrum corresponding to the source speech and a second Fourier spectrum corresponding to the target speech.

[0032] S202: Perform spectrum conversion on all the first Fourier spectra and all the second Fourier spectra to obtain a first Mel spectrum and a second Mel spectrum.

[0033] It can be understood that the first Fourier spectrum and the second Fourier spectrum are obtained by performing audio processing on the source speech and the target speech. The first Mel spectrum and the second Mel spectrum are Mel spectrum diagrams of the source speech and the target speech.

[0034] Specifically, all source speech and target speech are obtained, and audio processing is performed on all source speech and target speech, that is, the source speech and target speech are framed at a fixed time period (such as 25 milliseconds), that is, the source speech and target speech are divided into frames of audio signals to obtain multiple frame units. Among them, in order to avoid excessive changes in adjacent frame units, there is an overlapping area between two adjacent frame units. Then, the audio signal of each frame is windowed, that is, each frame unit is multiplied by a window function so that the left and right ends of each frame unit are continuous, and a windowed audio signal is obtained. The windowed audio signal is discretely Fourier transformed, that is, the windowed audio signal is converted into energy in the frequency domain, and the spectrum distributed in different time windows on the time axis is obtained, so that the first Fourier spectrum corresponding to the source speech and the second Fourier spectrum corresponding to the target speech can be obtained.

[0035] Furthermore, the first Fourier spectrum and the second Fourier spectrum are low-pass filtered by a Mel filter, that is, the linear natural spectrum is converted into a Mel spectrum that reflects the characteristics of human hearing, and a Mel (scaled) filter group is used to transform the linear spectrum group into a Mel spectrum group to simulate the linear perception relationship of the human ear to frequency, so as to obtain the first Mel spectrum corresponding to the source speech and the second Mel spectrum corresponding to the target speech.

[0036] The embodiment of the present invention realizes converting a one-dimensional time series signal that is difficult to process into two-dimensional frequency domain data that is easy to process and has richer information by performing audio processing on the source speech and the target speech. Furthermore, by performing spectrum conversion on all first Fourier spectra and all second Fourier spectra, the first Mel spectrum and the second Mel spectrum are determined.

[0037] In one embodiment, in step S30, that is, extracting content features from the first Mel-spectrogram by a preset content encoder to obtain content features corresponding to the source speech includes: S301 , performing convolution processing on the first mel spectrum through a convolution layer in the preset content encoder to obtain content convolution features.

[0038] It can be understood that the content convolution feature is obtained by convolving the first Mel spectrum through the convolution layer.

[0039] Specifically, after obtaining the first Mel spectrum, the first Mel spectrum is input into the preset content encoder, and the first Mel spectrum is convolved through a preset number of layers (such as 3 layers, etc.) of convolution layers in the preset content encoder, that is, each convolution layer first convolves the first Mel spectrum with a convolution matrix with a convolution kernel of 5*1 and a channel number of 512, and then the convolution result is batch normalized and activated through the ReLU function to obtain the first convolution feature. In this way, the upper convolution features are sequentially convolved through the preset number of convolution layers to obtain the content convolution feature. Among them, before passing through the convolution layer, the first Mel spectrum is preprocessed by two LSTM layers and a fully connected layer to obtain a preprocessing result, and then the preprocessing result and the first Mel spectrum are connected, and the connection feature is input into the convolution layer.

[0040] S302, extracting content from the content convolution feature through the BLSTM layer in the preset content encoder to obtain content features.

[0041] Specifically, after obtaining the content convolution feature, it is passed to a stack consisting of two bidirectional LSTM layers for content extraction, that is, the content convolution feature is extracted through the BLSTM layer in the preset content encoder, that is, the hidden layer feature of the previous moment and the content convolution feature input at the current moment are selected through the forget gate in the forward or reverse LSTM, and a value between 0 and 1 is output, and the content convolution feature is selected and forgotten according to the value. The sigmoid layer in the input gate determines the information to be updated from the hidden layer feature of the previous moment and the content convolution feature input at the current moment, and a new candidate feature is created through the tanh layer, and the two features are multiplied to obtain a feature result. The content convolution feature selected to be retained at the current moment is multiplied with the output feature of the previous moment and the feature result is added to obtain the updated feature. Finally, the sigmoid layer in the output gate determines the information to be output, and the updated feature is normalized through the tanh layer to obtain the normalized feature. The normalized feature and the output information are multiplied to obtain the forward multiplication result and the reverse multiplication result. The forward multiplication result and the reverse multiplication result are then downsampled to obtain the content features. For the forward multiplication result, the time step is retained as [0, 32, 64, ...], and for the reverse multiplication result, the time step is retained as [31, 63, 95, ...].

[0042] The embodiment of the present invention performs convolution processing on the first mel spectrum by using a multi-layer convolution layer in a preset content encoder, thereby obtaining content convolution features. Content convolution features are extracted by using a BLSTM layer, thereby extracting content features from both forward and reverse directions, thereby decoupling content features from other features.

[0043] In one embodiment, in step S40, that is, downsampling the phoneme sequence aligned with the content feature to obtain a hidden state sequence includes: S401, aligning the content feature and the phoneme sequence to obtain an aligned feature.

[0044] S402, performing down-sampling processing on the alignment features through a preset first gated loop module to obtain a hidden state sequence.

[0045] Understandably, the alignment feature is a time-aligned frame-level sequence obtained by forcibly aligning the content feature and the phoneme sequence through a tool. The hidden state sequence is a representation of the hidden state extraction of the phoneme in the alignment feature. The first gated loop module is preset to be a GRU module, including a reset gate and an update gate. The larger the update gate, the more state information of the previous moment is brought in. The smaller the reset gate, the less information of the previous state is written.

[0046] Specifically, after obtaining the content feature, the phoneme sequence corresponding to the source speech is obtained, and the content feature and the phoneme sequence are forced to be aligned through the MFA tool, that is, the content feature and the phoneme sequence are aligned in frames in the time dimension, and the alignment feature can be obtained. Then, the alignment feature is input into the preset first gated loop module, and the hidden state in the alignment feature is extracted through the preset first gated loop module, that is, the preset first gated loop module is used to generate a hidden state vector for the alignment feature frame by frame, and the information of the previous state in the alignment feature is first written to the current candidate state through the reset gate, and then the state information of the previous moment is controlled by the update gate. The degree of bringing the current state into the state can obtain the hidden state vector corresponding to each frame. Among them, the length of the hidden state vector is the same as that of the content feature, that is, the hidden state vector and the content feature are aligned. And the hidden state of the last output of the GRU module is used as the downsampling vector to obtain the hidden state sequence.

[0047] The embodiment of the present invention achieves the acquisition of a time-aligned frame-level sequence by aligning the content feature with the phoneme sequence, thereby achieving the acquisition of the alignment feature. The alignment feature is downsampled by presetting the first gated loop module, thereby achieving the extraction of the hidden state vector of each frame of phonemes, thereby achieving the acquisition of the hidden state sequence.

[0048] In one embodiment, in step S50, style feature extraction is performed on the second Mel spectrum by a preset style encoder to obtain style features corresponding to the target speech, including: S501 , extracting features from the second mel-frequency spectrum through the LSTM layer in the preset style encoder to obtain encoding features.

[0049] S502, performing convolution processing on the encoding feature through a convolution layer in the preset style encoder to obtain a style feature.

[0050] It can be understood that the encoding feature is obtained by extracting the second Mel spectrum through the LSTM layer. The style feature is the speaking style of each speaker, that is, other speaking habits except timbre.

[0051] Specifically, after obtaining the second Mel spectrum, the second Mel spectrum is input into the preset style encoder, and the second Mel spectrum is extracted through the preset number of layers (such as 2 layers, etc.) of LSTM layers in the preset style encoder, that is, the hidden layer features of the previous moment and the input features of the current moment are selected through the forget gate in the LSTM network, and a value between 0 and 1 is output, and the encoding features are selected and forgotten according to the value. The sigmoid layer in the input gate determines the information to be updated from the hidden layer features of the previous moment and the input features of the current moment, and a new candidate feature is created through the tanh layer, and the two features are multiplied to obtain a feature result. The input feature selected to be retained at the current moment is multiplied with the output feature of the previous moment and the feature result is added to obtain the updated feature. Finally, the sigmoid layer in the output gate determines the information to be output, and the updated feature is normalized through the tanh layer to obtain the normalized feature. The normalized feature and the output information are multiplied, and the multiplication result is input into the second LSTM layer, so that the encoding feature can be obtained through multiple LSTM layers. Furthermore, the encoded features are convolved through the convolutional layers in the preset style encoder, that is, the encoded features are convolved through a preset number of layers (such as 3 layers, etc.), and the convolution results are batch normalized and activated through the ReLU function to obtain the convolutional features of the first layer. The upper layer convolutional features are convolved multiple times through the convolutional layers of the preset number of layers to obtain the style features.

[0052] The embodiment of the present invention extracts features of the second mel spectrum through the LSTM layer in the preset style encoder, thereby obtaining the encoding features. The encoding features are convolved by the convolution layer in the preset style encoder, thereby extracting the speech style from the target speech, decoupling the style features from other features, and further obtaining the style features.

[0053] In one embodiment, in step S60, aligning and embedding the style feature and the hidden state sequence to obtain the target style feature includes: S601, embedding the style feature through a preset second gated loop module to obtain a local rhythm embedding and a global style embedding.

[0054] Specifically, after obtaining the style features, the style features are input into a preset second gated loop module, and the style features are embedded through the preset second gated loop module, that is, the reset gate in the preset second gated loop module is used to control how much information of the previous state in the alignment features is written to the current candidate state, and then the update gate in the preset second gated loop module is used to control the degree to which the state information at the previous moment is brought into the current state, so that the hidden state of each layer output can be obtained, and the hidden state of the last layer output is determined as the global style embedding, and the sequence composed of the hidden states of the output of each layer is determined as the local rhythm embedding.

[0055] S602, aligning and embedding the local prosody embedding and the hidden state sequence through a preset attention module to obtain a local embedding.

[0056] S603, performing expansion processing on the global style embedding to obtain a global embedding, and determining a target style feature according to the global embedding and the local embedding.

[0057] Specifically, the local prosody embedding is divided into two sequences from the feature dimension, the first sequence is determined as the key vector and the second sequence is determined as the value vector, and the hidden state sequence is determined as the query vector. Then, the query vector, key vector and value vector are calculated through the attention module, that is, the correlation score between the query vector and the key vector in the input vector is calculated using the dot product method, that is, each vector in the hidden state sequence is multiplied by each vector in the first sequence, and the correlation score between the query vector and the key vector is normalized. Then the score vector between the vectors is converted into a probability distribution between [0, 1], and according to the probability distribution between the vectors, it is multiplied by the corresponding value vector, and the local prosody embedding and the hidden state sequence are aligned to obtain the local embedding. Further, the global style embedding is expanded, that is, the global style embedding is repeated to a sequence of the same length as the local embedding to obtain the global embedding. And the global embedding and the local embedding are spliced ​​and fused in the time dimension in frames to obtain the target style features.

[0058] The embodiment of the present invention realizes the extraction of hidden states in the style features by embedding the style features, thereby realizing the acquisition of local rhythm embedding and global style embedding. By aligning and embedding the local rhythm embedding and the hidden state sequence, the alignment of local features is realized, thereby realizing the determination of local embedding. The global style embedding is expanded to realize the acquisition of global embedding, thereby realizing the acquisition of target style features.

[0059] In one embodiment, in step S70, determining the speech conversion result according to the hidden state sequence and the target style feature includes: S701, fusing the hidden state sequence and the target style feature to obtain a fused feature.

[0060] S702, decoding the fusion feature through a preset decoder to obtain a target spectrum diagram.

[0061] Understandably, the target spectrogram is obtained by decoding the features after style conversion.

[0062] Specifically, the hidden state sequence of the content features corresponding to the same source speech and the target style features are fused to obtain the fused features. The fused features are first upsampled, that is, the fused features are restored to their original dimensions, and then the upsampled fused features are input into the preset decoder, and the fused features are decoded by the preset decoder, that is, the convolution matrix with a convolution kernel of 5*1 and a channel number of 512 is first passed through the first preset number of layers (such as 3 layers), as well as batch normalization and function activation, to obtain a deep convolution matrix, and then the deep convolution matrix is ​​processed through a multi-layer (such as 3 layers) LSTM network, and the processing result is subjected to one-dimensional convolution to obtain a one-dimensional convolution result. Then, the convolution result is convolved, batch standardized, and function activated through a convolution matrix with a second preset number of layers (such as 4 layers) with a convolution kernel of 5*1 and a channel number of 512, to obtain the first convolution matrix. Finally, the first convolution result is obtained by convolving the first convolution matrix with a convolution kernel of 5*1 and a channel number of 80, performing batch standardized processing and function activation, and adding the remaining one-dimensional convolution results to obtain the target spectrum graph.

[0063] S703, performing inverse transformation processing on the target spectrogram through a preset vocoder to obtain a speech conversion result.

[0064] Specifically, the target spectrum graph is input into a preset vocoder, and the target spectrum graph is inversely transformed by the preset vocoder, that is, the target spectrum graph is first passed through the causal convolution layer in the preset vocoder, that is, a one-dimensional convolution is performed on the target spectrum graph to obtain a causal convolution vector. The causal convolution vector is deeply convolved by an enlarged convolution network of a preset number of layers, that is, the causal convolution vector is first enlarged, that is, the convolution kernel skips the input convolution when it is applied to data with an area larger than its own length, and the enlarged convolution vector can be obtained. Then the enlarged convolution vector is calculated by the tanh function and the sigmoid function respectively to obtain the output waveform of the tanh function and the output value range of the sigmoid function. The calculation results of the two functions are multiplied and one-dimensionally convolved to obtain a vector convolution result, and then the vector convolution result and the input causal convolution vector are added and input to the next layer of the enlarged convolution network. In this way, by expanding the convolution network with a preset number of layers and adding the vector convolution results of each layer, activating it through two ReLU functions and performing 1*1 convolution, and finally predicting it through the Softmax function, the speech conversion result can be obtained.

[0065] The embodiment of the present invention achieves the acquisition of fusion features by fusing the hidden state sequence and the target style features. The target spectrogram is acquired by decoding the upsampled fusion features through a preset decoder. The target spectrogram is inversely transformed through a preset vocoder to acquire the speech conversion result.

[0066] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0067] In one embodiment, a speech style conversion device is provided, and the speech style conversion device corresponds to the speech style conversion method in the above embodiment. Figure 5 As shown, the speech style conversion device includes a speech acquisition module 11, a spectrum extraction module 12, a content feature module 13, a downsampling module 14, a style feature module 15, an alignment embedding module 16 and a conversion result module 17. The functional modules are described in detail as follows: A speech acquisition module 11 is used to acquire a speech data set, wherein the speech data set includes at least one source speech and a target speech corresponding to the source speech; A spectrum extraction module 12 is used to extract spectrum features of all the source speech and all the target speech to obtain a first Mel spectrum corresponding to the source speech and a second Mel spectrum corresponding to the target speech; A content feature module 13 is used to extract content features from the first Mel spectrum using a preset content encoder to obtain content features corresponding to the source speech; A downsampling module 14, configured to perform downsampling processing on the phoneme sequence aligned with the content feature to obtain a hidden state sequence; A style feature module 15, configured to extract style features from the second Mel spectrum using a preset style encoder to obtain style features corresponding to the target speech; An alignment embedding module 16, used for aligning and embedding the style feature and the hidden state sequence to obtain a target style feature; The conversion result module 17 is used to determine the speech conversion result according to the hidden state sequence and the target style feature.

[0068] In one embodiment, the spectrum extraction module 12 includes: An audio processing unit, configured to perform audio processing on all the source speech and all the target speech to obtain a first Fourier spectrum corresponding to the source speech and a second Fourier spectrum corresponding to the target speech; The spectrum conversion unit is used to perform spectrum conversion on all the first Fourier spectra and all the second Fourier spectra to obtain a first Mel spectrum and a second Mel spectrum.

[0069] In one embodiment, the content feature module 13 includes: a content convolution feature unit, configured to perform convolution processing on the first mel spectrum through a convolution layer in the preset content encoder to obtain a content convolution feature; The content extraction unit is used to extract the content convolution feature through the BLSTM layer in the preset content encoder to obtain the content feature.

[0070] In one embodiment, the downsampling module 14 includes: An alignment feature unit, used for aligning the content feature with the phoneme sequence to obtain an alignment feature; The down-sampling unit is used to perform down-sampling processing on the alignment features through a preset first gated cycle module to obtain a hidden state sequence.

[0071] In one embodiment, the style feature module 15 includes: A coding feature unit, used for extracting features from the second Mel frequency spectrum through the LSTM layer in the preset style encoder to obtain coding features; The style extraction unit is used to perform convolution processing on the encoding feature through the convolution layer in the preset style encoder to obtain the style feature.

[0072] In one embodiment, the alignment embedding module 16 includes: An embedding unit, used for embedding the style feature through a preset second gated loop module to obtain a local rhythm embedding and a global style embedding; A local embedding unit, used for aligning and embedding the local prosodic embedding and the hidden state sequence through a preset attention module to obtain a local embedding; The global embedding unit is used to expand the global style embedding to obtain a global embedding, and determine a target style feature according to the global embedding and the local embedding.

[0073] In one embodiment, the conversion result module 17 includes: A fusion feature unit, used to fuse the hidden state sequence and the target style feature to obtain a fusion feature; A target spectrum graph unit is used to decode the fusion feature through a preset decoder to obtain a target spectrum graph; The conversion result unit is used to perform inverse transformation processing on the target spectrogram through a preset vocoder to obtain a speech conversion result.

[0074] For the specific definition of the speech style conversion device, please refer to the definition of the speech style conversion method above, which will not be repeated here. Each module in the above-mentioned speech style conversion device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0075] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data used in the voice style conversion method in the above-mentioned embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a voice style conversion method is implemented.

[0076] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned speech style conversion method when executing the computer program.

[0077] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned speech style conversion method is implemented.

[0078] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0079] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0080] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for voice style conversion, characterized in that: include: Acquire a speech data set, wherein the speech data set includes at least one source speech and a target speech corresponding to the source speech; Performing spectrum feature extraction on all the source speech and all the target speech to obtain a first Mel spectrum corresponding to the source speech and a second Mel spectrum corresponding to the target speech; Extracting content features from the first Mel-frequency spectrum using a preset content encoder to obtain content features corresponding to the source speech; Downsampling the phoneme sequence aligned with the content feature to obtain a hidden state sequence; Extracting style features from the second Mel-spectrogram using a preset style encoder to obtain style features corresponding to the target speech; Aligning and embedding the style feature and the hidden state sequence to obtain a target style feature; A speech conversion result is determined according to the hidden state sequence and the target style feature.

2. The method for voice style conversion according to claim 1, characterized in that: The aligning and embedding the style feature and the hidden state sequence to obtain the target style feature includes: The style feature is embedded by a preset second gated loop module to obtain a local rhythm embedding and a global style embedding; Aligning and embedding the local prosodic embedding and the hidden state sequence through a preset attention module to obtain a local embedding; The global style embedding is expanded to obtain a global embedding, and a target style feature is determined according to the global embedding and the local embedding.

3. The method for voice style conversion according to claim 1, characterized in that: The downsampling process is performed on the phoneme sequence aligned with the content feature to obtain a hidden state sequence, including: Aligning the content feature with the phoneme sequence to obtain an alignment feature; The alignment features are downsampled by a preset first gated loop module to obtain a hidden state sequence.

4. The method for voice style conversion according to claim 1, wherein: The step of determining the speech conversion result according to the hidden state sequence and the target style feature includes: Fusing the hidden state sequence and the target style feature to obtain a fused feature; The fusion feature is decoded by a preset decoder to obtain a target spectrum graph; The target spectrogram is inversely transformed by a preset vocoder to obtain a speech conversion result.

5. The method for voice style conversion according to claim 1, wherein: The extracting content features from the first Mel-spectrogram by using a preset content encoder to obtain content features corresponding to the source speech includes: Performing convolution processing on the first Mel spectrum through a convolution layer in the preset content encoder to obtain content convolution features; The content convolution feature is subjected to content extraction through the BLSTM layer in the preset content encoder to obtain content features.

6. The method for voice style conversion according to claim 1, wherein: The extracting style features of the second Mel spectrum by using a preset style encoder to obtain style features corresponding to the target speech includes: Performing feature extraction on the second Mel frequency spectrum through the LSTM layer in the preset style encoder to obtain encoding features; The encoding feature is convolved by a convolution layer in the preset style encoder to obtain a style feature.

7. The method for voice style conversion according to claim 1, wherein: The extracting spectrum features of all the source speech and all the target speech to obtain a first Mel spectrum corresponding to the source speech and a second Mel spectrum corresponding to the target speech includes: Performing audio processing on all the source speech and all the target speech to obtain a first Fourier spectrum corresponding to the source speech and a second Fourier spectrum corresponding to the target speech; Spectrum conversion is performed on all the first Fourier spectra and all the second Fourier spectra to obtain a first Mel spectrum and a second Mel spectrum.

8. A speech style conversion device, characterized in that: include: A speech acquisition module, used to acquire a speech data set, wherein the speech data set includes at least one source speech and a target speech corresponding to the source speech; A spectrum extraction module, used for extracting spectrum features of all the source speech and all the target speech to obtain a first Mel spectrum corresponding to the source speech and a second Mel spectrum corresponding to the target speech; A content feature module, configured to extract content features from the first Mel-spectrogram through a preset content encoder to obtain content features corresponding to the source speech; A downsampling module, used for downsampling the phoneme sequence aligned with the content feature to obtain a hidden state sequence; A style feature module, configured to extract style features from the second Mel spectrum using a preset style encoder to obtain style features corresponding to the target speech; An alignment embedding module, used for aligning and embedding the style feature and the hidden state sequence to obtain a target style feature; The conversion result module is used to determine the speech conversion result according to the hidden state sequence and the target style feature.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for voice style conversion according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for voice style conversion according to any one of claims 1 to 7 is implemented.

Images