Timbre conversion method, device, equipment and storage medium

Abstract

The present disclosure provides a timbre conversion method, device, equipment and storage medium, which belongs to the technical field of computer. The method comprises: encoding content information, pitch information of a first audio and voiceprint information of a second audio into a first feature vector, the first feature vector representing content features, pitch features of the first audio and timbre features of the second audio; mapping the first feature vector to a latent space based on distribution information of the first feature vector to obtain a latent feature vector; and decoding the latent feature vector to obtain a third audio, the timbre of the third audio being a target timbre, and the content of the third audio being the same as that of the first audio. The technical solution can quickly and accurately convert the source audio into the target audio with any timbre, and can preserve the pitch information and content information of the source audio during the conversion process, thereby ensuring the accuracy of the timbre conversion, preserving the detailed information in the audio, and improving the naturalness and fluency of the target audio.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a timbre conversion method and device, equipment and storage medium. BACKGROUND

[0002] The timbre conversion technology refers to converting the timbre of audio from one timbre to another timbre without changing the audio content. At present, the timbre conversion technology is widely used in virtual image, digital person or real person voice scenes.

[0003] In the related art, the frequency or amplitude of the voice can be changed through signal processing to achieve timbre conversion. For example, in the above-mentioned scene, the terminal can change the amplitude, frequency and other physical characteristics of the audio input by the user according to the target timbre selected by the user, so as to convert the timbre of the audio to the target timbre without changing the audio content, thereby improving the diversity and interest of voice interaction.

[0004] However, the audio obtained by the above method is usually not natural, and the pitch of the audio is easily changed, resulting in low naturalness and accuracy of timbre conversion. SUMMARY

[0005] The present disclosure provides a timbre conversion method, device, equipment and storage medium, which can quickly and accurately convert source audio to target audio of any timbre, and can also preserve the pitch information and content information of the source audio during the conversion process, thereby improving the naturalness and fluency of the target audio. The technical solutions of the present disclosure are as follows:

[0006] According to an aspect of an embodiment of the present disclosure, a timbre conversion method is provided, comprising:

[0007] encoding content information, pitch information of a first audio and voiceprint information of a second audio into a first feature vector, the first audio being an audio of a to-be-converted timbre, the content information including phonemes corresponding to a plurality of audio frames in the first audio, the pitch information including pitches of the plurality of audio frames, the timbre of the second audio being a target timbre, the voiceprint information being used to represent the target timbre, the first feature vector representing content features, pitch features of the first audio and timbre features of the second audio;

[0008] mapping the first feature vector to a latent space based on distribution information of the first feature vector to obtain a latent feature vector, the distribution information including a mean and a variance;

[0009] decoding the latent feature vector to obtain a third audio, the timbre of the third audio being the target timbre, and the content of the third audio being the same as that of the first audio.

[0010] According to another aspect of the embodiments of the present disclosure, a timbre conversion device is provided, comprising:

[0011] a coding unit configured to code content information of a first audio, pitch information, and voiceprint information of a second audio into a first feature vector, the first audio being an audio whose timbre is to be converted, the content information comprising phonemes corresponding to a plurality of audio frames in the first audio, the pitch information comprising pitches of the plurality of audio frames, the timbre of the second audio being a target timbre, the voiceprint information being used to represent the target timbre, the first feature vector representing content features of the first audio, pitch features, and timbre features of the second audio;

[0012] a mapping unit configured to map the first feature vector into a latent space based on distribution information of the first feature vector, to obtain a latent feature vector, the distribution information comprising a mean and a variance;

[0013] a decoding unit configured to decode the latent feature vector to obtain a third audio, the timbre of the third audio being the target timbre, the content of the third audio being the same as that of the first audio.

[0014] In some embodiments, the mapping unit is configured to determine, based on the distribution information of the first feature vector, a plurality of Gaussian distributions of the first feature vector in the latent space, the number of the plurality of Gaussian distributions being the same as the dimension of the first feature vector; and sample the plurality of Gaussian distributions respectively to obtain the latent feature vector, the dimension of the latent feature vector being the same as the number of the plurality of Gaussian distributions.

[0015] In some embodiments, the content information of the first audio is a content feature vector, a plurality of elements in the content feature vector corresponding to a plurality of audio frames in the first audio one by one, and the device further comprises:

[0016] a feature extraction unit configured to perform feature extraction on the first audio to obtain an audio feature vector, the audio feature vector being used to represent audio features of a plurality of audio frames in the first audio; for any audio frame, a plurality of elements capable of reflecting the audio features of the audio frame are determined from the audio feature vector, the audio features being capable of representing phonemes corresponding to the audio frame; and a sub-feature vector composed of the plurality of elements is mapped into an element in the content feature vector.

[0017] In some embodiments, the extracting unit is configured to determine similarities between the sub-feature vectors and clustering centers of a plurality of feature clusters respectively, the plurality of feature clusters are obtained by clustering audio feature vectors of a plurality of reference audios, the clustering centers of the feature clusters are used to represent corresponding phonemes of the feature clusters; determine a target clustering center based on the similarities between the sub-feature vectors and the clustering centers, the target clustering center is a clustering center with the highest similarity to the sub-feature vectors; and map the target clustering center as an element in the content feature vector.

[0018] In some embodiments, the encoding unit is configured to encode the content information and the pitch information of the first audio by an attention mechanism to obtain an intermediate feature vector; perform nonlinear mapping on a weighted sum of the intermediate feature vector and the voiceprint information by a plurality of activation functions to obtain a plurality of initial data; and determine a product of the plurality of initial data as the first feature vector.

[0019] In some embodiments, the content information is a content feature vector; the encoding unit is configured to encode the pitch information of the first audio to obtain a pitch feature vector; perform dimension alignment on the content feature vector and the pitch feature vector to make the dimension of the content feature vector the same as the dimension of the pitch feature vector; fuse the content feature vector and the pitch feature vector to obtain a fused feature vector; and encode the fused feature vector by the attention mechanism to obtain the intermediate feature vector.

[0020] In some embodiments, the apparatus further comprises:

[0021] The predicting unit is configured to input the first feature vector into a deep neural network, the deep neural network comprises a plurality of network layers connected in series, the deep neural network is used to predict distribution information of the first feature vector; process the first feature vector by a first network layer of the deep neural network to obtain output data of the network layer; for any intermediate network layer, process intermediate input data by the network layer to obtain output data of the network layer, the intermediate input data is a product of a plurality of intermediate data, the intermediate data is obtained by processing a weighted sum of output data of a previous network layer and the voiceprint information by an activation function; and predict the distribution information of the first feature vector based on output data of a last network layer.

[0022] In some embodiments, the timbre conversion method is performed by a timbre conversion model, the timbre conversion model comprising an encoding module, a mapping module, and a decoding module, the encoding module being configured to encode the content information and the pitch information, the mapping module being configured to map the first feature vector into the latent feature vector, and the decoding module being configured to decode the latent feature vector; the apparatus further comprises:

[0023] a training unit configured to obtain a sample audio and a sample latent feature vector, the sample latent feature vector being obtained by encoding a spectrogram of the sample audio; encode, by the encoding module, content information and pitch information of the sample audio to obtain a second feature vector; map, by the mapping module, the sample latent feature vector into a third feature vector based on voiceprint information of the sample audio; decode, by the decoding module, the sample latent feature vector to obtain a target audio; determine a first training loss based on the second feature vector and the third feature vector, and determine a second training loss based on the sample audio and the target audio; and train the timbre conversion model based on the first training loss and the second training loss.

[0024] According to another aspect of embodiments of the present disclosure, an electronic device is provided, the electronic device comprising:

[0025] one or more processors;

[0026] a memory for storing program code executable by the processor;

[0027] wherein the processor is configured to execute the program code to implement the above-described timbre conversion method.

[0028] According to another aspect of embodiments of the present disclosure, a computer-readable storage medium is provided, when program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is enabled to perform the above-described timbre conversion method.

[0029] According to another aspect of embodiments of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the above-described timbre conversion method.

[0030] The embodiment of the present disclosure provides a timbre conversion scheme, which can encode content information, pitch information of the first audio and voiceprint information of the second audio into a feature vector in the process of converting the timbre of the first audio by referring to the timbre of the second audio, so that the feature vector contains the content and pitch of the first audio and the timbre of the second audio. Then, the feature vector is mapped into a latent feature vector in a latent space according to distribution information of the feature vector. Then, the third audio is decoded according to the latent feature vector. Through the above method, the source audio can be quickly and accurately converted into the target audio with any timbre. Further, since the pitch information and content information of the source audio are retained in the conversion process, more detailed information in the audio can be retained while ensuring the accuracy of the timbre conversion, thereby improving the naturalness and fluency of the target audio.

[0031] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings incorporated in the specification and forming a part of it, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure without imposing undue limitation on the disclosure.

[0033] Figure 1 is a schematic diagram of an implementation environment of a timbre conversion method according to an exemplary embodiment.

[0034] Figure 2 is a flowchart of a timbre conversion method according to an exemplary embodiment.

[0035] Figure 3 is a flowchart of another timbre conversion method according to an exemplary embodiment.

[0036] Figure 4 is a schematic diagram of a clustering process according to an exemplary embodiment.

[0037] Figure 5 is a schematic diagram of predicting distribution information according to an exemplary embodiment.

[0038] Figure 6 is a model architecture diagram of a timbre conversion model according to an exemplary embodiment.

[0039] Figure 7 is a schematic diagram of training a timbre conversion model according to an exemplary embodiment.

[0040] Figure 8 is a block diagram of a timbre conversion device according to an exemplary embodiment.

[0041] Figure 9 is a block diagram of another timbre conversion device according to an example embodiment.

[0042] Figure 10 is a block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0043] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings.

[0044] It should be noted that the terms "first", "second" and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following example embodiments does not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0045] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the first audio and the second audio involved in the present disclosure are obtained under sufficient authorization.

[0046] Figure 1 is a schematic diagram of an implementation environment of a timbre conversion method according to an example embodiment. Referring to Figure 1 , the implementation environment specifically includes a terminal 101 and a server 102. The terminal 101 can be connected to the server 102 through a wireless network or a wired network.

[0047] The terminal 101 can be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, an MP3 player, an MP4 player, and a laptop computer. The terminal 101 is installed or runs a target application. The target application is any type of application that can provide a timbre conversion function, such as a video application, a singing application, or a social application. The user can input a first audio with a target timbre and a second audio with a target timbre in the target application through the terminal 101, and the target application converts the first audio to the target timbre according to the target timbre. Alternatively, the user can select a target timbre from a plurality of timbres provided by the target application through the terminal 101, and the target application converts the first audio to the target timbre according to the target timbre. The target application is associated with the server 102, and the server 102 provides background services.

[0048] The terminal 101 can refer to one of a plurality of terminals, and the embodiment is exemplified by the terminal 101. Those skilled in the art can know that the number of terminals can be more or less. For example, the terminals can be several, or the terminals can be dozens or hundreds, or more, and the number of terminals and the type of equipment are not limited in the embodiment.

[0049] The server 102 is at least one of a server, a plurality of servers, a cloud computing platform, and a virtualization center. Optionally, the number of servers can be more or less, and the embodiment does not limit this. Of course, the server 102 can also include other functional servers to provide more comprehensive and diverse services. In some embodiments, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or the server 102 and the terminal 101 adopt a distributed computing architecture for collaborative computing. The server 102 can be connected to the terminal 101 and other terminals through a wireless network or a wired network, and the number of servers can be more or less, and the embodiment does not limit this.

[0050] Figure 2 is a flowchart of a timbre conversion method according to an exemplary embodiment, as shown in Figure 2 The method is performed by an electronic device, including the following steps:

[0051] In step S201, content information of the first audio, pitch information of the first audio, and voiceprint information of the second audio are encoded into a first feature vector.

[0052] In the embodiments of the present disclosure, the first audio is audio to be converted in timbre. Optionally, the first audio is speech, singing, or an audiobook, etc. The second audio is audio referenced when the first audio is converted in timbre. The timbre of the second audio is a target timbre. The voiceprint information of the second audio can represent the timbre of the second audio, i.e., can represent the target timbre.

[0053] In the process of converting the first audio in timbre according to the timbre of the second audio, the electronic device encodes, by an encoder, the content information of the first audio, the pitch information of the first audio, and the voiceprint information of the second audio into a first feature vector, so that the first feature vector contains the content and the pitch of the first audio, and contains the timbre of the second audio. Correspondingly, the first feature vector can represent the content feature and the pitch feature of the first audio, and can also represent the timbre feature of the second audio.

[0054] The first audio includes a plurality of audio frames. The content information of the first audio includes phonemes corresponding to the plurality of audio frames in the first audio. Each audio frame corresponds to at least one phoneme. A phoneme is the smallest unit of speech in human language. The number and types of phonemes are different in different languages. Since the content information includes a plurality of phonemes, the content information can represent the content of the first audio. The pitch information of the first audio includes pitches of the plurality of audio frames in the first audio. The pitch is also called tone. The pitch mainly depends on the frequency of the vibration of the sound-producing body. Generally, the higher the vibration frequency, the higher the pitch; the lower the vibration frequency, the lower the pitch.

[0055] In step S202, the first feature vector is mapped to a latent space based on distribution information of the first feature vector, to obtain a latent feature vector.

[0056] In the embodiments of the present disclosure, the electronic device can determine the distribution information of the first feature vector according to the numerical values of the plurality of elements in the first feature vector. The distribution information includes a mean and a variance. By determining the distribution information of the first feature vector, not only the content, the pitch, and the timbre represented by the first feature vector can be determined, but also the distribution probability of the above features can be determined. The electronic device maps the first feature vector to the latent space according to the distribution information of the first feature vector, to obtain a latent feature vector.

[0057] The latent space generally refers to a low-dimensional space to which the first feature vector is mapped. The distribution information can indicate a distribution of the first feature vector in the latent space, such as a Gaussian distribution, a normal distribution, or the like. According to the distribution information, the electronic device can determine the distribution of the first feature vector in the latent space, and then map the first feature vector to a latent feature vector in the latent space according to the distribution. By mapping the first feature vector to the latent feature vector, the dimension of the first feature vector can be reduced while retaining the key information in the first feature vector, thereby effectively realizing data compression.

[0058] In step S203, the latent feature vector is decoded to obtain the third audio.

[0059] In the embodiments of the present disclosure, since the latent feature vector can represent not only the content information and the pitch information but also the target timbre, the electronic device can obtain the third audio by decoding the latent feature vector through the vocoder. The timbre of the third audio is the target timbre, and the content of the third audio is the same as that of the first audio. Further, since the pitch information is considered in the timbre conversion process, when the first audio is a singing voice, a song, or the like with relatively rich pitch, the detailed information in the pitch aspect can be retained in the timbre conversion process, thereby ensuring the accuracy of the timbre conversion of the singing voice, the song, or the like.

[0060] The embodiments of the present disclosure provide a timbre conversion method, which can encode the content information, the pitch information of the first audio, and the voiceprint information of the second audio into a feature vector during the process of converting the timbre of the first audio by referring to the timbre of the second audio, so that the feature vector contains not only the content and the pitch of the first audio but also the timbre of the second audio. Then, the feature vector is mapped to a latent feature vector in a latent space according to the distribution information of the feature vector. Then, the third audio is decoded from the latent feature vector. Through the above method, the source audio can be quickly and accurately converted into the target audio with any timbre. Further, since the pitch information and the content information of the source audio are retained during the conversion, more detailed information in the audio can be retained while ensuring the accuracy of the timbre conversion, thereby improving the naturalness and fluency of the target audio.

[0061] In some embodiments, the first feature vector is mapped to the latent space based on the distribution information of the first feature vector to obtain the latent feature vector, including:

[0062] Based on the distribution information of the first feature vector, a plurality of Gaussian distributions of the first feature vector in the latent space are determined, and the number of the plurality of Gaussian distributions is the same as the dimension of the first feature vector.

[0063] The plurality of Gaussian distributions are respectively sampled to obtain the latent feature vector, and a dimension of the latent feature vector is same as a number of the plurality of Gaussian distributions.

[0064] In the embodiments of the present disclosure, the plurality of Gaussian distributions of the first feature vector in the latent space can be accurately determined according to the distribution information of the first feature vector. The latent feature vector similar to the first feature vector can be sampled from the Gaussian distributions, so that the latent feature vector can also represent the content information, the pitch information and the voiceprint information, and the accuracy of the latent feature vector and the amount of information represented by the latent feature vector are improved.

[0065] In some embodiments, the content information of the first audio is a content feature vector, a plurality of elements in the content feature vector correspond to a plurality of audio frames in the first audio in a one-to-one manner, and the method further includes:

[0066] The first audio is subjected to feature extraction to obtain an audio feature vector, and the audio feature vector is used to represent audio features of the plurality of audio frames in the first audio.

[0067] For any audio frame, a plurality of elements capable of reflecting the audio features of the audio frame are determined from the audio feature vector, and the audio features are capable of representing phonemes corresponding to the audio frame.

[0068] The sub-feature vector composed of the plurality of elements is mapped to an element in the content feature vector.

[0069] In the embodiments of the present disclosure, the audio features of the plurality of audio frames in the first audio can be quickly extracted by performing feature extraction on the first audio frame. Since the audio features can reflect the phonemes corresponding to the audio frame, the content information of the first audio can be accurately extracted by mapping the audio features of any audio frame to an element in the content feature vector, so that the content information can accurately reflect the content of the first audio, and the accuracy and efficiency of extracting the content information are improved.

[0070] In some embodiments, mapping the sub-feature vector composed of the plurality of elements to the element in the content feature vector includes:

[0071] Similarities between the sub-feature vector and clustering centers of a plurality of feature clusters are respectively determined, the plurality of feature clusters are obtained by clustering audio feature vectors of a plurality of reference audios, and the clustering center of the feature cluster is used to represent a phoneme corresponding to the feature cluster.

[0072] Based on the similarities between the sub-feature vector and the plurality of clustering centers, a target clustering center is determined, and the target clustering center is a clustering center with the highest similarity to the sub-feature vector.

[0073] The target clustering center is mapped to the element in the content feature vector.

[0074] In the embodiments of the present disclosure, by determining the phoneme corresponding to the audio frame according to the similarity between the sub-feature vector and the plurality of cluster centers, the phoneme represented by the cluster center most similar to the sub-feature vector can be determined as the phoneme corresponding to the audio frame in the content information, thereby improving the accuracy of extracting the content information. Further, since the number of cluster centers is usually large, the content information extracted by the above method has a small granularity, thereby further improving the accuracy of extracting the content information.

[0075] In some embodiments, the content information, the pitch information of the first audio, and the voiceprint information of the second audio are encoded into a first feature vector, including:

[0076] The content information and the pitch information of the first audio are encoded by an attention mechanism to obtain an intermediate feature vector;

[0077] The weighted sum of the intermediate feature vector and the voiceprint information is nonlinearly mapped by a plurality of activation functions to obtain a plurality of initial data;

[0078] The product of the plurality of initial data is determined as the first feature vector.

[0079] In the embodiments of the present disclosure, by encoding the content information and the pitch information by using the attention mechanism, not only the long-distance dependency in the input information can be accurately captured in the encoding process, but also the content information and the pitch information can be processed with emphasis in the encoding process, thereby improving the accuracy and efficiency of the encoding. In addition, by nonlinearly mapping the weighted sum of the intermediate feature vector and the voiceprint information by using the plurality of activation functions, the voiceprint information and the intermediate feature vector can be mapped into the first feature vector in a nonlinear manner, so that the first feature vector can express the content information, the pitch information, and the voiceprint information at the same time, thereby improving the amount of information expressed by the first feature vector.

[0080] In some embodiments, the content information is a content feature vector; the content information and the pitch information of the first audio are encoded by an attention mechanism to obtain an intermediate feature vector, including:

[0081] The pitch information of the first audio is encoded to obtain a pitch feature vector;

[0082] The content feature vector and the pitch feature vector are dimensionally aligned so that the dimension of the content feature vector is the same as the dimension of the pitch feature vector;

[0083] The content feature vector and the pitch feature vector are fused to obtain a fused feature vector;

[0084] The fused feature vector is encoded by an attention mechanism to obtain an intermediate feature vector.

[0085] In the embodiments of the present disclosure, by performing dimension alignment and vector fusion on the content information and the pitch information before encoding, not only the data amount of encoding can be reduced and the efficiency of encoding can be improved, but also the information amount expressed by the intermediate feature vector obtained after encoding can be improved.

[0086] In some embodiments, the method further comprises:

[0087] inputting the first feature vector into a deep neural network, the deep neural network comprising a plurality of network layers connected in series, the deep neural network being configured to predict distribution information of the first feature vector;

[0088] processing the first feature vector through a first network layer of the deep neural network to obtain output data of the network layer;

[0089] for any intermediate network layer, processing intermediate input data through the network layer to obtain output data of the network layer, the intermediate input data being a product of a plurality of intermediate data, the intermediate data being obtained by processing a weighted sum of output data of a previous network layer and the voiceprint information through an activation function;

[0090] predicting the distribution information of the first feature vector based on the output data of the last network layer.

[0091] In the embodiments of the present disclosure, the deep neural network with a multi-layer network structure is used to process the first feature vector and the voiceprint information of the second audio, so that the distribution information of the first feature vector can be predicted quickly and accurately, and the accuracy and efficiency of predicting the distribution information are both high.

[0092] In some embodiments, the timbre conversion method is performed by a timbre conversion model, the timbre conversion model comprising an encoding module, a mapping module and a decoding module, the encoding module being configured to encode the content information and the pitch information, the mapping module being configured to map the first feature vector into a latent feature vector, and the decoding module being configured to decode the latent feature vector;

[0093] The training process of the timbre conversion model comprises:

[0094] obtaining a sample audio and a sample latent feature vector, the sample latent feature vector being obtained by encoding a spectrum graph of the sample audio;

[0095] encoding the content information and the pitch information of the sample audio through the encoding module to obtain a second feature vector;

[0096] mapping the sample latent feature vector into a third feature vector based on the voiceprint information of the sample audio through the mapping module;

[0097] The sample latent feature vector is decoded by the decoding module to obtain the target audio;

[0098] The first training loss is determined based on the second feature vector and the third feature vector, and the second training loss is determined based on the sample audio and the target audio;

[0099] The timbre conversion model is trained based on the first training loss and the second training loss.

[0100] In the embodiments of the present disclosure, the timbre conversion model is trained by the above method, which not only enables the timbre conversion model to have the ability to convert any audio to any timbre, but also guarantees the accuracy and efficiency of the timbre conversion model in timbre conversion.

[0101] The above Figure 2 The flow of a timbre conversion method according to the present disclosure is shown, and the timbre conversion scheme provided by the present disclosure is further described below. Figure 3 is a flowchart of another timbre conversion method according to an exemplary embodiment, which is performed by an electronic device, see Figure 3 The method comprises the following steps:

[0102] In step S301, the content information and the pitch information of the first audio are obtained, the first audio being audio to be converted in timbre, the content information including phonemes corresponding to a plurality of audio frames in the first audio, and the pitch information including pitches of the plurality of audio frames.

[0103] In the embodiments of the present disclosure, the first audio is audio to be converted in timbre. Optionally, the first audio is a voice, a singing voice, etc. input by a user through an electronic device. Alternatively, the first audio is a singing voice, an audio book, etc. downloaded by a user through an electronic device, and the embodiments of the present disclosure do not limit this. In order to guarantee the content and the pitch of the first audio unchanged in the process of timbre conversion, the electronic device extracts the content information and the pitch information from the first audio.

[0104] The content information includes phonemes corresponding to a plurality of audio frames in the first audio. Each audio frame corresponds to at least one phoneme. A phoneme is the smallest unit of speech in human language, which is divided into two categories: vowels and consonants. The number and types of phonemes are different in different languages. For example, "Mandarin" contains "p, u, t, o, ng, h, u, a" eight phonemes. Since the content information includes a plurality of phonemes, the content information can represent the content of the first audio. For example, in the case of the first audio being speech, the content information can reflect the text told by the speech; in the case of the first audio being singing, the content information can reflect the lyrics sung by the singing. The pitch information includes the pitches of a plurality of audio frames in the first audio. The pitch is also called tone. The pitch mainly depends on the frequency of the vibration of the sound-producing body. Generally, the higher the vibration frequency, the higher the pitch; the lower the vibration frequency, the lower the pitch.

[0105] In some embodiments, the content information of the first audio is a content feature vector. The process of extracting the content feature vector by the electronic device is described below. The electronic device extracts features of the first audio by a feature extraction model to obtain an audio feature vector. The audio feature vector is used to represent audio features of a plurality of audio frames in the first audio. The audio features can reflect a plurality of features of the audio frames, such as content, timbre, speech rate, and the like. For any audio frame in the first audio, the electronic device determines a plurality of elements from the audio feature vector. The plurality of elements of the electronic device can reflect the audio features of the audio frame. The audio features can represent the phonemes corresponding to the audio frame, that is, can reflect the content features of the audio frame. Then, the electronic device maps a sub-feature vector composed of the plurality of elements into an element in the content feature vector. The sub-feature vector is a feature vector that can reflect the content features of the audio frame. The content feature vector is the content information of the first audio. The elements in the content information are used to represent the phonemes of the audio frame. The plurality of elements in the content feature vector correspond one-to-one to a plurality of audio frames in the first audio. Alternatively, the content feature vector can also be referred to as a phoneme sequence. By extracting features of the first audio frame, the audio features of a plurality of audio frames in the first audio can be quickly extracted. Since the audio features can reflect the phonemes corresponding to the audio frame, by mapping the audio features of any audio frame into an element in the content feature vector, the content information of the first audio can be accurately extracted, so that the content information can accurately reflect the content of the first audio, improving the accuracy and efficiency of extracting the content information.

[0106] In some embodiments, the electronic device maps the cluster center closest to the sub-feature vector as an element in the content feature vector. After obtaining the sub-feature vector reflecting the audio feature of the audio frame for any audio frame, the electronic device determines the similarity between the sub-feature vector and the cluster centers of the plurality of feature clusters respectively. The plurality of feature clusters are obtained by clustering the audio feature vectors of the plurality of reference audios. In the clustering process, the audio feature vectors with the same phoneme are usually clustered into the same feature cluster. The cluster center of the feature cluster is used to represent the phoneme corresponding to the feature cluster. The electronic device determines the target cluster center according to the similarity between the sub-feature vector and the plurality of cluster centers. The target cluster center is the cluster center with the highest similarity to the sub-feature vector. Since the target cluster center has the highest similarity to the sub-feature vector, it indicates that the phoneme corresponding to the audio frame is most likely to be the phoneme represented by the target cluster center. Therefore, the electronic device can map the target cluster center as an element in the content feature vector, so that the element can represent the phoneme corresponding to the audio frame. By determining the phoneme corresponding to the audio frame according to the similarity between the sub-feature vector and the plurality of cluster centers, the phoneme represented by the cluster center most similar to the sub-feature vector can be determined as the phoneme corresponding to the audio frame in the content information, thereby improving the accuracy of extracting the content information. Further, since the number of cluster centers is usually large, the content information extracted by the above method has a small granularity, thereby further improving the accuracy of extracting the content information.

[0107] The process of obtaining the plurality of cluster centers by clustering the audio feature vectors of the plurality of reference audios is described below. Figure 4 is a schematic diagram of a clustering process according to an example embodiment. As shown in Figure 4 , in the clustering process, the electronic device can obtain the audio feature vectors of 200 million reference audios. The 200 million reference audios cover audio samples of 500,000 speakers. For each audio feature vector, the electronic device randomly extracts 1 / 12 elements from the plurality of elements of the audio feature vector in the time dimension. Then, the electronic device randomly arranges and splices the extracted elements to obtain an audio feature set. Then, the electronic device performs Mini-batch K-means clustering (a variant of K-means clustering) on the audio feature set with a batch size of 1 million, to obtain 10,000 cluster centers. Alternatively, in the first round of clustering, the electronic device extracts 1 million data from the audio feature set and clusters the 1 million data to obtain 10,000 cluster centers. Then, the electronic device repeatedly performs the above clustering process through multiple iterations, thereby constantly updating the cluster centers until the cluster centers are stable or the preset number of iterations is reached.

[0108] In some embodiments, the electronic device performs pitch extraction on the first audio by a pitch extraction algorithm to obtain pitch information of the first audio. The pitch information can also be referred to as a pitch sequence. The pitch information includes the pitches of a plurality of audio frames. For example, the electronic device can perform pitch extraction on the first audio by a Robust Model for Vocal Pitch Estimation in polyphonic music (RMVPE) to obtain the pitch information. The embodiments of the present disclosure do not limit the algorithm and model used to extract the pitch information. By extracting the pitch information of the first audio by the pitch extraction algorithm, the pitches of the plurality of audio frames in the first audio can be quickly and accurately obtained, and the accuracy and efficiency of pitch extraction are improved.

[0109] In step S302, the content information of the first audio, the pitch information, and the voiceprint information of the second audio are encoded into a first feature vector.

[0110] In the embodiments of the present disclosure, the second audio is a reference audio in the timbre conversion process. The electronic device can convert the timbre of the first audio by referring to the timbre of the second audio, so as to convert the timbre of the first audio to the timbre of the second audio. The timbre of the second audio is the target timbre. The voiceprint information of the second audio can represent the timbre of the second audio, that is, the target timbre.

[0111] In the process of converting the timbre of the first audio by the user through the electronic device, the user can input the second audio through the electronic device, and the user can also select any audio from a plurality of reference audios provided by the electronic device as the second audio, which is not limited in the embodiments of the present disclosure. In the case that the user inputs the second audio, the electronic device can extract the voiceprint information of the second audio by a voiceprint extraction model. The voiceprint extraction model includes but is not limited to Wespeaker, ASV-Subtools, or X-Vectors, etc., which is not limited in the embodiments of the present disclosure.

[0112] After the electronic device obtains the content information and the pitch information of the first audio, and the voiceprint information of the second audio, the electronic device can encode the above content information, pitch information, and voiceprint information into a first feature vector by an encoder. The first feature vector represents the content features, the pitch features of the first audio, and the timbre features of the second audio.

[0113] In some embodiments, the electronic device can first encode the content information and the pitch information of the first audio through an attention mechanism to obtain an intermediate feature vector. The core idea of the attention mechanism is to determine different weights for different parts of the input data according to the importance of the data. By encoding the content information and the pitch information through the attention mechanism, the content information and the pitch information can be processed with emphasis in the encoding process, such as focusing on the content with higher weight in the content information and appropriately ignoring the content with lower weight in the content information, thereby improving the efficiency of encoding and the expression effect of the intermediate feature vector. The intermediate feature vector obtained by encoding through the attention mechanism can express both the content information of the first audio and the pitch information of the first audio. Then, the electronic device performs nonlinear mapping on the weighted sum of the intermediate feature vector and the voiceprint information through multiple activation functions to obtain multiple initial data. The activation functions include but are not limited to tanh (hyperbolic tangent), sigmoid (S-shaped function), etc. Then, the electronic device determines the product of the multiple initial data as the first feature vector, so that the first feature vector contains the voiceprint information of the second audio. By encoding the content information and the pitch information through the attention mechanism, not only can the long-distance dependency in the input information be accurately captured in the encoding process, but also the content information and the pitch information can be processed with emphasis in the encoding process, thereby improving the accuracy and efficiency of encoding. In addition, by performing nonlinear mapping on the weighted sum of the intermediate feature vector and the voiceprint information through multiple activation functions, the voiceprint information and the intermediate feature vector can be mapped into the first feature vector in a nonlinear manner, so that the first feature vector can express the content information, the pitch information and the voiceprint information at the same time, thereby improving the amount of information expressed by the first feature vector.

[0114] For example, the electronic device can perform nonlinear mapping on the weighted sum of the intermediate feature vector and the voiceprint information through tanh function and sigmoid function respectively to obtain the following two initial data: tanh(w1*x+w2*condition), sigmoid(w3*x+w4*condition). Wherein, x is the intermediate feature vector, condition is the voiceprint information. w1 and w3 are the weights of the intermediate feature vector, and w2 and w4 are the weights of the voiceprint information. Then, the electronic device determines the product of the two initial data to obtain the first feature vector: tanh(w1*x+w2*condition)*sigmoid(w3*x+w4*condition). Wherein, the dimensions of the intermediate feature vector x and the voiceprint information condition are the same.

[0115] In some embodiments, the electronic device can encode the content information and the pitch information by dimension alignment, vector fusion, etc. The electronic device encodes the pitch information of the first audio by the encoder to obtain a pitch feature vector. An element in the pitch feature vector is used to represent the pitch of an audio frame in the first audio. Then, the electronic device aligns the dimensions of the content feature vector and the pitch feature vector, so that the dimension of the content feature vector is the same as the dimension of the pitch feature vector. For example, in the case where the dimension of the content feature vector is higher than a preset dimension, the electronic device reduces the dimension of the content feature vector to the preset dimension by dimension reduction. In the case where the dimension of the pitch feature vector is lower than the preset dimension, the electronic device increases the dimension of the pitch feature vector to the preset dimension by zero padding or interpolation. Then, the electronic device fuses the content feature vector and the pitch feature vector to obtain a fused feature vector. Optionally, the electronic device can fuse the content feature vector and the pitch feature vector according to the dimensions to obtain the fused feature vector. The above-mentioned vector fusion can make the value of any dimension in the fused feature vector equal to the sum of the value of the corresponding dimension in the content feature vector and the value of the corresponding dimension in the pitch feature vector. Then, the electronic device encodes the fused feature vector by the attention mechanism to obtain an intermediate feature vector. By aligning the dimensions and fusing the vectors of the content information and the pitch information before encoding, the amount of data for encoding can be reduced, the efficiency of encoding can be improved, and the amount of information expressed by the intermediate feature vector obtained after encoding can be improved.

[0116] In step S303, distribution information of the first feature vector is determined, and the distribution information includes a mean value and a variance.

[0117] In the embodiments of the present disclosure, after obtaining the first feature vector, the electronic device determines the distribution information of the first feature vector. The distribution information includes a mean value and a variance. By determining the distribution information of the first feature vector, the content, pitch, timbre, etc. represented by the first feature vector can be determined, and the distribution probability of the above-mentioned features can be determined.

[0118] In some embodiments, the electronic device predicts the distribution information of the first feature vector by a deep neural network. The deep neural network includes a plurality of network layers connected in series. The electronic device inputs the first feature vector into the deep neural network. Then, the electronic device processes the first feature vector by a first network layer of the deep neural network to obtain output data of the network layer. For any intermediate network layer, the electronic device can process intermediate input data by the network layer to obtain output data of the network layer. The intermediate input data is a product of a plurality of intermediate data. The intermediate data is obtained by processing a weighted sum of the output data of the previous network layer and the voiceprint information by an activation function. The activation functions corresponding to the plurality of intermediate data are usually not completely the same. In other words, the input data of any intermediate network layer includes the output data of the previous network layer and the voiceprint information of the second audio. Then, the electronic device can predict the distribution information of the first feature vector according to the output data of the last network layer by the deep neural network. By processing the first feature vector and the voiceprint information of the second audio by the deep neural network with a multi-layer network structure, the distribution information of the first feature vector can be predicted more quickly and accurately, and the accuracy and efficiency of predicting the distribution information are both high.

[0119] Figure 5 is a schematic diagram of predicting distribution information according to an exemplary embodiment. As shown in Figure 5 , the deep neural network includes N network layers. The electronic device inputs the first feature vector into the network layer 1, processes the first feature vector by the network layer 1 to obtain output data y. Then, the electronic device processes a weighted sum of the output data y and the voiceprint information condition by two activation functions to obtain input data of the network layer 2: tanh(w1*y+w2*condition)*sigmoid(w3*y+w4*condition). The electronic device processes the input data by the network layer 2 to obtain output data z. Then, the electronic device processes the same by the network layer 3 to the network layer N to obtain output data k of the network layer N. Then, the electronic device predicts the distribution information, i.e., predicts the mean and variance, according to the output data k by the activation function layer (such as the softmax layer) of the deep neural network.

[0120] In step S304, the first feature vector is mapped to the latent space to obtain the latent feature vector based on the distribution information of the first feature vector.

[0121] In the embodiments of the present disclosure, after the electronic device obtains the distribution information of the first feature vector, the electronic device maps the first feature vector to a latent feature vector in a latent space according to the distribution information. The distribution information can indicate the distribution of the first feature vector in the latent space, such as Gaussian distribution, normal distribution, etc. The latent space is usually a low-dimensional space. According to the distribution information, the electronic device can determine the distribution of the first feature vector in the latent space, and then map the first feature vector to the latent feature vector in the latent space according to the distribution.

[0122] In some embodiments, the mean and the variance in the distribution information are both vectors. For ease of description, the mean is referred to as a mean vector and the variance is referred to as a variance vector below. The dimension of the mean vector and the dimension of the variance vector are both the same as the dimension of the first feature vector. The plurality of elements in the mean vector are used to represent the mean of the first feature vector in different dimensions. The plurality of elements in the variance vector are used to represent the variance of the first feature vector in different dimensions. For example, in the case where the first feature vector is a 256-dimensional vector, the mean vector and the variance vector are also 256-dimensional vectors. According to the mean vector and the variance vector in the distribution information, the electronic device determines a plurality of Gaussian distributions of the first feature vector in the latent space. The number of the plurality of Gaussian distributions is the same as the dimension of the first feature vector. In other words, the number of means in the mean vector (or the number of variances in the variance vector) is the number of Gaussian distributions determined by the electronic device. Since the formula of the Gaussian distribution contains the mean and the variance, the electronic device can obtain a Gaussian distribution by filling the mean and the variance in the same dimension into the formula of the Gaussian distribution. Then, the electronic device randomly samples the plurality of Gaussian distributions to obtain the latent feature vector. The dimension of the latent feature vector is the same as the number of the plurality of Gaussian distributions. The numerical value of the element in the latent feature vector is the numerical value obtained by randomly sampling the Gaussian distribution.

[0123] According to the distribution information of the first feature vector, the plurality of Gaussian distributions of the first feature vector in the latent space can be accurately determined. By randomly sampling the plurality of Gaussian distributions to obtain the latent feature vector, a latent feature vector similar to the first feature vector can be sampled in the Gaussian distribution, so that the latent feature vector can also represent the content information, the pitch information and the voiceprint information, and the accuracy and the amount of information represented by the latent feature vector are improved.

[0124] In step S305, the latent feature vector is decoded to obtain a third audio. The timbre of the third audio is the target timbre, and the content of the third audio is the same as that of the first audio.

[0125] In the embodiments of the present disclosure, since the latent feature vector can represent not only the content information and the pitch information of the first audio, but also the voiceprint information of the second audio. Therefore, the electronic device can reconstruct the third audio with the timbre being the target timbre by decoding the latent feature vector through the vocoder. The content of the third audio is the same as that of the first audio. Further, since the pitch information is considered in the above timbre conversion process, when the first audio is a singing voice, a song or the like with rich pitch, the details of the pitch can be preserved in the timbre conversion process, thereby ensuring the accuracy of the timbre conversion of the singing voice, the song or the like.

[0126] Optionally, the electronic device can also input the voiceprint information of the second audio to the vocoder to decode the latent feature vector by the vocoder referring to the voiceprint information, thereby further improving the accuracy of the timbre conversion.

[0127] In some embodiments, the vocoder includes but is not limited to HifiGAN (a generative adversarial network), nsf-HifiGAN (another generative adversarial network), or Vocos (a fast neural vocoder), etc., which are not limited in the embodiments of the present disclosure.

[0128] The above embodiments mainly introduce the process of the electronic device converting the timbre of the first audio. In some embodiments, the electronic device can convert the timbre of the first audio through a timbre conversion model. The timbre conversion model includes an encoding module, a mapping module and a decoding module.

[0129] The encoding module is configured to encode the content information and the pitch information. For example, the encoder includes a feature extraction model, a quantizer and an encoder. The electronic device can extract the audio feature vector of the first audio through the feature extraction model, and then map the audio features of the first audio to the content feature vector, i.e., the phoneme sequence, through the quantizer. The larger the codebook size of the quantizer, the more accurate the phoneme sequence obtained after mapping. The codebook size of the quantizer is the number of cluster centers. It should be noted that in the case of using different phoneme dictionaries for the above mapping, the phonemes in the obtained phoneme sequence may not be exactly the same. Therefore, in the case of using a non-international standard phoneme dictionary for the above mapping, the obtained phoneme sequence can also be referred to as a pseudo-phoneme sequence. The encoder is configured to encode the content information and the pitch information to obtain an intermediate feature vector.

[0130] The mapping module is configured to encode the voiceprint information of the second audio to the intermediate feature vector through a plurality of activation functions to obtain a first feature vector. In addition, the mapping module is also configured to determine the distribution information of the first feature vector, and map the first feature vector to a latent feature vector according to the distribution information.

[0131] The decoding module is used to decode the latent feature vectors; that is, the decoding module is used to reconstruct the third audio based on the latent feature vectors.

[0132] Optionally, the timbre conversion model also includes a voiceprint extraction module. The voiceprint extraction module is used to extract voiceprint information from the second audio.

[0133] Figure 6 This is a model architecture diagram illustrating a timbre conversion model according to an exemplary embodiment. For example... Figure 6 As shown, the audio-visual conversion model includes an encoding module, a mapping module, a voiceprint extraction module, and a decoding module. After the electronic device inputs the first audio to the encoding module, it can extract features from the first audio using the feature extraction model in the encoding module to obtain an audio feature vector. Optionally, the feature extraction model is Hubert (Hidden-unit Bert). Then, the electronic device maps the audio feature vector to a discrete content feature vector, i.e., a pseudo-phoneme sequence, through the quantizer in the encoding module. Next, the electronic device performs dimension alignment and vector fusion on the pseudo-phoneme sequence and the pitch information F0 of the first audio using an encoder to obtain a fused feature vector. Then, the electronic device encodes the fused feature vector using an attention mechanism through the encoder to obtain an intermediate feature vector. After the encoder obtains the intermediate feature vector, it can input the intermediate feature vector to the mapping module. Additionally, after the electronic device inputs the second audio to the voiceprint extraction module, it extracts the voiceprint information of the second audio using the voiceprint extraction model. Optionally, the voiceprint extraction model is Wespeaker. The voiceprint extraction model can also input the extracted voiceprint information into the mapping module. Accordingly, the electronic device, through the acoustic model in the mapping module, uses multiple activation functions to perform a nonlinear mapping on the weighted sum of the intermediate feature vector and voiceprint information, mapping the voiceprint information encoding and the intermediate feature vector into a first feature vector. Then, the electronic device predicts the distribution information of the first feature vector through the multi-layer network of the acoustic model, and maps the first feature vector to the latent space according to the distribution information to obtain a latent feature vector. The acoustic model can input the latent feature vector into the decoding module. Optionally, the acoustic model can be WaveNet based flow (a deep neural network based on a flow model), diffusion, or flow-matching, etc., and this embodiment is not limited in this respect. Accordingly, the electronic device can decode the latent feature vector according to the HifiGAN vocoder in the decoding module, thereby reconstructing the third audio. Optionally, the electronic device can also decode the latent feature vector according to HifiGAN, referring to the voiceprint information of the second audio, thereby improving the decoding accuracy.

[0134] It should be noted that in the above timbre conversion model, the feature extraction model, the quantizer and the voiceprint extraction model are already trained models. In other words, in the process of training the above timbre conversion model, only the parameters of the encoder, the acoustic model and the vocoder need to be updated, and the parameters of the feature extraction model, the quantizer and the voiceprint extraction model do not need to be updated. The training process of the timbre conversion model is described below in combination with the schematic diagram of training the timbre conversion model shown in FIG. 3. Figure 7 As shown in FIG. 3, the modules labeled with snowflake icons are modules that do not need to be updated in the training process. Figure 7

[0135] (1) The electronic device obtains sample audio and a sample latent feature vector. The sample latent feature vector is obtained by encoding the spectrogram of the sample audio. For example, the electronic device can encode the spectrogram of the sample audio into the sample latent feature vector through a spec encoder (a kind of encoder). Optionally, the spectrogram of the sample audio is a mel spectrogram.

[0136] The sample audio usually includes tens of millions or even hundreds of millions of reference audios. The sample audio usually covers audios of speakers of different regions, different genders and different ages, so that the timbre conversion model trained by the sample audio has the ability to convert the timbre of any audio to any timbre.

[0137] (2) The electronic device encodes the content information (content feature vector) and the pitch information F0 of the sample audio through the encoder in the encoding module to obtain a second feature vector; and the electronic device reversely maps the sample latent feature vector into a third feature vector based on the voiceprint information of the sample audio through the acoustic model in the mapping module. The second feature vector and the third feature vector can both express the content information and the pitch information of the sample audio. The electronic device decodes the sample latent feature vector through the vocoder in the decoding module to obtain a target audio. The content of the target audio is the content of the sample audio, and the timbre of the target audio is the timbre of the sample audio. Therefore, the training process of the timbre conversion model can be regarded as an audio reconstruction process.

[0138] (3) The electronic device determines a first training loss according to the second feature vector and the third feature vector. The first training loss is positively related to the distance between the second feature vector and the third feature vector. The electronic device determines a second training loss according to the sample audio and the target audio. The second training loss is negatively related to the similarity between the sample audio and the target audio.

[0139] ​(4) The electronic device jointly trains the encoder, the acoustic model, and the vocoder in the timbre conversion model according to the first training loss and the second training loss by using a back propagation algorithm until a training end condition is met. The training end condition can be that any training loss is less than a preset threshold or that the number of training times reaches a preset number, and the present disclosure does not limit this.

[0140] Optionally, the electronic device can further determine a third training loss according to the spectrogram of the target audio and the spectrogram of the sample audio. Then, the electronic device trains the timbre conversion model according to the first training loss, the second training loss, and the third training loss.

[0141] By training the timbre conversion model by using the above steps (1)-(4), the timbre conversion model not only has the ability to convert any audio to any timbre, but also guarantees the accuracy and efficiency of the timbre conversion of the timbre conversion model.

[0142] It should be noted that the encoding module, the mapping module, and the decoding module in the timbre conversion model all use a non-autoregressive architecture. Compared with the autoregressive structure, the timbre conversion model with the non-autoregressive architecture does not need to process the audio frame by frame, but can process multiple audio frames in the audio at one time, reducing the timbre conversion delay and improving the timbre conversion efficiency.

[0143] In addition, experiments have proved that the timbre conversion model trained by using the above training method can convert the timbre of any audio to the timbre of a reference audio according to the voiceprint information of the reference audio with a length of 3 seconds, realizing the conversion of the timbre with extremely low samples.

[0144] In addition, since the timbre conversion model can preserve the content information and the pitch information of the source audio during the timbre conversion process, when the timbre conversion model is used to convert the timbre of an audio with complex pitch (such as singing, songs, etc.), the pitch information in the audio can be well preserved, avoiding the loss of the pitch information of the song during the timbre conversion process, guaranteeing the accuracy of the timbre conversion, improving the fine granularity of the timbre conversion, and further making the converted audio more smooth and natural.

[0145] The embodiment of the present disclosure provides a timbre conversion method, which can encode content information, pitch information of first audio and voiceprint information of second audio into a feature vector in the process of converting the timbre of the first audio according to the timbre of the second audio, so that the feature vector contains the content and pitch of the first audio and the timbre of the second audio. Then, the feature vector is mapped into a latent feature vector in a latent space according to distribution information of the feature vector. Then, a third audio is decoded according to the latent feature vector. Through the above method, the source audio can be quickly and accurately converted into target audio with any timbre. Further, since the pitch information and content information of the source audio are retained in the conversion process, more detailed information in the audio can be retained while ensuring the accuracy of the timbre conversion, thereby improving the naturalness and fluency of the target audio.

[0146] All the optional technical solutions described above can be combined to form optional embodiments of the present disclosure, which will not be described one by one here.

[0147] Figure 8 is a block diagram of a timbre conversion device according to an exemplary embodiment. As shown in Figure 8 The device includes an encoding unit 801, a mapping unit 802 and a decoding unit 803.

[0148] The encoding unit 801 is configured to encode content information, pitch information of first audio and voiceprint information of second audio into a first feature vector, the first audio being audio with a timbre to be converted, the content information including phonemes corresponding to a plurality of audio frames in the first audio, the pitch information including pitches of the plurality of audio frames, the timbre of the second audio being a target timbre, the voiceprint information being used to represent the target timbre, the first feature vector representing content features, pitch features of the first audio and timbre features of the second audio.

[0149] The mapping unit 802 is configured to map the first feature vector to a latent space based on distribution information of the first feature vector to obtain a latent feature vector, the distribution information including a mean and a variance.

[0150] The decoding unit 803 is configured to decode the latent feature vector to obtain a third audio, the timbre of the third audio being the target timbre, and the content of the third audio being the same as that of the first audio.

[0151] In some embodiments, the mapping unit 802 is configured to determine a plurality of Gaussian distributions of the first feature vector in the latent space based on the distribution information of the first feature vector, the number of the plurality of Gaussian distributions being the same as the dimension of the first feature vector; and sample the plurality of Gaussian distributions respectively to obtain the latent feature vector, the dimension of the latent feature vector being the same as the number of the plurality of Gaussian distributions.

[0152] In some embodiments, the content information of the first audio is a content feature vector, and a plurality of elements in the content feature vector correspond to a plurality of audio frames in the first audio in a one-to-one manner. Figure 9 FIG. 8 is a block diagram of another timbre conversion device according to an example embodiment. Figure 9 As shown in FIG. 8, the device further includes:

[0153] The extraction unit 804 is configured to perform feature extraction on the first audio to obtain an audio feature vector, the audio feature vector being used to represent audio features of a plurality of audio frames in the first audio. For any audio frame, a plurality of elements capable of reflecting the audio features of the audio frame are determined from the audio feature vector, the audio features being capable of representing phonemes corresponding to the audio frame. A sub-feature vector composed of the plurality of elements is mapped to an element in the content feature vector.

[0154] In some embodiments, the extraction unit 804 is configured to respectively determine similarities between the sub-feature vector and clustering centers of a plurality of feature clusters, the plurality of feature clusters being obtained by clustering audio feature vectors of a plurality of reference audios, and the clustering center of a feature cluster being used to represent a phoneme corresponding to the feature cluster. Based on the similarities between the sub-feature vector and the plurality of clustering centers, a target clustering center is determined, the target clustering center being the clustering center with the highest similarity to the sub-feature vector. The target clustering center is mapped to the element in the content feature vector.

[0155] In some embodiments, the encoding unit 801 is configured to encode the content information and the pitch information of the first audio by an attention mechanism to obtain an intermediate feature vector. The intermediate feature vector and a weighted sum of the voiceprint information are non-linearly mapped by a plurality of activation functions to obtain a plurality of initial data. A product of the plurality of initial data is determined as the first feature vector.

[0156] In some embodiments, the content information is a content feature vector. The encoding unit 801 is configured to encode the pitch information of the first audio to obtain a pitch feature vector. The content feature vector and the pitch feature vector are dimensionally aligned so that the dimension of the content feature vector is the same as the dimension of the pitch feature vector. The content feature vector and the pitch feature vector are fused to obtain a fused feature vector. The fused feature vector is encoded by an attention mechanism to obtain the intermediate feature vector.

[0157] In some embodiments, the device further includes:

[0158] The prediction unit 805 is configured to input the first feature vector into a deep neural network, the deep neural network comprising a plurality of network layers connected in series, the deep neural network being used to predict distribution information of the first feature vector; the first feature vector is processed through a first network layer of the deep neural network to obtain output data of the network layer; for any intermediate network layer, the intermediate input data is processed through the network layer to obtain output data of the network layer, the intermediate input data being a product of a plurality of intermediate data, the intermediate data being obtained by processing a weighted sum of output data of a previous network layer and the voiceprint information through an activation function; and the distribution information of the first feature vector is predicted based on the output data of the last network layer.

[0159] In some embodiments, the timbre conversion method is performed by a timbre conversion model, the timbre conversion model comprising an encoding module, a mapping module, and a decoding module, the encoding module being used to encode content information and pitch information, the mapping module being used to map the first feature vector into a latent feature vector, and the decoding module being used to decode the latent feature vector; the apparatus further comprises:

[0160] The training unit 806 is configured to obtain a sample audio and a sample latent feature vector, the sample latent feature vector being obtained by encoding a spectrum graph of the sample audio; the content information and the pitch information of the sample audio are encoded through the encoding module to obtain a second feature vector; the sample latent feature vector is inversely mapped into a third feature vector based on voiceprint information of the sample audio through the mapping module; the sample latent feature vector is decoded through the decoding module to obtain a target audio; a first training loss is determined based on the second feature vector and the third feature vector, and a second training loss is determined based on the sample audio and the target audio; and the timbre conversion model is trained based on the first training loss and the second training loss.

[0161] The embodiments of the present disclosure provide a timbre conversion apparatus, which can encode content information and pitch information of a first audio and voiceprint information of a second audio into a feature vector in the process of converting the timbre of the first audio with reference to the timbre of the second audio, so that the feature vector contains not only the content and the pitch of the first audio but also the timbre of the second audio. Then, the feature vector is mapped into a latent feature vector in a latent space according to distribution information of the feature vector. Then, a third audio is decoded from the latent feature vector. Through the above method, the source audio can be quickly and accurately converted into the target audio with any timbre. Further, since the pitch information and the content information of the source audio are preserved in the conversion process, more detailed information in the audio can be preserved to improve the naturalness and fluency of the target audio while ensuring the accuracy of the timbre conversion.

[0162] It should be noted that the tone conversion apparatus provided in the above embodiments is only exemplified by the division of the above functional units. In actual application, the above functions can be completed by different functional units according to needs, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the above-described functions. In addition, the tone conversion apparatus and the tone conversion method provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be described here.

[0163] As to the tone conversion apparatus in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0164] Figure 10 is a block diagram of an electronic device according to an example embodiment. Generally, the electronic device 1000 includes a processor 1001 and a memory 1002.

[0165] The processor 1001 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1001 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1001 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1001 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 1001 can also include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.

[0166] The memory 1002 can include one or more computer-readable storage media. The computer-readable storage media can be non-transitory. The memory 1002 can also include high-speed random access memory and can include nonvolatile memory, such as one or more magnetic disk storage devices, optical storage devices, flash memory devices, or other nonvolatile solid-state storage devices. In some embodiments, the non-transitory computer-readable storage medium of the memory 1002 is used to store at least one program code for being executed by the processor 1001 to implement the timbre conversion method provided by the method embodiments of the present disclosure.

[0167] In some embodiments, the electronic device 1000 can further optionally include a peripheral device interface 1003 and at least one peripheral device. The processor 1001, the memory 1002, and the peripheral device interface 1003 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1003 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007, and a power supply 1008.

[0168] The peripheral device interface 1003 can be used to connect at least one peripheral device related to input / output (I / O) to the processor 1001 and the memory 1002. In some embodiments, the processor 1001, the memory 1002, and the peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1001, the memory 1002, and the peripheral device interface 1003 can be implemented on a separate chip or circuit board, and the present embodiment is not limited in this regard.

[0169] The radio frequency circuit 1004 is configured to transmit and receive RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1004 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 1004 converts electrical signals into electromagnetic signals for transmission, or vice versa. Optionally, the radio frequency circuit 1004 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a code chip set, a subscriber identity module card, and the like. The radio frequency circuit 1004 can communicate with other electronic devices through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to, a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G, and 5G), a wireless local area network, and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1004 can also include NFC (Near Field Communication) related circuitry, which is not limited by the present disclosure.

[0170] The display screen 1005 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1005 is a touch display screen, the display screen 1005 also has the ability to collect touch signals on or above the surface of the display screen 1005. The touch signals can be input as control signals to the processor 1001 for processing. At this time, the display screen 1005 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1005 can be one, arranged on the front panel of the electronic device 1000; in other embodiments, the display screen 1005 can be at least two, arranged on different surfaces of the electronic device 1000 or in a folding design; in still other embodiments, the display screen 1005 can be a flexible display screen, arranged on a curved surface or a folding surface of the electronic device 1000. Even, the display screen 1005 can also be arranged in an irregular shape other than a rectangle, i.e., a special-shaped screen. The display screen 1005 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), and the like.

[0171] The camera component 1006 is configured to capture images or videos. Optionally, the camera component 1006 includes a front-facing camera and a rear-facing camera. Generally, the front-facing camera is disposed on the front panel of the electronic device, and the rear-facing camera is disposed on the back of the electronic device. In some embodiments, the rear-facing camera is at least two, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function by fusing the main camera and the depth-of-field camera, the panorama shooting and VR (Virtual Reality) shooting function by fusing the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera component 1006 can further include a flash. The flash can be a single-color-temperature flash or a dual-color-temperature flash. The dual-color-temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0172] The audio circuit 1007 can include a microphone and a speaker. The microphone is configured to capture sound waves of a user and an environment, and convert the sound waves into an electrical signal input to the processor 1001 for processing or to the radio frequency circuit 1004 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, which are respectively disposed at different parts of the electronic device 1000. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is configured to convert an electrical signal from the processor 1001 or the radio frequency circuit 1004 into sound waves. The speaker can be a traditional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert an electrical signal into a sound wave audible to humans, but also convert an electrical signal into an inaudible sound wave to humans for ranging purposes. In some embodiments, the audio circuit 1007 can further include a headphone jack.

[0173] The power supply 1008 is configured to supply power to each component in the electronic device 1000. The power supply 1008 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1008 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0174] Those skilled in the art can understand that the structure shown in the above description is not a limitation on the electronic device 1000, and the electronic device 1000 can include more or fewer components than those shown in the figure, or combine some components, or use a different arrangement of components. Figure 10 The above description does not constitute a limitation on the electronic device 1000, and the electronic device 1000 can include more or fewer components than those shown in the figure, or combine some components, or use a different arrangement of components.

[0175] In an example embodiment, a computer readable storage medium, for example, a memory 1002 including instructions, is also provided, which can be executed by the processor 1001 of the electronic device 1000 to complete the above-mentioned timbre conversion method. Optionally, the computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0176] A computer program product including a computer program which, when executed by a processor, implements the above-mentioned timbre conversion method.

[0177] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including such departures from the present disclosure that come within known, accepted, and / or customary practice in the art to which the present disclosure pertains. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0178] It should be understood that the present disclosure is not limited to the precise structures as herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A timbre conversion method, characterized in that, The method includes: The content information, pitch information, and voiceprint information of the first audio are encoded into a first feature vector. The first audio is the audio whose timbre is to be converted. The content information includes phonemes corresponding to multiple audio frames in the first audio. The pitch information includes the pitch of the multiple audio frames. The timbre of the second audio is the target timbre. The voiceprint information is used to represent the target timbre. The first feature vector represents the content features, pitch features, and timbre features of the first audio and the second audio. Based on the distribution information of the first feature vector, multiple Gaussian distributions of the first feature vector in the latent space are determined. The number of the multiple Gaussian distributions is the same as the dimension of the first feature vector. The distribution information includes the mean and variance. Each of the multiple Gaussian distributions is sampled to obtain a latent feature vector, wherein the dimension of the latent feature vector is the same as the number of the multiple Gaussian distributions; The latent feature vector is decoded to obtain a third audio, the timbre of which is the target timbre, and the content of the third audio is the same as that of the first audio.

2. The timbre conversion method according to claim 1, characterized in that, The content information of the first audio is a content feature vector, and multiple elements in the content feature vector correspond one-to-one with multiple audio frames in the first audio. The method further includes: Feature extraction is performed on the first audio to obtain an audio feature vector, which is used to represent the audio features of multiple audio frames in the first audio. For any audio frame, multiple elements are determined from the audio feature vector. These multiple elements can reflect the audio features of the audio frame, and the audio features can represent the phonemes corresponding to the audio frame. The sub-feature vector composed of the multiple elements is mapped to the elements in the content feature vector.

3. The timbre conversion method according to claim 2, characterized in that, The step of mapping the sub-feature vector composed of the multiple elements to elements in the content feature vector includes: The similarity between the sub-feature vector and the cluster centers of multiple feature clusters is determined respectively. The multiple feature clusters are obtained by clustering the audio feature vectors of multiple reference audios. The cluster centers of the feature clusters are used to represent the phonemes corresponding to the feature clusters. Based on the similarity between the sub-feature vector and multiple cluster centers, a target cluster center is determined, which is the cluster center with the highest similarity to the sub-feature vector; The target cluster center is mapped to an element in the content feature vector.

4. The timbre conversion method according to claim 1, characterized in that, The step of encoding the content information, pitch information, and voiceprint information of the first audio into a first feature vector includes: The content and pitch information of the first audio are encoded using an attention mechanism to obtain an intermediate feature vector; Multiple initial data are obtained by nonlinearly mapping the weighted sum of the intermediate feature vector and the voiceprint information through multiple activation functions; The product of the multiple initial data is determined as the first feature vector.

5. The timbre conversion method according to claim 4, characterized in that, The content information is a content feature vector; The process of encoding the content and pitch information of the first audio file using an attention mechanism to obtain an intermediate feature vector includes: The pitch information of the first audio is encoded to obtain a pitch feature vector; The content feature vector and the pitch feature vector are dimension aligned so that the dimension of the content feature vector is the same as the dimension of the pitch feature vector. The content feature vector and the pitch feature vector are fused to obtain a fused feature vector; The intermediate feature vector is obtained by encoding the fused feature vector through the attention mechanism.

6. The timbre conversion method according to claim 1, characterized in that, The method further includes: The first feature vector is input into a deep neural network, which includes multiple network layers connected in series, and the deep neural network is used to predict the distribution information of the first feature vector. The first feature vector is processed through the first network layer of the deep neural network to obtain the output data of the network layer; For any intermediate network layer, the intermediate input data is processed through the network layer to obtain the output data of the network layer. The intermediate input data is the product of multiple intermediate data. The intermediate data is obtained by processing the weighted sum of the output data of the previous network layer and the voiceprint information through an activation function. Based on the output data of the last network layer, predict the distribution information of the first feature vector.

7. The timbre conversion method according to any one of claims 1-6, characterized in that, The timbre conversion method is executed by a timbre conversion model, which includes an encoding module, a mapping module, and a decoding module. The encoding module is used to encode the content information and the pitch information. The mapping module is used to map the first feature vector to the latent feature vector. The decoding module is used to decode the latent feature vector. The training process of the timbre conversion model includes: The sample audio and sample latent feature vector are obtained by encoding the spectrogram of the sample audio. The encoding module encodes the content and pitch information of the sample audio to obtain a second feature vector. The mapping module maps the latent feature vector of the sample audio to a third feature vector based on the voiceprint information of the sample audio. The target audio is obtained by decoding the latent feature vector of the sample through the decoding module. Based on the second feature vector and the third feature vector, a first training loss is determined, and based on the sample audio and the target audio, a second training loss is determined. The timbre conversion model is trained based on the first training loss and the second training loss.

8. A tone conversion device, characterized in that, The device includes: The encoding unit is configured to encode the content information, pitch information, and voiceprint information of the first audio into a first feature vector. The first audio is the audio whose timbre is to be converted. The content information includes phonemes corresponding to multiple audio frames in the first audio. The pitch information includes the pitch of the multiple audio frames. The timbre of the second audio is the target timbre. The voiceprint information is used to represent the target timbre. The first feature vector represents the content features, pitch features, and timbre features of the first audio. The mapping unit is configured to determine multiple Gaussian distributions of the first feature vector in the latent space based on the distribution information of the first feature vector, wherein the number of the multiple Gaussian distributions is the same as the dimension of the first feature vector, and the distribution information includes mean and variance; and to sample the multiple Gaussian distributions respectively to obtain latent feature vectors, wherein the dimension of the latent feature vectors is the same as the number of the multiple Gaussian distributions. The decoding unit is configured to decode the latent feature vector to obtain a third audio, the timbre of which is the target timbre, and the content of the third audio is the same as the content of the first audio.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory used to store the executable program code of the processor; The processor is configured to execute the program code to implement the timbre conversion method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the timbre conversion method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the timbre conversion method as described in any one of claims 1 to 7.

Images