Tone conversion methods, devices, electronic devices, storage media and software products

Abstract

The present disclosure relates to a method, device, electronic device, storage medium and program product for voice conversion. The method comprises: obtaining a training audio set, wherein the training audio set comprises a plurality of training audios; for a first training audio in the training audio set, obtaining a first content sequence corresponding to the first training audio, and obtaining a first audio coding sequence corresponding to the first training audio; obtaining a second audio coding sequence corresponding to the second training audio, wherein the first training audio and the second training audio correspond to the same speaker; processing the first content sequence and the second audio coding sequence through a voice conversion model to obtain a third audio coding sequence, wherein the voice conversion model adopts a network structure of a neural language model; and training the voice conversion model according to the first audio coding sequence and the third audio coding sequence.

Description

Technical Field

[0001] The present disclosure relates to the technical field of audio processing, and in particular to a training method for a timbre conversion model, a timbre conversion method, a training device for a timbre conversion model, a timbre conversion device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Voice conversion refers to a technical method that converts an input speech into the timbre of a specific speaker without changing the audio content.

[0003] Voice conversion technology has a wide range of application scenarios. In gaming entertainment, through voice conversion, recording staff can play different roles and achieve the desired sound effects, so that they can dub movies, animations, video games, etc. In human-computer interaction, by adjusting the voice to match the user's favorite role, the voice assistant can become more friendly and attractive. Voice conversion technology allows ordinary users to convert their voices into different styles or roles, and has a wide range of application scenarios.

[0004] Therefore, it is of great significance to improve the audio effect after timbre conversion. Summary of the invention

[0005] The present invention provides a technical solution for tone conversion.

[0006] According to one aspect of the present disclosure, a method for training a timbre conversion model is provided, wherein the timbre conversion model adopts a network structure of a neural language model, and the method comprises:

[0007] Acquire a training audio set, wherein the training audio set includes a plurality of training audios;

[0008] For a first training audio in the training audio set, obtaining a first content sequence corresponding to the first training audio, and obtaining a first audio coding sequence corresponding to the first training audio;

[0009] Obtaining a second audio code sequence corresponding to a second training audio, wherein the first training audio and the second training audio correspond to the same speaker;

[0010] Processing the first content sequence and the second audio code sequence by using the timbre conversion model to obtain a third audio code sequence;

[0011] The timbre conversion model is trained according to the first audio coding sequence and the third audio coding sequence.

[0012] In one possible implementation,

[0013] The method further comprises: obtaining a first emotion label corresponding to the first training audio;

[0014] The step of processing the first content sequence and the second audio coding sequence by the timbre conversion model to obtain a third audio coding sequence includes: processing the first content sequence, the second audio coding sequence and the first emotion tag by the timbre conversion model to obtain a third audio coding sequence.

[0015] In a possible implementation manner, obtaining a first content sequence corresponding to the first training audio includes:

[0016] Obtaining first content encoding information corresponding to the first training audio;

[0017] Based on the first content coding information, speech frames in the first training audio are classified respectively to obtain a first content sequence corresponding to the first training audio.

[0018] In one possible implementation,

[0019] The obtaining of the first audio code sequence corresponding to the first training audio comprises: performing L-level encoding on the first training audio to obtain the first audio code sequence corresponding to the first training audio;

[0020] The obtaining of the second audio code sequence corresponding to the second training audio includes: performing L-level encoding on the second training audio to obtain the second audio code sequence corresponding to the second training audio.

[0021] In a possible implementation, the step of processing the first content sequence, the second audio code sequence, and the first emotion tag by the timbre conversion model to obtain a third audio code sequence includes:

[0022] splicing the first content sequence, the second audio coding sequence, and the first emotion tag to obtain first splicing information;

[0023] The first splicing information is input into the timbre conversion model, and a third audio coding sequence is obtained through the timbre conversion model.

[0024] In a possible implementation, the training of the timbre conversion model according to the first audio coding sequence and the third audio coding sequence includes:

[0025] Determining a value of a loss function corresponding to the timbre conversion model according to difference information between coding information of the same level of corresponding speech frames in the first audio coding sequence and the third audio coding sequence, wherein the corresponding speech frames in the first audio coding sequence and the third audio coding sequence represent speech frames in the first audio coding sequence and the third audio coding sequence that are corresponding in time;

[0026] The timbre conversion model is trained according to the value of the loss function.

[0027] According to one aspect of the present disclosure, there is provided a method for timbre conversion, comprising:

[0028] Acquire a timbre conversion model trained by the timbre conversion model training method;

[0029] Get the audio to be converted and the target audio;

[0030] Obtaining a second content sequence corresponding to the audio to be converted;

[0031] Obtaining a fourth audio code sequence corresponding to the target timbre audio;

[0032] Processing the second content sequence and the fourth audio code sequence by using the timbre conversion model to obtain a fifth audio code sequence;

[0033] The fifth audio coding sequence is decoded by an audio decoder to obtain a synthesized audio corresponding to the audio to be converted and the target timbre audio.

[0034] In one possible implementation,

[0035] The method further comprises: obtaining a target emotion label;

[0036] The step of processing the second content sequence and the fourth audio coding sequence through the timbre conversion model to obtain a fifth audio coding sequence includes: processing the second content sequence, the fourth audio coding sequence and the target emotion label through the timbre conversion model to obtain a fifth audio coding sequence.

[0037] According to one aspect of the present disclosure, a training device for a timbre conversion model is provided, wherein the timbre conversion model adopts a network structure of a neural language model, and the device comprises:

[0038] A first acquisition module is used to acquire a training audio set, wherein the training audio set includes a plurality of training audios;

[0039] A first obtaining module, configured to obtain, for a first training audio in the training audio set, a first content sequence corresponding to the first training audio, and obtain a first audio coding sequence corresponding to the first training audio;

[0040] A second obtaining module, configured to obtain a second audio code sequence corresponding to a second training audio, wherein the first training audio and the second training audio correspond to the same speaker;

[0041] A first processing module, configured to process the first content sequence and the second audio code sequence by using the timbre conversion model to obtain a third audio code sequence;

[0042] A training module is used to train the timbre conversion model according to the first audio coding sequence and the third audio coding sequence.

[0043] In one possible implementation,

[0044] The device further comprises: a fifth obtaining module, configured to obtain a first emotion label corresponding to the first training audio;

[0045] The first processing module is used to process the first content sequence, the second audio coding sequence and the first emotion tag through the timbre conversion model to obtain a third audio coding sequence.

[0046] In a possible implementation manner, the first obtaining module is used to:

[0047] Obtaining first content encoding information corresponding to the first training audio;

[0048] Based on the first content coding information, speech frames in the first training audio are classified respectively to obtain a first content sequence corresponding to the first training audio.

[0049] In one possible implementation,

[0050] The first obtaining module is used to: perform L-level encoding on the first training audio to obtain a first audio code sequence corresponding to the first training audio;

[0051] The second obtaining module is used to: perform L-level encoding on the second training audio to obtain a second audio code sequence corresponding to the second training audio.

[0052] In a possible implementation manner, the first processing module is used to:

[0053] splicing the first content sequence, the second audio coding sequence, and the first emotion tag to obtain first splicing information;

[0054] The first splicing information is input into the timbre conversion model, and a third audio coding sequence is obtained through the timbre conversion model.

[0055] In a possible implementation, the training module is used to:

[0056] Determining a value of a loss function corresponding to the timbre conversion model according to difference information between coding information of the same level of corresponding speech frames in the first audio coding sequence and the third audio coding sequence, wherein the corresponding speech frames in the first audio coding sequence and the third audio coding sequence represent speech frames in the first audio coding sequence and the third audio coding sequence that are corresponding in time;

[0057] The timbre conversion model is trained according to the value of the loss function.

[0058] According to one aspect of the present disclosure, there is provided a timbre conversion device, comprising:

[0059] A second acquisition module is used for the timbre conversion model trained by the training device of the timbre conversion model;

[0060] The third acquisition module is used to acquire the audio to be converted and the target timbre audio;

[0061] A third obtaining module, used to obtain a second content sequence corresponding to the audio to be converted;

[0062] A fourth obtaining module, used to obtain a fourth audio code sequence corresponding to the target timbre audio;

[0063] A second processing module, configured to process the second content sequence and the fourth audio code sequence by using the timbre conversion model to obtain a fifth audio code sequence;

[0064] A decoding module is used to decode the fifth audio coding sequence through an audio decoder to obtain a synthesized audio corresponding to the audio to be converted and the target timbre audio.

[0065] In one possible implementation,

[0066] The device further comprises: a sixth obtaining module, configured to obtain a target emotion label;

[0067] The second processing module is used to process the second content sequence, the fourth audio coding sequence and the target emotion label through the timbre conversion model to obtain a fifth audio coding sequence.

[0068] According to one aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute the above method.

[0069] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0070] According to one aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in an electronic device, a processor in the electronic device executes the above method.

[0071] In an embodiment of the present disclosure, a timbre conversion model is implemented by adopting a network structure of a neural language model, a training audio set is obtained, wherein the training audio set includes multiple training audios, for a first training audio in the training audio set, a first content sequence corresponding to the first training audio is obtained, and a first audio code sequence corresponding to the first training audio is obtained, and a second audio code sequence corresponding to the second training audio is obtained, wherein the first training audio and the second training audio correspond to the same speaker, the first content sequence and the second audio code sequence are processed by the timbre conversion model to obtain a third audio code sequence, and the timbre conversion model is trained according to the first audio code sequence and the third audio code sequence, thereby applying the network structure of the neural language model to the timbre conversion task, that is, applying the network structure of the neural language model to the timbre conversion model, and directly capturing timbre information from the speech of the target speaker through the timbre conversion model. Since there is no need to extract the voiceprint features of the target speaker from the speech of the target speaker separately, the timbre conversion model provided by the embodiment of the present disclosure can more efficiently capture the timbre information of the target speaker from the speech of the target speaker. The timbre conversion model thus trained only requires a small amount (e.g., a few seconds) of the target speaker's audio to convert the audio to be converted into the timbre of the target speaker, and can improve the timbre similarity between the converted audio and the timbre of the target speaker, thereby achieving low-cost timbre conversion. In addition, in practical applications, it is difficult to find audio of different people speaking the same content. Therefore, when training the timbre conversion model, the disclosed embodiment uses non-parallel corpus, that is, the first training audio and the second training audio belong to the same speaker, thereby further reducing the difficulty of training the timbre conversion model.

[0072] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0073] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0075] Figure 1 A schematic diagram showing an overall block diagram of a timbre conversion model commonly used in the related art.

[0076] Figure 2 A flowchart of a method for training a timbre conversion model provided by an embodiment of the present disclosure is shown.

[0077] Figure 3 A schematic diagram of a content encoder provided by an embodiment of the present disclosure is shown.

[0078] Figure 4 A schematic diagram showing a method for training a timbre conversion model provided by an embodiment of the present disclosure.

[0079] Figure 5 A schematic diagram of a timbre conversion method provided by an embodiment of the present disclosure is shown.

[0080] Figure 6 A block diagram of a training device for a timbre conversion model provided by an embodiment of the present disclosure is shown.

[0081] Figure 7 A block diagram of an electronic device 1900 provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0082] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0083] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0084] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0085] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0086] In the related art, the timbre conversion model is usually implemented using a neural network. Figure 1 A schematic diagram showing an overall block diagram of a timbre conversion model commonly used in the related art.

[0087] like Figure 1 As shown, in the related art, the audio to be converted (i.e. Figure 1 The input audio in the audio is first passed through a content encoder to obtain a coded representation of the content information; the audio or ID (identification information) corresponding to the target speaker is passed through a speaker encoder to obtain a coded representation of the speaker; the coded representation of the content information and the coded representation of the speaker are simultaneously input into a timbre conversion model to obtain an output audio with the same content as the audio to be converted and the same timbre as the target speaker.

[0088] In the related art, the timbre conversion model can adopt the method of acoustic model plus vocoder. First, the acoustic model converts the input information into a spectrogram (such as Mel spectrum) of the target timbre speech, and then the vocoder generates output audio according to the spectrogram.

[0089] In the related art, in order to obtain high-quality synthesized audio with a timbre similar to that of the target speaker, a large amount of audio of the target speaker is required to extract speaker information and train the timbre conversion model, but data collection is quite difficult. Generally speaking, the amount of speech data of the target speaker takes from more than ten minutes to several hours. Therefore, in order to achieve the timbre conversion effect of a specific speaker, it takes a lot of cost to collect the speech data of the target speaker, which limits the application scope of timbre conversion and increases the cost of building a timbre conversion system.

[0090] In order to solve the problem of large target speaker data volume requirements, some papers have proposed methods for voice conversion using a small amount of target speaker speech. In papers such as AutoVC, YourTTS, and LVC-VC, a pre-trained speaker voiceprint feature extractor (Speaker Embedding Extractor) is used to extract voiceprint features (speaker embedding) from a target speaker's speech, and then the extracted voiceprint features are used to guide the voice conversion model to synthesize the target speaker's voice. Although this method reduces the amount of data required for the target speaker's speech, it is limited by the actual effect of the voiceprint feature extractor, and the synthesized audio often lacks in timbre similarity.

[0091] In order to solve the technical problems similar to those described above, the embodiment of the present disclosure provides a training method for a timbre conversion model, which implements the timbre conversion model by adopting the network structure of a neural language model, obtains a training audio set, wherein the training audio set includes multiple training audios, obtains a first content sequence corresponding to the first training audio, obtains a first audio code sequence corresponding to the first training audio, obtains a second audio code sequence corresponding to the second training audio, wherein the first training audio and the second training audio correspond to the same speaker, processes the first content sequence and the second audio code sequence by the timbre conversion model to obtain a third audio code sequence, and trains the timbre conversion model according to the first audio code sequence and the third audio code sequence, thereby applying the network structure of the neural language model to the timbre conversion task, that is, applying the network structure of the neural language model to the timbre conversion model, and directly captures timbre information from the speech of the target speaker by the timbre conversion model. Since there is no need to extract the voiceprint features of the target speaker from the speech of the target speaker separately, the timbre conversion model provided by the embodiment of the present disclosure can more efficiently capture the timbre information of the target speaker from the speech of the target speaker. The timbre conversion model trained in this way only requires a small amount (e.g., a few seconds) of the target speaker's audio to convert the audio to be converted into the timbre of the target speaker, and can improve the timbre similarity between the converted audio and the timbre of the target speaker, thereby achieving low-cost timbre conversion.

[0092] In addition, in practical applications, it is difficult to find audios of different people speaking the same content. Therefore, the disclosed embodiment uses non-parallel corpus when training the voice conversion model, that is, the first training audio and the second training audio belong to the same speaker, thereby further reducing the difficulty of training the voice conversion model.

[0093] Figure 2A flow chart of a method for training a timbre conversion model provided in an embodiment of the present disclosure is shown. In one possible implementation, the executor of the method for training a timbre conversion model may be a training device for the timbre conversion model. For example, the method for training a timbre conversion model may be executed by a terminal device or a server or other electronic device. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device or a wearable device. In some possible implementations, the method for training a timbre conversion model may be implemented by a processor calling computer-readable instructions stored in a memory. In an embodiment of the present disclosure, the timbre conversion model adopts the network structure of a neural language model. As Figure 2 As shown, the training method of the timbre conversion model includes steps S21 to S25.

[0094] In step S21, a training audio set is obtained, wherein the training audio set includes a plurality of training audios.

[0095] In step S22, for the first training audio in the training audio set, a first content sequence corresponding to the first training audio is obtained, and a first audio coding sequence corresponding to the first training audio is obtained.

[0096] In step S23, a second audio code sequence corresponding to a second training audio is obtained, wherein the first training audio and the second training audio correspond to the same speaker.

[0097] In step S24, the first content sequence and the second audio code sequence are processed by the timbre conversion model to obtain a third audio code sequence.

[0098] In step S25, the timbre conversion model is trained according to the first audio coding sequence and the third audio coding sequence.

[0099] In the disclosed embodiment, the timbre conversion model adopts the network structure of a neural language model (NLM). In some application scenarios, the neural language model can also be called a neural network language model (NNLM), etc., which is not limited here.

[0100] In the disclosed embodiment, the training audio set may represent an audio set for training a timbre conversion model. The first training audio may be any training audio in the training audio set, and the second training audio may be any audio in the training audio set whose speaker is the same as the first training audio, that is, the speaker in the first training audio and the second training audio is the same speaker. In an example, the first training audio may be represented by I, the second training audio may be represented by S, and the speakers corresponding to the first training audio I and the second training audio S may be represented by A.

[0101] In the embodiment of the present disclosure, the first content sequence may represent a content sequence corresponding to the first training audio. The number of content items in the first content sequence may be the same as the number of speech frames in the first training audio. For example, the first content sequence may be Represents, where T1 can represent the number of speech frames in the first training audio.

[0102] In a possible implementation, obtaining the first content sequence corresponding to the first training audio includes: obtaining first content coding information corresponding to the first training audio; and classifying speech frames in the first training audio based on the first content coding information to obtain the first content sequence corresponding to the first training audio.

[0103] In this implementation, the first content coding information may represent the content coding information corresponding to the first training audio. The first content coding information may include content coding information corresponding to each speech frame in the first training audio. Based on the first content coding information, the speech frames in the first training audio may be classified respectively to obtain a first content sequence corresponding to the first training audio. For example, the content coding information corresponding to each speech frame in the first content coding information may be K-means classified respectively to obtain a first content sequence. Of course, in addition to the K-means classification method, other classification methods may also be used, which are not limited here.

[0104] The first content sequence may include content categories corresponding to speech frames in the first training audio. For example, the first content sequence is c_t may represent a content category corresponding to the t-th speech frame in the first training audio, where 1≤t≤T1.

[0105] In this implementation, by obtaining the first content coding information corresponding to the first training audio, and based on the first content coding information, classifying the speech frames in the first training audio respectively, a first content sequence corresponding to the first training audio is obtained. The first content sequence thus obtained can be easily processed by a timbre conversion model with a network structure of a neural language model.

[0106] As an example of this implementation, the first content encoding information corresponding to the first training audio can be obtained through a content encoder, and the speech frames in the first content encoding information are respectively classified to obtain a first content sequence corresponding to the first content encoding information. The content encoder can include an Automatic Speech Recognition (ASR) model and a K-means classification module.

[0107] Among them, the speech recognition model can adopt models such as Transformer and Conformer, which are not limited here. The speech recognition model can be used to extract content coding information corresponding to the input audio. For example, the speech recognition model can be used to extract the first content coding information corresponding to the first training audio. Among them, after the audio is input into the speech recognition model, the output of the hidden layer of the speech recognition model can be used as the content coding information corresponding to the input audio. For example, the output of a specified hidden layer of the speech recognition model can be used as the content coding information corresponding to the input audio; for example, the content coding information corresponding to the input audio can be obtained by merging the outputs of at least two specified hidden layers of the speech recognition model; and so on.

[0108] The K-means classification module can be used to classify the speech frames in the content coding information corresponding to the input audio, and obtain the content sequence corresponding to the content coding information. For example, the K-means classification module can be used to classify the speech frames in the first content coding information, and obtain the first content sequence corresponding to the first content coding information.

[0109] In one example, before obtaining the first content sequence corresponding to the first training audio, a speech recognition model and a K-means classification module may be pre-trained.

[0110] The speech recognition model can be trained through a series of "audio-text pairs". The speech recognition model can be trained using a training method in the related art, which will not be described in detail here. The trained speech recognition model can be used to extract content encoding information.

[0111] The K-means classification module can be trained based on the output of the hidden layer of the speech recognition model. For example, the content encoding information corresponding to all the training audios in the training audio set can be extracted through the trained speech recognition model as the training data of the K-means classification module. The K-means classification module can be used for K classification.

[0112] Figure 3 Schematic diagram of a content encoder provided by an embodiment of the present disclosure is shown. Figure 3As shown, the content encoder may include an ASR model and a K-means classification module. After the speech is input into the speech recognition model, the output of the hidden layer of the speech recognition model may be input into the K-means classification module, and the content sequence corresponding to the input speech is obtained through the K-means classification module.

[0113] In the embodiment of the present disclosure, the first audio coding sequence may represent an audio coding sequence corresponding to the first training audio, and the second audio coding sequence may represent an audio coding sequence corresponding to the second training audio. The audio coding sequence may represent a sequence obtained by encoding the audio.

[0114] In a possible implementation, obtaining a first audio coding sequence corresponding to the first training audio includes: performing L-level encoding on the first training audio to obtain a first audio coding sequence corresponding to the first training audio; obtaining a second audio coding sequence corresponding to the second training audio includes: performing L-level encoding on the second training audio to obtain a second audio coding sequence corresponding to the second training audio.

[0115] Here, L may represent the number of levels for encoding the first training audio and the second training audio.

[0116] As an example of this implementation, L may be an integer greater than or equal to 2.

[0117] As an example of this implementation, the first training audio can be encoded at level L by an audio encoder to obtain a first audio encoding sequence corresponding to the first training audio, and the second training audio can be encoded at level L by an audio encoder to obtain a second audio encoding sequence corresponding to the second training audio. Among them, the audio encoder can adopt models such as EnCodec and SoundStream, which are not limited here. In some application scenarios, the audio encoder can also be called an audio codec, a neural network audio encoder, a neural audio codec (neural audio codec), etc., which are not limited here.

[0118] In this implementation, the audio coding sequence may be a token sequence of level L. For example, the audio coding sequence may be , where T can represent the total number of frames output by the audio encoder. Through the corresponding audio decoder, the audio coding sequence can be restored to the input audio while ensuring the restored sound quality. For example, the second audio coding sequence T2 may represent the total number of frames outputted by the second training audio after passing through the audio encoder.

[0119] In a possible implementation, the method also includes: obtaining a first emotion label corresponding to the first training audio; processing the first content sequence and the second audio coding sequence through the timbre conversion model to obtain a third audio coding sequence, including: processing the first content sequence, the second audio coding sequence and the first emotion label through the timbre conversion model to obtain a third audio coding sequence.

[0120] As an example of this implementation, an emotion tag corresponding to the training audio may be extracted by an emotion recognition (ER). For example, a first emotion tag corresponding to the first training audio may be extracted by an emotion recognition.

[0121] In one example, the emotion recognition model can be pre-trained. In this example, the emotion recognition model can be trained through a series of "audio-emotion label pairs". Among them, the input of the emotion recognition model can be speech features (such as MFCC or FBank, etc.) or audio, and the output layer can correspond to different emotion labels. In this example, the emotion recognition model can be trained using the training method in the relevant technology, which will not be repeated here. The trained emotion recognition model can be used as an emotion extractor to extract the emotion label corresponding to the audio.

[0122] Among them, the emotion recognizer can adopt models such as CRNN and wav2vec2, which are not limited here.

[0123] In one example, the sentiment label may be represented by e.

[0124] In one example, the first content sequence can be represented by C_1, the second audio code sequence can be represented by Q_2, and the first emotion tag can be represented by e_1. The timbre conversion model can take {C_1, Q_2, e_1} as input and generatively output the third audio code sequence

[0125] The timbre conversion methods in related technologies can usually only change the timbre of the input audio, but cannot control the emotion of the converted audio. In this implementation, by obtaining the first emotion label corresponding to the first training audio, the first content sequence, the second audio coding sequence and the first emotion label are processed by the timbre conversion model to obtain a third audio coding sequence, thereby adding the emotion label to the input of the timbre conversion model, and uniformly modeling the audio to be converted, the target speaker audio and the emotion label, which can explicitly control the emotion of the converted speech, thereby achieving low-cost and emotion-controllable timbre conversion.

[0126] The timbre conversion model trained by this implementation method can extract information from the input audio to be converted, the target timbre audio (i.e., the voice of the target speaker), and the emotion label, and generatively synthesize the audio after timbre conversion without the need for an additional voiceprint feature extractor, thereby improving the timbre similarity of the converted audio.

[0127] Emotionally controllable timbre conversion technology can be applied to many application scenarios. For example, emotionally controllable timbre conversion technology can help users generate voices with more natural tones and richer emotions. This technology can also be applied in areas such as audiobook production to enhance the emotional expression of voices.

[0128] In one example, the input of the voice conversion model can be called a prompt.

[0129] In the embodiments of the present disclosure, a Transformer model containing only a decoder structure (Decoder-OnlyTransformer) can be used as a timbre conversion model. This structure is similar to a commonly used neural language model and is composed of multiple layers of transformers.

[0130] In a possible implementation, the processing of the first content sequence, the second audio coding sequence and the first emotion tag by the timbre conversion model to obtain a third audio coding sequence includes: splicing the first content sequence, the second audio coding sequence and the first emotion tag to obtain first splicing information; inputting the first splicing information into the timbre conversion model, and obtaining the third audio coding sequence by the timbre conversion model.

[0131] For example, the first content sequence is The second audio code sequence is The first emotion label is e, then the first concatenation information can be in, <sep>It is a preset special symbol used to separate different information.

[0132] In the disclosed embodiment, the training goal of the timbre conversion model is to make the third audio code sequence O output by the timbre conversion model as similar as possible to the first audio code sequence Q_1 corresponding to the first training audio.

[0133] In a possible implementation, since the audio code sequence consists of L levels, and the neural language model is an auto-regressive (AR) model, the timbre conversion model can output L-level codes through different methods. For example, the timbre conversion model can output only one code in each iteration, and output L codes corresponding to time t in L consecutive iterations, and then generate L codes corresponding to time t+1. For another example, the timbre conversion model can simultaneously output L codes at time t in each iteration, and then simultaneously output L codes at time t+1 in the next iteration.

[0134] In the disclosed embodiment, the value of the loss function corresponding to the timbre conversion model can be determined based on the difference information between the first audio coding sequence and the third audio coding sequence, and the parameters of the timbre conversion model can be updated based on the value of the loss function.

[0135] In a possible implementation, the training of the timbre conversion model based on the first audio coding sequence and the third audio coding sequence includes: determining the value of the loss function corresponding to the timbre conversion model based on difference information between the same level of coding information of corresponding speech frames in the first audio coding sequence and the third audio coding sequence, wherein the corresponding speech frames in the first audio coding sequence and the third audio coding sequence represent speech frames in the first audio coding sequence and the third audio coding sequence that correspond in time; and training the timbre conversion model based on the value of the loss function.

[0136] For example, the first audio coding sequence is The third audio code sequence is The value of the loss function corresponding to the timbre conversion model may be the sum of the cross entropies between the same level of coding information of the corresponding speech frames in the first audio coding sequence and the third audio coding sequence.

[0137] For example, in the first audio coding sequence and the third audio coding sequence is the coding information of the corresponding speech frame, and and the third audio coding sequence is the same level of coding information of the corresponding speech frame, the first audio coding sequence and the third audio coding sequence is the same level of coding information of the corresponding speech frame; for example, and the third audio coding sequence is the coding information of the corresponding speech frame, and and the third audio coding sequence The same level of coding information as the corresponding speech frame; and so on.

[0138] In this implementation, the value of the loss function corresponding to the timbre conversion model is determined based on the difference information between the same level of coding information of corresponding speech frames in the first audio coding sequence and the third audio coding sequence, and the timbre conversion model is trained based on the value of the loss function, thereby improving the timbre similarity after timbre conversion by the timbre conversion model.

[0139] The disclosed embodiment also provides a timbre conversion method, including: obtaining a timbre conversion model trained by the timbre conversion model training method; obtaining audio to be converted and target timbre audio; obtaining a second content sequence corresponding to the audio to be converted; obtaining a fourth audio coding sequence corresponding to the target timbre audio; processing the second content sequence and the fourth audio coding sequence through the timbre conversion model to obtain a fifth audio coding sequence; decoding the fifth audio coding sequence through an audio decoder to obtain a synthesized audio corresponding to the audio to be converted and the target timbre audio.

[0140] The audio to be converted may represent the audio to be converted into the timbre of the target speaker. The target timbre audio may represent the audio of the target speaker, that is, the speaker corresponding to the target timbre audio may be the target speaker. The audio length of the target timbre audio may be short, for example, only a few seconds. In most application scenarios, the speaker corresponding to the audio to be converted is not the target speaker.

[0141] In a possible implementation manner, the audio to be converted may be input into a content encoder, and the second content sequence corresponding to the audio to be converted may be obtained through the content encoder.

[0142] In a possible implementation, the target timbre audio may be input into an audio encoder, and a fourth audio coding sequence corresponding to the target timbre audio may be obtained through the audio encoder.

[0143] In the embodiment of the present disclosure, during the use phase of the timbre conversion model, the input of the timbre conversion model may include a second content sequence corresponding to the audio to be converted and a fourth audio coding sequence corresponding to the target timbre audio. The second content sequence may represent a content sequence corresponding to the audio to be converted, and the fourth audio coding sequence may represent an audio coding sequence corresponding to the target timbre audio.

[0144] The fifth audio coding sequence output by the timbre conversion model can be decoded by an audio decoder to obtain synthesized audio.

[0145] In a possible implementation, the method also includes: obtaining a target emotion tag; the processing of the second content sequence and the fourth audio coding sequence through the timbre conversion model to obtain a fifth audio coding sequence includes: processing the second content sequence, the fourth audio coding sequence and the target emotion tag through the timbre conversion model to obtain a fifth audio coding sequence.

[0146] In this implementation, during the use phase of the timbre conversion model, the input of the timbre conversion model may also include a target emotion label.

[0147] As an example of this implementation, the target emotion tag may be an emotion tag specified by a user.

[0148] As another example of this implementation, the target emotion tag may be an emotion tag corresponding to the audio to be converted. That is, the target emotion tag may be an emotion tag obtained by performing emotion recognition on the audio to be converted. For example, an emotion recognizer may be used to perform emotion recognition on the audio to be converted to obtain the target emotion tag.

[0149] In a possible implementation, the second content sequence, the fourth audio code sequence and the target emotion tag may be concatenated to obtain second concatenated information, and the second concatenated information may be input into a timbre conversion model, and a fifth audio code sequence may be output through the timbre conversion model.

[0150] The following describes the training method of the timbre conversion model and the timbre conversion method provided by the embodiments of the present disclosure through a specific application scenario.

[0151] 1. Training Phase

[0152] 1. Training the Content Encoder

[0153] The content encoder may include a speech recognition model and a K-means classification module. Among them, Conformer can be used as the speech recognition model. The speech recognition model can be used to extract content coding information corresponding to the input audio. The K-means classification module can be used to classify the speech frames in the content coding information corresponding to the input audio, and obtain the content sequence corresponding to the content coding information.

[0154] Among them, the speech recognition model can be trained through a series of "audio-text pairs".

[0155] The K-means classification module can be trained based on the output of the hidden layer of the speech recognition model. For example, the content encoding information corresponding to all training audios in the training audio set can be extracted through the trained speech recognition model as training data for the K-means classification module.

[0156] 2. Training the audio encoder

[0157] The audio encoder can be used to extract an audio coding sequence corresponding to the audio.

[0158] Among them, EnCodec can be used as the audio encoder.

[0159] 3. Training the emotion recognizer

[0160] The emotion recognition model can be trained through a series of "audio-emotion label pairs". The input of the emotion recognition model can be speech features (such as MFCC or FBank, etc.) or audio, and the output layer can correspond to different emotion labels. The trained emotion recognition model can be used as an emotion extractor to extract the emotion label corresponding to the audio.

[0161] Among them, wav2vec2 can be used as an emotion recognizer.

[0162] 4. Training the timbre conversion model

[0163] All training audios in the training audio set are processed by the content encoder, audio encoder and emotion recognizer respectively to obtain the corresponding content sequence, audio coding sequence and emotion label.

[0164] Figure 4 A schematic diagram of a method for training a timbre conversion model provided by an embodiment of the present disclosure is shown. Figure 4 As shown, for the first training audio in the training audio set, a first content sequence corresponding to the first training audio can be obtained, a first audio coding sequence corresponding to the first training audio can be obtained, and a first emotion label corresponding to the first training audio can be obtained. A second audio coding sequence corresponding to the second training audio can be obtained, wherein the first training audio and the training audio correspond to the same speaker. The first content sequence, the second audio coding sequence and the first emotion label can be spliced ​​to obtain first splicing information, and the first splicing information can be input into a timbre conversion model to obtain a third audio coding sequence through the timbre conversion model. The value of the loss function corresponding to the timbre conversion model can be determined based on the difference information between the same level of coding information of corresponding speech frames in the first audio coding sequence and the third audio coding sequence, and the timbre conversion model can be trained based on the value of the loss function.

[0165] 2. Use Phase

[0166] Figure 5 FIG. 1 is a schematic diagram showing a method for converting timbre provided by an embodiment of the present disclosure. Figure 5 As shown, a second content sequence corresponding to the audio to be converted can be obtained through a content encoder, and a fourth audio code sequence corresponding to the target timbre audio can be obtained through an audio encoder. A target emotion tag specified by a user can be obtained. The second content sequence, the fourth audio code sequence and the target emotion tag can be spliced ​​to obtain second splicing information. The second splicing information can be input into a timbre conversion model, and a fifth audio code sequence can be output through the timbre conversion model. The fifth audio code sequence can be decoded by an audio decoder to obtain a synthesized audio.

[0167] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0168] In addition, the present disclosure also provides a training device for a timbre conversion model, a timbre conversion device, an electronic device, a computer-readable storage medium, and a computer program product, all of which can be used to implement any timbre conversion model training method or timbre conversion method provided by the present disclosure. The corresponding technical solutions and technical effects can be found in the corresponding records in the method part and will not be repeated here.

[0169] Figure 6 FIG. 1 is a block diagram of a training device for a timbre conversion model provided by an embodiment of the present disclosure. The timbre conversion model adopts a network structure of a neural language model. Figure 6 As shown, the training device of the timbre conversion model includes:

[0170] A first acquisition module 61 is used to acquire a training audio set, wherein the training audio set includes a plurality of training audios;

[0171] A first obtaining module 62 is used to obtain, for a first training audio in the training audio set, a first content sequence corresponding to the first training audio, and obtain a first audio code sequence corresponding to the first training audio;

[0172] A second obtaining module 63 is used to obtain a second audio code sequence corresponding to a second training audio, wherein the first training audio and the second training audio correspond to the same speaker;

[0173] A first processing module 64, configured to process the first content sequence and the second audio code sequence by using the timbre conversion model to obtain a third audio code sequence;

[0174] The training module 65 is used to train the timbre conversion model according to the first audio coding sequence and the third audio coding sequence.

[0175] In one possible implementation,

[0176] The device further comprises: a fifth obtaining module, configured to obtain a first emotion label corresponding to the first training audio;

[0177] The first processing module 64 is used to process the first content sequence, the second audio code sequence and the first emotion tag through the timbre conversion model to obtain a third audio code sequence.

[0178] In a possible implementation, the first obtaining module 62 is used to:

[0179] Obtaining first content encoding information corresponding to the first training audio;

[0180] Based on the first content coding information, speech frames in the first training audio are classified respectively to obtain a first content sequence corresponding to the first training audio.

[0181] In one possible implementation,

[0182] The first obtaining module 62 is used to: perform L-level encoding on the first training audio to obtain a first audio code sequence corresponding to the first training audio;

[0183] The second obtaining module 63 is used to perform L-level encoding on the second training audio to obtain a second audio code sequence corresponding to the second training audio.

[0184] In a possible implementation, the first processing module 64 is used to:

[0185] splicing the first content sequence, the second audio coding sequence, and the first emotion tag to obtain first splicing information;

[0186] The first splicing information is input into the timbre conversion model, and a third audio coding sequence is obtained through the timbre conversion model.

[0187] In a possible implementation, the training module 65 is used to:

[0188] Determining a value of a loss function corresponding to the timbre conversion model according to difference information between coding information of the same level of corresponding speech frames in the first audio coding sequence and the third audio coding sequence, wherein the corresponding speech frames in the first audio coding sequence and the third audio coding sequence represent speech frames in the first audio coding sequence and the third audio coding sequence that are corresponding in time;

[0189] The timbre conversion model is trained according to the value of the loss function.

[0190] The present disclosure also provides a device for converting timbre, including:

[0191] A second acquisition module is used for the timbre conversion model trained by the training device of the timbre conversion model;

[0192] The third acquisition module is used to acquire the audio to be converted and the target timbre audio;

[0193] A third obtaining module, used to obtain a second content sequence corresponding to the audio to be converted;

[0194] A fourth obtaining module, used to obtain a fourth audio code sequence corresponding to the target timbre audio;

[0195] A second processing module, configured to process the second content sequence and the fourth audio code sequence by using the timbre conversion model to obtain a fifth audio code sequence;

[0196] A decoding module is used to decode the fifth audio coding sequence through an audio decoder to obtain a synthesized audio corresponding to the audio to be converted and the target timbre audio.

[0197] In one possible implementation,

[0198] The device further comprises: a sixth obtaining module, configured to obtain a target emotion label;

[0199] The second processing module is used to process the second content sequence, the fourth audio coding sequence and the target emotion label through the timbre conversion model to obtain a fifth audio coding sequence.

[0200] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. Its specific implementation and technical effects can refer to the description of the above method embodiments. For the sake of brevity, they will not be repeated here.

[0201] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a non-volatile computer-readable storage medium, or may be a volatile computer-readable storage medium.

[0202] The embodiment of the present disclosure further provides a computer program, including a computer-readable code. When the computer-readable code is executed in an electronic device, a processor in the electronic device executes the above method.

[0203] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in an electronic device, a processor in the electronic device executes the above method.

[0204] An embodiment of the present disclosure also provides an electronic device, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute the above method.

[0205] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0206] Figure 7 1 is a block diagram of an electronic device 1900 provided in an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal. Figure 7 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0207] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (MacOS X TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0208] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0209] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0210] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0211] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0212] The computer program instructions for performing the disclosed operation may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of a computer-readable program instruction, and the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0213] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0214] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0215] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0216] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0217] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0218] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0219] If the technical solution of the embodiment of the present disclosure involves personal information, the product using the technical solution of the embodiment of the present disclosure has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of the embodiment of the present disclosure involves sensitive personal information, the product using the technical solution of the embodiment of the present disclosure has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as the camera, a clear and obvious sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he agrees to collect his personal information; or on the device for processing personal information, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained by means of pop-up information or by asking the individual to upload his personal information by himself; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0220] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.< / sep>

Claims

1. A method for training a timbre conversion model, characterized in that: The timbre conversion model adopts the network structure of a neural language model, and the method includes: Acquire a training audio set, wherein the training audio set includes a plurality of training audios; For a first training audio in the training audio set, obtaining a first content sequence corresponding to the first training audio, and performing L-level encoding on the first training audio to obtain a first audio encoding sequence corresponding to the first training audio; Performing L-level encoding on the second training audio to obtain a second audio code sequence corresponding to the second training audio, wherein the first training audio and the second training audio correspond to the same speaker; Processing the first content sequence and the second audio code sequence by using the timbre conversion model to obtain a third audio code sequence; According to the difference information between the same level of coding information of the corresponding speech frames in the first audio coding sequence and the third audio coding sequence, the value of the loss function corresponding to the timbre conversion model is determined, and according to the value of the loss function, the timbre conversion model is trained, wherein the corresponding speech frames in the first audio coding sequence and the third audio coding sequence represent the speech frames in the first audio coding sequence and the third audio coding sequence that correspond in time.

2. The method according to claim 1, characterized in that The method further comprises: obtaining a first emotion label corresponding to the first training audio; The step of processing the first content sequence and the second audio coding sequence by the timbre conversion model to obtain a third audio coding sequence includes: processing the first content sequence, the second audio coding sequence and the first emotion tag by the timbre conversion model to obtain a third audio coding sequence.

3. The method according to claim 1 or 2, characterized in that: The obtaining of a first content sequence corresponding to the first training audio includes: Obtaining first content encoding information corresponding to the first training audio; Based on the first content coding information, speech frames in the first training audio are classified respectively to obtain a first content sequence corresponding to the first training audio.

4. The method according to claim 2, characterized in that: The step of processing the first content sequence, the second audio code sequence, and the first emotion tag by the timbre conversion model to obtain a third audio code sequence includes: splicing the first content sequence, the second audio coding sequence, and the first emotion tag to obtain first splicing information; The first splicing information is input into the timbre conversion model, and a third audio coding sequence is obtained through the timbre conversion model.

5. A timbre conversion method, characterized in that: include: Obtaining a timbre conversion model trained by the timbre conversion model training method according to any one of claims 1 to 4; Get the audio to be converted and the target audio; Obtaining a second content sequence corresponding to the audio to be converted; Obtaining a fourth audio code sequence corresponding to the target timbre audio; Processing the second content sequence and the fourth audio code sequence by using the timbre conversion model to obtain a fifth audio code sequence; The fifth audio coding sequence is decoded by an audio decoder to obtain a synthesized audio corresponding to the audio to be converted and the target timbre audio.

6. The method according to claim 5, characterized in that The method further comprises: obtaining a target emotion label; The step of processing the second content sequence and the fourth audio coding sequence through the timbre conversion model to obtain a fifth audio coding sequence includes: processing the second content sequence, the fourth audio coding sequence and the target emotion label through the timbre conversion model to obtain a fifth audio coding sequence.

7. A training device for a timbre conversion model, characterized in that: The timbre conversion model adopts the network structure of a neural language model, and the device includes: A first acquisition module is used to acquire a training audio set, wherein the training audio set includes a plurality of training audios; A first obtaining module is configured to obtain, for a first training audio in the training audio set, a first content sequence corresponding to the first training audio, and perform L-level encoding on the first training audio to obtain a first audio code sequence corresponding to the first training audio; A second obtaining module is used to perform L-level encoding on the second training audio to obtain a second audio code sequence corresponding to the second training audio, wherein the first training audio and the second training audio correspond to the same speaker; A first processing module, configured to process the first content sequence and the second audio code sequence by using the timbre conversion model to obtain a third audio code sequence; A training module is used to determine the value of the loss function corresponding to the timbre conversion model based on the difference information between the same level of coding information of the corresponding speech frames in the first audio coding sequence and the third audio coding sequence, and train the timbre conversion model based on the value of the loss function, wherein the corresponding speech frames in the first audio coding sequence and the third audio coding sequence represent the speech frames in the first audio coding sequence and the third audio coding sequence that correspond in time.

8. A timbre conversion device, characterized in that: include: A second acquisition module, used to acquire the timbre conversion model trained by the timbre conversion model training device according to claim 7; A third acquisition module is used to acquire the audio to be converted and the target timbre audio; A third obtaining module, used to obtain a second content sequence corresponding to the audio to be converted; A fourth obtaining module, used to obtain a fourth audio code sequence corresponding to the target timbre audio; A second processing module, configured to process the second content sequence and the fourth audio code sequence by using the timbre conversion model to obtain a fifth audio code sequence; A decoding module is used to decode the fifth audio coding sequence through an audio decoder to obtain a synthesized audio corresponding to the audio to be converted and the target timbre audio.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing executable instructions; The one or more processors are configured to call executable instructions stored in the memory to execute the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

11. A computer program product comprising computer readable code, or a non-volatile computer readable storage medium carrying computer readable code, characterized in that: When the computer readable code is executed in an electronic device, a processor in the electronic device executes the method according to any one of claims 1 to 6.

Images