Method, apparatus, device and medium for model training and tone conversion

The timbre conversion model training method improves timbre conversion accuracy and reliability by using a network and vocoder to separate timbre and semantic features, enabling unsupervised training and preserving original audio content.

JP2024547129APending Publication Date: 2024-12-26BIGO TECH PTE LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024538190
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-12-22
Filing Date
2022-12-20
Publication Date
2024-12-26

AI Technical Summary

Technical Problem

Current timbre conversion technologies face challenges in accurately preserving the original audio content while converting the timbre of voices, leading to reduced reliability and naturalness in synthesized audio data.

Method used

A timbre conversion model training method that utilizes a timbre extraction network, a removed timbre network, and a vocoder to extract and separate timbre and semantic features, allowing for unsupervised training without pre-labeling sample audio data, thereby improving timbre accuracy and conversion reliability.

Benefits of technology

The method enhances the accuracy and reliability of timbre conversion by accurately extracting timbre and semantic features, resulting in synthesized audio data that closely resembles the target speaker's voice while maintaining the original content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024547129000001_ABST
    Figure 2024547129000001_ABST
Patent Text Reader

Abstract

The present application provides a model training and timbre conversion method, apparatus, device and medium. The timbre extraction network can obtain the first timbre feature of the input sample audio data, which is advantageous for accurately obtaining the timbre information of the input sample audio data, and subsequently obtaining synthetic audio data according to the timbre feature, thereby improving the accuracy of the timbre of the synthetic audio data. The timbre removal network can obtain the first semantic feature of the sample audio data based on the first timbre feature, which realizes accurate acquisition of the features related to the pronunciation content of the sample audio data regardless of the speaker's timbre, and subsequently obtaining synthetic audio data according to the first semantic feature, thereby ensuring the accuracy of the pronunciation content of the synthetic audio data. After obtaining a trained timbre conversion model, the timbre conversion is performed according to the timbre conversion model, thereby improving the conversion effect and reliability of the timbre conversion.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This application claims priority to a Chinese patent application with application number 202111577618.0 filed with the China Patent Office on December 22, 2021, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the technical field of natural language understanding, and in particular to a method, apparatus, device and medium for model training and tone conversion. [Background technology]

[0003] Tone conversion technology is an important technology in the audio field, and is widely applied in the fields of audio content generation, entertainment audio creation, and private communication. Tone conversion technology refers to converting the voice tone of an original audio into the voice tone of another speaker. In the process of tone conversion, it is necessary to ensure that the voice tone of the audio after tone conversion is similar to the voice tone of the other speaker, and the content of the audio is kept intact and unchanged. The difficulty of tone conversion technology is how to convert the tone while keeping the content information of the original audio.

[0004] In view of this, how to perform tone conversion and obtain a stable tone conversion effect is a technical problem that must be solved as soon as possible. Summary of the Invention [Means for solving the problem]

[0005] In the embodiments of the present application, a model training and tone conversion method, apparatus, device and medium are provided to solve the problem that the conversion effect of the current tone conversion is poor and the reliability of the tone conversion is reduced.

[0006] In an embodiment of the present application, a method for training a tone transformation model is provided, the method including: Obtain a sample set, where the sample set includes sample audio data of different speakers, each of the sample audio data corresponds to a target audio data, and the target audio data and the sample audio data have the same semantic information; For any one of the sample audio data, a timbre extraction network in an original timbre conversion model obtains a first timbre feature of the sample audio data, and a timbre removal network in the original timbre conversion model obtains a first semantic feature based on the first timbre feature and a linear spectrum corresponding to the sample audio data, where the first semantic feature is a feature related to semantic information and is unrelated to the speaker's timbre in the sample audio data; and a vocoder in the original timbre conversion model obtains synthetic audio data based on the first semantic feature and a second timbre feature of target audio data corresponding to the sample audio data; training the original timbre transformation model based on target audio data and synthesized audio data corresponding to each of the sample audio data, respectively, to obtain a trained timbre transformation model; Includes.

[0007] In an embodiment of the present application, a tone conversion method is provided, the method comprising: Obtaining source audio data and timbre characteristics of a target speaker; Acquire timbre features of the source audio data by a timbre extraction network in a pre-trained timbre transformation model, acquire semantic features based on the timbre features and a linear spectrum corresponding to the source audio data by a timbre removal network in the timbre transformation model, where the semantic features are features related to semantic information in the source audio data that are unrelated to speaker timbre, and acquire synthetic audio data based on the semantic features and the timbre features of the target speaker by a vocoder in the timbre transformation model; Includes.

[0008] In an embodiment of the present application, a tone transformation model training device is provided, the device including: an acquisition unit, a processing unit, and a training unit; The acquiring unit is configured to acquire a sample set, where the sample set includes sample audio data of different speakers, each of the sample audio data corresponds to a target audio data, and the target audio data and the sample audio data have the same semantic information; The processing unit is configured to, for any one of the sample audio data, obtain a first timbre feature of the sample audio data through a timbre extraction network in an original timbre transformation model, obtain a first semantic feature according to the first timbre feature and a linear spectrum corresponding to the sample audio data through a timbre removal network in the original timbre transformation model, where the first semantic feature is a feature related to semantic information in the sample audio data that is unrelated to a speaker's timbre, and obtain a synthetic audio data according to the first semantic feature and a second timbre feature of a target audio data corresponding to the sample audio data through a vocoder in the original timbre transformation model; The training unit is configured to train the original timbre transformation model according to target audio data and synthesized audio data respectively corresponding to each of the sample audio data, thereby obtaining a trained timbre transformation model.

[0009] In an embodiment of the present application, a tone conversion device is provided, the device including: an acquisition module and a synthesis module; The acquisition module is configured to acquire source audio data and timbre features of a target speaker; The synthesis module is configured to obtain timbre features of the source audio data by a timbre extraction network in a pre-trained timbre transformation model, and obtain semantic features based on the timbre features and a linear spectrum corresponding to the source audio data by a timbre removal network in the timbre transformation model, where the semantic features are features in the source audio data that are unrelated to speaker timbre and relate to semantic information, and obtain synthetic audio data based on the semantic features and the timbre features of the target speaker by a vocoder in the timbre transformation model.

[0010] In an embodiment of the present application, an electronic device is provided, the electronic device including at least a processor and a memory, the processor being configured to realize the steps of the timbre transformation model training method as described above or the steps of the timbre transformation method as described above when executing a computer program stored in the memory.

[0011] In an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, realizes the steps of the timbre conversion model training method as described above or the steps of the timbre conversion method as described above.

[0012] An embodiment of the present application also provides a computer program product, which includes a computer program, and when the computer program is executed, the steps of the tone transformation model training method as described above or the steps of the tone transformation method as described above can be realized.

[0013] In the process of training the timbre conversion model, there is no need to pre-label the sample audio data in the sample set, which reduces the cost of labeling the sample audio data and facilitates subsequent training of the timbre conversion model based on the sample audio data in the sample set. The original timbre conversion model includes a timbre extraction network, a timbre removal network and a vocoder, so that the timbre extraction network can obtain a first timbre feature of the input sample audio data, thereby accurately obtaining timbre information of the input sample audio data, and subsequently obtaining synthetic audio data using the timbre feature, which is favorable for improving the timbre accuracy of the synthetic audio data. The timbre removal network can obtain a first semantic feature of the input sample audio data based on the first timbre feature, which realizes that the features related to the pronunciation content of the sample audio data are accurately obtained regardless of the speaker's timbre, and subsequently obtaining synthetic audio data using the first semantic feature, which is favorable for ensuring the accuracy of the pronunciation content of the synthetic audio data. The vocoder can obtain synthetic audio data based on the first semantic feature and the second timbre feature of the target audio data corresponding to the sample audio data. The original timbre conversion model can be trained based on the target audio data and the synthetic audio data corresponding to each sample audio data, respectively, to obtain a trained timbre conversion model, thereby realizing a timbre conversion model trained without a teacher, and greatly reducing the difficulty of obtaining a timbre conversion model. Then, timbre conversion can be performed based on the trained timbre conversion model, thereby improving the conversion effect and reliability of the timbre conversion.

[0014] In order to more clearly explain the technical solutions in the embodiments of the present application, the following provides a brief description of the drawings that need to be used in the description of the embodiments. The drawings described below are only some embodiments of the present application, and it is obvious to those skilled in the art that other drawings can also be obtained from these drawings without paying any creative labor. [Brief description of the drawings]

[0015] [Figure 1] FIG. 2 is a schematic diagram of the process of training a tone transformation model provided in an embodiment of the present application; [Diagram 2] FIG. 2 is a schematic diagram of the structure of a tone conversion model provided in an embodiment of the present application. [Diagram 3] FIG. 2 is a schematic diagram of the tone conversion process provided in an embodiment of the present application. [Figure 4] FIG. 2 is a schematic diagram of the structure of a tone conversion model provided in an embodiment of the present application. [Diagram 5] FIG. 2 is a schematic diagram of a tone transformation model training device provided in an embodiment of the present application; [Figure 6] FIG. 2 is a schematic diagram of the structure of a tone conversion device provided in an embodiment of the present application; [Figure 7] 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. [Figure 8] FIG. 2 is a schematic diagram of the structure of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be described in more detail below with reference to the accompanying drawings, and the described embodiments are only some of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments that a person skilled in the art can obtain without paying creative labor are all within the scope of protection of the present application.

[0017] Those skilled in the art will appreciate that the embodiments of the present application can be realized as a system, an apparatus, a device, a method, or a computer program product. Therefore, the present application can be specifically realized in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0018] It can be understood that in this application, any number of elements in the figures is used for illustration purposes, not limitation, and any names are used for distinction purposes only, and have no limiting meaning.

[0019] Currently, tone conversion can be achieved by adopting tone conversion technology using Test to Speech (TTS), tone conversion technology using Automatic Speech Recognition (ASR), tone conversion technology using Generative Adversarial Networks (GAN), and tone conversion technology using Variational Auto Encoder (VAE). When using these methods to convert tone, if you try to ensure the accuracy of the pronunciation content of the synthetic audio data, tone features that are unrelated to the semantic content such as tone in the synthetic audio data (e.g., sighing, screaming, etc.) cannot be ensured, and the naturalness of the synthetic audio data and the tone conversion effect are reduced.

[0020] Taking ASR timbre conversion technology as an example, the method generally employs a pre-trained ASR model to extract semantic information of audio, extracts timbre information in the audio data of the target speaker through a timbre extraction model, and generates synthetic audio data of the target speaker based on the semantic information and timbre information. Since the method relies heavily on the pre-trained ASR model, the accuracy of the ASR model directly affects the timbre conversion effect of the synthetic audio data. At the same time, since the ASR model is mainly used for extracting meanings, it may miss timbre information in the audio data that is unrelated to the semantic content, such as tone, so that the synthetic audio data loses information unrelated to the semantic content, such as tone.

[0021] Take the TTS timbre conversion technology as an example, in this method, a large amount of audio data of the target speaker needs to be collected in advance, and then the target speaker's vocal model and vocoder are trained based on each collected audio data and the semantic information corresponding to each audio data. Then, the target speaker's synthetic audio data is obtained based on the text features of the text information and the trained vocal model and vocoder of the target speaker. Generally, the target speaker's audio data needs to be collected in advance for more than 30,000 sentences or more than 30 hours, and in order to ensure the accuracy of the target speaker's vocal model and vocoder, the semantic of each audio data needs to be labeled, which increases the difficulty of obtaining the target speaker's vocal model and vocoder and the cost is also very high, and the vocal information such as tone in the obtained synthetic audio data is relatively fixed, which reduces the naturalness of the synthetic audio data and the vocal conversion effect.

[0022] In order to solve the above problems, the embodiments of the present application provide a model training and timbre conversion method, apparatus, device, and medium. In the process of training the timbre conversion model, there is no need to pre-label the sample audio data in the sample set, and the cost of labeling the sample audio data is reduced, so that the training of the timbre conversion model is facilitated based on the sample audio data in the subsequent sample set. In the process of training the timbre conversion model, there is no need to pre-label the sample audio data in the sample set, and the cost of labeling the sample audio data is reduced, so that the training of the timbre conversion model is facilitated based on the sample audio data in the subsequent sample set. The original timbre conversion model includes a timbre extraction network, a timbre removal network, and a vocoder, so that the timbre extraction network can obtain a first timbre feature of the input sample audio data, thereby accurately obtaining timbre information of the input sample audio data, and subsequently obtaining synthetic audio data using the timbre feature, which is advantageous for improving the accuracy of the timbre of the synthetic audio data; the timbre removal network can obtain a first semantic feature of the input sample audio data based on the first timbre feature, which realizes accurate acquisition of features related to the pronunciation content of the sample audio data regardless of the speaker's timbre, and subsequently obtaining synthetic audio data using the first semantic feature, which is advantageous for ensuring the accuracy of the pronunciation content of the synthetic audio data; and the vocoder can obtain synthetic audio data based on the first semantic feature and the second timbre feature of the target audio data corresponding to the sample audio data. By training the original tone conversion model based on the target audio data and the synthetic audio data corresponding to each sample audio data, a trained tone conversion model can be obtained, a tone conversion model trained without a teacher can be realized, and the difficulty of obtaining a tone conversion model can be significantly reduced.Subsequently, by performing timbre conversion based on the trained timbre conversion model, the conversion effect and reliability of the timbre conversion can be improved.

[0023] It should be noted that the application scenarios shown in the above embodiments are merely exemplary scenarios presented for ease of explanation, and are not intended to limit the application scenarios of the model training and tone conversion methods, apparatuses, devices and media provided in the embodiments of the present application. Those skilled in the art will appreciate that as new business scenarios emerge, the technical solutions provided in the embodiments of the present application can be similarly applied to similar technical problems.

[0024] Example 1: FIG. 1 is a schematic diagram of a tone conversion model training process provided in an embodiment of the present application, which includes: S101: obtaining a sample set, where the sample set includes sample audio data of different speakers, each of the sample audio data corresponds to a target audio data, and the target audio data and the sample audio data have the same semantic information.

[0025] The tone transformation model training method provided in the embodiments of the present application is applied to an electronic device, which may be, for example, a smart device such as a robot or a mobile terminal, or a server.

[0026] In general, audio data of different speakers can be collected and these audio data can be determined as sample audio data. The original timbre synthesis model is first trained using these sample audio data to obtain a trained timbre synthesis model, thereby improving the robustness of the timbre conversion model and increasing the variety of timbres that can be synthesized by the timbre conversion model.

[0027] In the embodiment of the present application, obtaining sample audio data in S101 includes at least one of the following manners:

[0028] Method 1: Recorded audio data of different speakers is determined as sample audio data.

[0029] In the process of collecting sample audio data, the speaker can record speech data in a professional recording environment and determine the recorded speech data as sample audio data; the speaker can record speech data through an intelligent terminal (e.g., a mobile phone, a tablet, etc.) and determine the sample audio data according to the recorded speech data.

[0030] For example, the speaker can input a trigger operation into the intelligent terminal. Here, there are many ways for the speaker to input the trigger operation into the intelligent terminal, the trigger operation may be that the speaker triggers a virtual button displayed on the display of the intelligent terminal, may be that the speaker inputs speech information into the intelligent terminal, or may be that the speaker draws a graphic command on the display of the intelligent terminal, and in the specific implementation process, it can be flexibly set according to the actual needs, and no specific limitations are given here. After receiving the trigger operation input by the speaker, the intelligent terminal can upload the pre-recorded speech data selected by the speaker to the electronic device, enter the speech recording function, start to record the speech data of the speaker in real time, and upload a plurality of speech data to be recorded to the electronic device, and can also determine the sample audio data according to the speech data received by the electronic device.

[0031] Since the speaker does not need to record speech data in a professional recording environment, but can record speech data by an intelligent terminal, the difficulty and cost of acquiring the speaker's speech data are reduced and the user experience is greatly improved.

[0032] When the speech data is recorded by an intelligent terminal, the speech data recorded by the intelligent terminal can be determined as the original speech data of the speaker. Since the original speech data may contain a large amount of noise in the working environment, the recorded original speech data is first subjected to audio processing, for example, noise reduction processing and / or reverberation removal processing, so as to obtain clean speech data. Then, the speech data after the audio processing is determined as the sample audio data.

[0033] In addition, the specific process of performing the noise reduction process and / or the dereverberation process on the original speech data is a prior art, so a detailed description thereof will be omitted here.

[0034] Method 2: Based on the above method 1, a stitching process can be performed on at least two speech data of the same speaker, and the stitched speech data acquired after the stitching process is determined as sample audio data. In this way, the acquired sample audio data can be expanded, which is advantageous in reducing the difficulty and cost of acquiring sample audio data, and using the large amount of acquired sample audio data to train a timbre conversion model, thereby improving the accuracy and robustness of the acquired timbre conversion model.

[0035] In one possible embodiment, based on the above method 1, performing a stitching process on at least two speech data of the same speaker, and determining the stitched speech data obtained after the stitching process as sample audio data can include several methods such as the following:

[0036] Method 1: The speech data recorded according to the above method 1 is determined as basic speech data. Here, the basic speech data may be the original speech data in the above embodiment, or may be speech data after audio processing in the above embodiment. For different speakers, stitching is performed on at least two different basic speech data of the speakers to determine stitched speech data (for ease of explanation, it is described as first stitched speech data). Each basic speech sample and each first stitched speech data are both determined as sample audio data.

[0037] Method 2: The speech data recorded according to Method 1 is determined as basic speech data. For different speakers, at least one basic speech data of the speaker is copied by a set multiple, and the at least one copied speech data and the corresponding basic speech data are stitched to determine stitched speech data (for ease of explanation, described as second stitched speech data). It can be understood that the second stitched speech data is stitched by at least two identical speech data. Each basic speech sample and each second stitched speech data are both determined as sample audio data.

[0038] Method 3: The speech data recorded according to the above method 1 is determined as basic speech data. For different speakers, at least one basic speech data of the speaker is copied by a set multiple, and at least two identical speech data (the copied speech data and the basic speech data) and at least one speech data of the speaker other than the speech data are stitched to determine stitched speech data (for ease of explanation, described as third stitched speech data). It can be understood that the third stitched speech data includes at least two identical speech data and at least two different speech data. Each basic speech sample and each third stitched speech data are determined as sample audio data.

[0039] In one possible embodiment, the acquired sample audio data can be enhanced by at least two of the above methods 1 to 3 at the same time.

[0040] Based on determining sample audio data based on the recorded speech data of the speaker, a stitching process is performed on at least two speech data of the speaker, and the stitched speech data can be determined as the sample audio data. This makes it possible to realize an extension to the speaker's sample audio data, and further reduces the difficulty and cost of obtaining the speaker's sample audio data. It is also advantageous in training a timbre conversion model using the large amount of sample audio data obtained, thereby improving the accuracy and robustness of the obtained timbre conversion model.

[0041] To obtain a relatively accurate timbre conversion model, it is necessary to supervise the timbre conversion effect of the timbre conversion model so that the synthetic audio data output by the timbre conversion model approaches the audio data of the same pronunciation content spoken by the target speaker (described as target audio data). Therefore, after obtaining each sample audio data in the sample set, the target audio data corresponding to the sample audio data can be determined, and then the timbre conversion effect of the original timbre conversion model can be determined by determining whether the synthetic audio data obtained based on the original timbre conversion model and the sample audio data is accurate or not according to the target audio data.

[0042] In one possible embodiment, for each sample audio data in the sample set, target audio data of a different speaker with the same semantic information as the sample audio data can be determined, where the target audio data includes at least one of sample audio data and non-sample audio data, i.e., the target audio data includes sample audio data of a different speaker with the same semantic information as the sample audio data in the sample set and / or non-sample audio data of a different speaker with the same semantic information as the sample audio data.

[0043] In another possible embodiment, in consideration of the fact that for each sample audio data in a sample set, the process of determining target audio data of different speakers that is the same as the semantic information of the sample audio data will incur a large amount of cost, so in the embodiment of the present application, for each sample audio data in a sample set, the sample audio data can be determined as the target audio data corresponding to the sample audio data, thereby realizing that it is not necessary to spend resources to determine target audio data that is different from the timbre of the sample audio data, and reducing the difficulty of training the timbre conversion model.

[0044] S102: For any one of the sample audio data, a first timbre feature of the sample audio data is obtained by a timbre extraction network in an original timbre conversion model, and a first semantic feature is obtained based on the timbre feature and a linear spectrum corresponding to the sample audio data by a timbre removal network in the original timbre conversion model, where the first semantic feature is a feature related to semantic information and is unrelated to a speaker's timbre in the sample audio data, and synthetic audio data is obtained based on the first semantic feature and a second timbre feature of target audio data corresponding to the sample audio data by a vocoder in the original timbre conversion model.

[0045] In the embodiment of the present application, the original timbre conversion model is pre-arranged, and the parameter values ​​of the parameters in the original timbre conversion model may be set randomly or may be pre-arranged artificially. After obtaining a sample set based on the above embodiment, the original timbre conversion model is trained based on each sample audio data included in the sample set, thereby obtaining a trained timbre conversion model.

[0046] In a specific implementation, one of the sample audio data in the sample set is obtained, and the sample audio data is input into an original timbre conversion model, which obtains synthetic audio data corresponding to the sample audio data according to the timbre features (referred to as second timbre features) of the sample audio data and the target audio data corresponding to the sample audio data.

[0047] In one example, the original timbre conversion model at least includes a timbre extraction network, a timbre removal network and a vocoder, and the timbre extraction network, the timbre removal network and the vocoder included in the original timbre conversion model perform accurate processing on the input sample audio data, where the timbre extraction network is respectively connected to the timbre removal network and the vocoder, and the timbre removal network is connected to the vocoder, the timbre extraction network can obtain a first timbre feature in the input sample audio data, so as to accurately obtain timbre information unrelated to semantic content in the input sample audio data, and then use the first timbre feature to obtain synthetic audio data, which is favorable for improving the accuracy of the timbre of the synthetic audio data; the timbre removal network can obtain a first semantic feature in the input sample audio data, which is unrelated to the timbre of the speaker of the sample audio data and only relates to the semantic information of the sample audio data, so as to then use the first semantic feature to obtain synthetic audio data, which is favorable for ensuring the accuracy of the pronunciation content of the synthetic audio data. The vocoder can obtain synthetic audio data based on the first semantic features and second tonal features of target audio data corresponding to the sample audio data.

[0048] In a specific implementation process, one of the sample audio data in the sample set is obtained, and the sample audio data is input into an original timbre conversion model. A timbre extraction network in the original timbre conversion model performs corresponding processing on the sample audio data to obtain a first timbre feature corresponding to the sample audio data. A timbre removal network in the original timbre conversion model obtains a semantic feature (referred to as a first semantic feature) based on the first timbre feature and a linear spectrum corresponding to the sample audio data. A vocoder in the original timbre conversion model obtains synthetic audio data corresponding to the sample audio data based on the obtained first semantic feature and a second timbre feature of target audio data corresponding to the sample audio data.

[0049] Here, the first semantic feature is a feature related to the pronunciation content and is unrelated to the speaker's timbre in the sample audio data, and therefore the influence of timbre on the semantic information is avoided, and the accuracy of the acquired semantic information is improved.

[0050] The linear spectrum corresponding to the sample audio data may be obtained by a pre-installed linear spectrum algorithm (such as a fast Fourier transform algorithm).

[0051] In one possible embodiment, before any one of the sample audio data in the sample set is input into the original timbre transformation model, the vocal characteristics of the sample audio data can be obtained, the vocal characteristics of the sample audio data can be input into the original timbre transformation model, and the vocal characteristics can be processed by the original timbre transformation model to obtain synthesized audio data.

[0052] Here, the vocal features may be any one of the types of vocal features such as Mel Spectrogram Mel Frequency Cepstrum Coefficients (MFCC), Bark Frequency Cepstrum Coefficients (BFCC), Inverse Mel Frequency Cepstrum Coefficients (IMFCC), Gammatone Frequency Cepstrum Coefficients (GFCC), Linear Prediction Cepstrum Coefficients (LPCCs), etc.

[0053] The vocal tract features may be obtained by a vocal tract feature extraction algorithm or a vocal tract feature extraction model.

[0054] For example, the vocal characteristic is a Mels spectrum, and the Mels spectrum of one of the sample audio data in the sample set is obtained, and the Mels spectrum of the sample audio data is input to an original timbre conversion model, and the timbre extraction network in the original timbre conversion model performs processing according to the input Mels spectrum to obtain a first timbre characteristic of the sample audio data, for example, a 256-dimensional timbre characteristic (tone_vector).

[0055] In addition, the timbre extraction network can be determined by a network layer included in a voiceprint model, such as an end-to-end voiceprint network (Deep Speaker RawNet, GE2E).

[0056] The elimination timbre network in the original timbre transformation model includes at least a posterior encoder, which accurately obtains features related to semantic information in the sample audio data regardless of speaker timbre, improves the accuracy of the obtained first semantic feature, and avoids the influence of the timbre features of the sample audio data on the semantic information of the sample audio data, where the posterior encoder is connected to the enhancement sub-network, and is configured to obtain hidden vectors related to pronunciation content from the sample audio data, and determine the first semantic feature of the sample audio data according to the hidden vector.

[0057] Specifically, after obtaining the first timbre feature output by the timbre extraction network according to the above embodiment, the post-encoder included in the timbre removal network in the original timbre transformation model can obtain a hidden vector of semantic information in the sample audio data according to the first timbre feature and a linear spectrum corresponding to the sample audio data, and then determine the first semantic feature according to the hidden vector.

[0058] Here, the hidden vector can be directly determined as the first semantic feature, or the hidden vector can be processed by a pre-set mathematical function (e.g., a logarithmic function) and the processed hidden vector can be determined as the first semantic feature.

[0059] In one possible embodiment, the removal timbre network in the original timbre transformation model includes an enhancement sub-network (e.g., a flow network) to enhance features related to speaker timbre and semantic information in the sample audio data, improve the distribution representation of semantic information, and further improve the accuracy of the obtained first semantic feature, and avoid the influence of the timbre features of the sample audio data on the semantic information of the sample audio data. The enhancement sub-network can be connected to a post-encoder in the removal timbre network, and the enhancement sub-network is arranged to perform enhancement on the hidden vector obtained by the post-encoder. It can be understood that the enhancement sub-network is arranged to extract higher-dimensional, more abstract features related to speaker timbre and semantic information from the hidden vector obtained by the post-encoder.

[0060] In a specific implementation process, after obtaining the first timbre feature output by the timbre extraction network according to the above embodiment, the post-encoder included in the timbre removal network in the original timbre transformation model can obtain a hidden vector of semantic information in the sample audio data according to the first timbre feature and the linear spectrum corresponding to the sample audio data. Then, the enhancement sub-network included in the timbre removal network in the original timbre transformation model obtains an enhanced hidden vector according to the hidden vector, that is, determines the first semantic feature.

[0061] After obtaining the first semantic feature of the input sample audio data based on the above embodiment, the vocoder in the timbre conversion model can obtain synthetic audio data whose pronunciation content satisfies the first semantic feature and whose timbre satisfies the second timbre feature based on the first semantic feature and the second timbre feature of the target audio data corresponding to the sample audio data.

[0062] Here, the vocoder may be a vocoder, for example, a high-efficiency (HiFiGAN) vocoder, a Linear Predictive Coding (LPC) vocoder, a World vocoder, etc., and can be flexibly set according to actual needs in the specific implementation process, and no specific limitations are provided here.

[0063] In one possible embodiment, a second timbre feature of the target audio data corresponding to the sample audio data is obtained by a timbre extraction network in the original timbre transformation network.

[0064] For example, when the target audio data is sample audio data, the first timbre feature of the sample audio data determined by the timbre extraction network in the original timbre transformation network is directly determined as the second timbre feature.

[0065] In addition, for example, when the target audio data is not sample audio data, that is, when the target audio data is sample audio data in a sample set that is different from the speaker of the sample audio data, or non-sample audio data that is different from the speaker of the sample audio data, when the sample audio data is input to the original timbre conversion model, the target audio data corresponding to the sample audio data is also input to the original timbre conversion model, thereby realizing the acquisition of the second timbre feature of the target audio data by the timbre extraction network in the original timbre conversion network.

[0066] S103: Training the original timbre transformation model based on the target audio data and the synthesized audio data respectively corresponding to each of the sample audio data, thereby obtaining a trained timbre transformation model.

[0067] The synthesized audio data corresponding to the sample audio data determined by the more accurate timbre conversion model is more similar to the target audio data corresponding to the sample audio data. Therefore, after obtaining the synthesized audio data corresponding to each sample audio data according to the above embodiment, the parameter values ​​of the parameters in the original timbre conversion model are adjusted according to the target audio data corresponding to each sample audio data and the synthesized audio data corresponding to each sample audio data, thereby obtaining a trained timbre conversion model. Exemplarily, a reconstruction loss value is determined according to the synthesized audio data corresponding to each target audio data corresponding to each sample audio data, and the parameter values ​​of the parameters in the original timbre conversion model are adjusted according to the reconstruction loss value, thereby obtaining a trained timbre conversion model.

[0068] In one possible embodiment, for each sample audio data, a sub-reconstruction loss value can be determined according to the target audio data corresponding to the sample audio data and the synthetic audio data corresponding to the sample audio data. A reconstruction loss value is determined according to the sum of all the sub-reconstruction loss values ​​obtained by the current iteration, and the parameter values ​​of the parameters in the original timbre transformation model are adjusted according to the reconstruction loss value, thereby obtaining a trained timbre transformation model.

[0069] Exemplarily, a sub-reconstruction loss value may be determined according to the target audio data corresponding to the sample audio data and the synthesized audio data corresponding to the sample audio data, according to the following formula:

number

[0070] Where:

number

number

number

number

[0071] Considering that the accuracy of semantic information in the audio data obtained by the timbre conversion model also affects the effect of timbre conversion, in the embodiment of the present application, during the training process of the timbre conversion model, the accuracy of the semantic features extracted by the timbre conversion model is supervised.

[0072] In one possible embodiment, the semantic information of each sample audio data in the sample set can be pre-labeled, and the accuracy of the semantic features extracted by the timbre conversion model can be supervised based on the difference between the semantic labeling corresponding to each sample audio data and the first semantic features corresponding to each sample audio data.

[0073] In another possible embodiment, when trying to obtain a relatively accurate timbre conversion model, a large amount of sample audio data is required, and labeling each sample audio data requires a large amount of work and a large cost, which increases the difficulty of obtaining a timbre conversion model. Therefore, in the embodiment of the present application, a semantic extraction network is added to the timbre conversion model, and the semantic information of the audio data can be adversarially learned by the semantic extraction network and the timbre removal network in the timbre conversion model. In the process of training the original timbre conversion model, for any one sample audio data in the sample set, the semantic feature (described as the second semantic feature) of the sample audio data is obtained based on the input sample audio data by the semantic extraction network in the original timbre conversion model. Here, the second semantic feature is also a feature of the sample audio data that is not related to the speaker's timbre and is related to the semantic information.

[0074] In one example, the semantic extraction network can include a first content sub-network, a second content sub-network and a third content sub-network, thereby realizing accurate acquisition of semantic information for removing timbre information in sample audio data. Here, the first content sub-network is connected to the second content sub-network, and the first content sub-network is configured to perform processing on the sample audio data to obtain relatively dense content features in the sample audio data, and the first content sub-network can be an h-net network or the like. The second content sub-network is connected to a third content sub-network, and the second content sub-network is configured to perform feature processing on the content output by the first content sub-network, obtain discretized content features, remove unnecessary details in the content features, and associate the discretized content features with the semantic information of the sample audio data, and the second content sub-network can be a vectorization layer (Vector Quantization, VQ), for example. The third content sub-network is configured to perform feature processing on the discretized content output by the second content sub-network, and obtain a second semantic feature of the sample audio data by exciting local features related to semantic information in the feature learning sample audio data of the discretized content. For example, the third content sub-network may be a g-net network, a Contrastive Predictive Coding (CPC) network using a Recurrent Neural Network (RNN), etc.

[0075] In a specific implementation process, after inputting any one of the sample audio data into an original timbre conversion model based on the above embodiments, a first content sub-network included in the network is extracted according to the meaning in the original timbre conversion model, and a content feature is obtained based on the sample audio data, a discretized content feature is obtained based on the content feature by a second content sub-network in the semantic extraction network, and a second semantic feature of the sample audio data is obtained based on the discretized content feature by a third content sub-network in the semantic extraction network.

[0076] 2 is a schematic diagram of the structure of a tone conversion model provided in an embodiment of the present application. Hereinafter, the tone conversion model training method provided in the embodiment of the present application will be described in conjunction with FIG.

[0077] The Mels spectrum of any one of the sample audio data in the sample set is obtained, and the Mels spectrum of the sample audio data is input to an original timbre conversion model. A timbre extraction network (speaker encoder) in the original timbre conversion model processes the Mels spectrum of the sample audio data accordingly to obtain a first timbre feature (tone_vector) corresponding to the sample audio data. A posterior encoder included in a timbre removal network in the original timbre conversion model calculates a hidden vector (z sq) can be obtained. The enhancement sub-network (flow) included in the timbre removal network in the original timbre conversion model obtains an enhanced hidden vector based on the hidden vector, that is, determines a first semantic feature. At the same time, the semantic extraction network (VQCPC net) in the original timbre conversion model obtains a second semantic feature in the sample audio data based on the input sample audio data. Then, the vocoder (decoder) shown in FIG. 2 obtains synthetic audio data (raw waveform) based on the first semantic feature and the second timbre feature (speaker inner embedding) of the target audio data corresponding to the sample audio data.

[0078] Specifically, the process of acquiring the second semantic feature of the sample audio data based on the input sample audio data by the semantic extraction network in the original timbre conversion model includes acquiring a content feature Z based on the Mels spectrum of the sample audio data by the first content sub-network (h-net) included in the semantic extraction network in the original timbre conversion model. The content feature Z of the k-th sample audio data as shown in FIG. 2 includes content data of each audio frame included in the k-th sample audio data. For example, the content data of the n-th audio frame included in the k-th sample audio data is z k,n A second content sub-network (Vector Quantization (VQ)) in the semantic extraction network extracts discretized content features Z based on the content features Z.

number

number

number

number

[0079] In addition, the h-net network can include a convolution layer, a normalization layer, a linear transformation layer, and a logic function layer. One convolution layer included in the h-net network shown in Figure 2 is connected in sequence to four sub-networks of the same connection structure, and for any one sub-network, the sub-network includes one normalization layer, two linear transformation layers, and one ReLU layer.

[0080] After obtaining the second semantic features of each sample audio data according to the above embodiment, the original timbre conversion model can be trained based on the target audio data corresponding to each sample audio data, the synthetic audio data corresponding to each sample audio data, the first semantic features corresponding to each sample audio data, and the second semantic features corresponding to each sample audio data, to obtain a trained timbre conversion model. That is, based on the difference between the target audio data corresponding to each sample audio data and the corresponding synthetic audio data, and the difference between the first semantic features corresponding to each sample audio data and the corresponding second semantic features, the parameter values ​​of the parameters in the original timbre conversion model are adjusted to obtain a trained timbre conversion model, thereby realizing training the ability of the timbre conversion model to extract semantic information without supervision.

[0081] In a specific implementation, a reconstruction loss value is determined according to the target audio data corresponding to each sample audio data and the synthesized audio data corresponding to each sample audio data, and a semantic loss value is determined according to the first semantic feature corresponding to each sample audio data and the second semantic feature corresponding to each sample audio data, and a total loss value is determined according to the reconstruction loss value and the semantic loss value, and a parameter value of the parameter in the original timbre conversion model is adjusted according to the total loss value to obtain a trained timbre conversion model.

[0082] For example, for each sample audio data in the sample set, a sub-reconstruction loss value is determined according to the difference between the target audio data corresponding to the sample audio data and the corresponding synthesized audio data, and a sub-semantic loss value is determined according to the difference between the first semantic feature and the second semantic feature of the sample audio data. A reconstruction loss value is determined according to the sum of all reconstruction loss values ​​determined by the current iteration, and a semantic loss value is determined according to the sum of all sub-semantic loss values ​​determined by the current iteration. A total loss value is determined according to the reconstruction loss value and the semantic loss value. A trained timbre conversion model is obtained by adjusting the parameter values ​​of the parameters in the original timbre conversion model according to the total loss value.

[0083] In one example, when the total loss value is determined by the reconstruction loss value and the semantic loss value, the total loss value can be determined by the reconstruction loss value and the corresponding first weight value, and the semantic loss value and the corresponding second weight value. For example, the product of the reconstruction loss value and the corresponding first weight value (described as the first product) and the product of the semantic loss value and the corresponding second weight value (described as the second product) are obtained. The total loss value is determined by the sum of the first product and the second product.

[0084] In one possible embodiment, the post-encoder included in the removal timbre network in the original timbre transformation model can obtain a hidden vector based on the linear spectrum corresponding to the sample audio data and the first timbre feature determined by the timbre extraction network in the original timbre transformation model, and can also obtain the mean vector and variance vector of the hidden vector. That is, the post-encoder included in the removal timbre network in the original timbre transformation model can obtain the mean vector and variance vector of the hidden vector based on the linear spectrum corresponding to the sample audio data and the first timbre feature determined by the timbre extraction network in the original timbre transformation model. Subsequently, when determining a semantic loss value based on the first semantic feature corresponding to each sample audio data and the second semantic feature corresponding to each sample audio data, the semantic loss value can be determined based on the first semantic feature, the second semantic feature, the mean vector and the variance vector corresponding to each sample audio data. That is, for each sample audio data, a sub-semantic loss value is determined by the first semantic feature, the second semantic feature, the mean vector and the variance vector of the sample audio data. The semantic loss value is determined by the sum of all the sub-semantic loss values ​​determined by the current iteration.

[0085] Exemplarily, a sub-semantic loss value can be determined based on the first semantic feature, the second semantic feature, the mean vector, and the variance vector of the sample audio data by the following formula:

number

[0086] where KL_loss denotes the sub-semantic loss value corresponding to the kth sample audio data,

number

number

number

number

[0087] In one possible embodiment, when the semantic extraction network in the original timbre conversion model includes a first content sub-network, a second content sub-network and a third content sub-network, it is also necessary to consider the existence of a loss value in the semantic extraction network in the process of training the original timbre conversion model. Therefore, in the embodiment of the present application, a quantification loss value can be determined according to the content features corresponding to each sample audio data and the discretized content features corresponding to each sample audio data, and a comparative learning loss value can be determined according to the discretized content features corresponding to each sample audio data and the second semantic features corresponding to each sample audio data, so as to subsequently adjust the parameter values ​​of the parameters in the original timbre conversion model according to the target audio data corresponding to each sample audio data and the synthetic audio data corresponding to each sample audio data, the first semantic features corresponding to each sample audio data and the second semantic features corresponding to each sample audio data, the quantification loss value and the comparative learning loss value, thereby facilitating obtaining a trained timbre conversion model.

[0088] In one example, when determining the quantification loss value, for each sample audio data, a sub-quantification loss value corresponding to the sample audio data can be determined according to the content features corresponding to each audio frame included in the sample audio data and the content features of the discretization, and the quantification loss value is determined by the sum of all the sub-quantification loss values ​​determined by the current iteration.

[0089] For example, a quantification loss value can be determined based on the content features respectively corresponding to each sample audio data and the content features of the discretization respectively corresponding to each sample audio data by the following formula:

number

[0090] Here, VQ_loss denotes the quantification loss value, K denotes the total number of sample audio data included in the sample set, N denotes the total number of audio frames included in each sample audio data, n denotes the nth audio frame included in the current kth sample audio data, and z k,n denotes a content feature corresponding to the n-th audio frame included in the k-th sample audio data,

number

number

[0091] In one example, when determining the adversarial learning loss value, for each sample audio data, the sub-comparison learning loss value corresponding to the sample audio data can be determined according to the content features and second semantic features of the discretization corresponding to each audio frame included in the sample audio data. The comparison learning loss value is determined by the sum of all the sub-comparison learning loss values ​​determined by the current iteration.

[0092] For example, a comparative training loss value is determined based on the discretized content features corresponding to each sample audio data and the second semantic features corresponding to each sample audio data, by the following formula:

number

[0093] where CPC_loss denotes the comparative training loss value, K denotes the total number of sample audio data included in the sample set,

number

number

number

[0094] In a specific implementation process, when training the original timbre conversion model, for each sample audio data, a sub-reconstruction loss value is determined according to the difference between the target audio data corresponding to the sample audio data and the corresponding synthesized audio data, a sub-semantic loss value is determined according to the difference between the first semantic feature and the second semantic feature of the sample audio data, a sub-quantification loss value corresponding to the sample audio data is determined according to the content feature and the discretization content feature corresponding to each audio frame included in the sample audio data, and a sub-comparison learning loss value corresponding to the sample audio data is determined according to the discretization content feature and the second semantic feature corresponding to each audio frame included in the sample audio data. A reconstruction loss value is determined according to the sum of all the sub-reconstruction loss values ​​determined by the current iteration, a semantic loss value is determined according to the sum of all the sub-comparison learning loss values ​​determined by the current iteration, a comparison learning loss value is determined according to the sum of all the sub-quantification loss values ​​determined by the current iteration, and a quantification loss value is determined according to the sum of all the sub-quantification loss values ​​determined by the current iteration. The determined reconstruction loss value, semantic loss value, comparative training loss value, and quantification loss value are used to determine an overall loss value, and the parameter values ​​of the parameters in the original timbre conversion model are adjusted according to the overall loss value to obtain a trained timbre conversion model.

[0095] For example, the total loss value can be determined by the determined reconstruction loss value and its corresponding first weight value, the semantic loss value and its corresponding second weight value, the comparative learning loss value and its corresponding third weight value, and the quantification loss value and its corresponding fourth weight value.

[0096] Considering that the timbre conversion effect of the synthetic audio data is also affected by the accuracy of the vocoder, in the embodiment of the present application, in the process of training the original timbre conversion model, considering that the vocoder in the original timbre conversion model may have a loss value, the timbre conversion model is trained according to the loss value of the vocoder. That is, in the process of training the original timbre conversion model, the original timbre conversion model is trained according to the target audio data and synthetic audio data corresponding to each sample audio data, and the loss value of the vocoder in the original timbre conversion model, to obtain a trained timbre conversion model.

[0097] For example, the vocoder is a HiFiGAN vocoder. The HiFi-GAN vocoder is a deep neural network model that adopts an end-to-end forward network structure to train a multi-scale classifier and realize efficient and high-quality speech synthesis. The HiFiGAN vocoder includes a generator and a classifier, and the classifier includes two types, a multi-scale classifier and a multi-cycle classifier, which identify the audio data generated by the generator in the HiFiGAN vocoder from two different angles. The HiFiGAN vocoder uses the feature matching loss as the non-fixed loss of the training generator to extract each intermediate feature of the classifier, thereby calculating the distance L1 between the target audio data and the synthesized audio data in each feature space, and stabilizing the GAN. Therefore, the loss value of the HiFiGAN vocoder includes a feature matching loss value, and the original timbre conversion model is trained according to the feature matching loss value.

[0098] Exemplarily, the feature matching loss value may be determined by the following formula:

number

[0099] Here, J is the number of layers that extract features in the classifier included in the vocoder, and D j () denotes the feature extracted by the j-th feature layer in the classifier, and Q j denotes the number of features extracted by the j-th extracted feature layer in the discriminator, x is the target audio data, and s is the Mels spectrum of the synthetic audio data generated by the generator.

[0100] Since the essence of the HiFiGAN vocoder is still a generative adversarial network, the synthetic audio data calculated by the discriminator in the HiFiGAN vocoder is the probability of the target audio data, and the generator in the HiFiGAN vocoder is arranged to synthesize audio data. In the process of training the HiFiGAN vocoder, it is desirable that the generator in the HiFiGAN vocoder can synthesize synthetic audio data close to the target audio data so that the discriminator in the HiFiGAN vocoder cannot distinguish whether it is the target audio data or the synthetic audio data. Based on this, the loss value of the HiFiGAN vocoder further includes a generative adversarial loss value. For example, the generative adversarial loss value can be determined according to the Mels spectrum of the target audio data and the synthetic audio data generated by the generator in the HiFiGAN vocoder.

[0101] For example, the adversarial loss value can be generated by the following formula:

number

[0102] Here, L adv (D;G) show the generative adversarial loss value of the classifier in the HiFiGAN vocoder, and L adv (G;D) denotes the generative adversarial loss value of the generator in the HiFiGAN vocoder, x is the target audio data, and s is the Mels spectrum of the synthetic audio data generated by the generator.

[0103] Based on the above embodiment, after obtaining the vocoder loss value, the parameter values ​​of each parameter in the original timbre conversion model are adjusted according to the vocoder loss value and the target audio data and the synthetic audio data corresponding to each sample audio data, respectively, to obtain a trained timbre conversion model.

[0104] For example, a reconstruction loss value is determined based on the target audio data and the synthesized audio data corresponding to each sample audio data, and a vocoder loss value in the original timbre conversion model is determined, and a total loss value is determined according to the reconstruction loss value and the vocoder loss value, and the parameter values ​​of each parameter in the original timbre conversion model are adjusted according to the total loss value.

[0105] Here, when the total loss value is determined based on the reconstruction loss value and the vocoder loss value, the total loss value can be determined based on the reconstruction loss value and its corresponding first weighting value, and the vocoder loss value and its corresponding fifth loss value.

[0106] Also, for example, a reconstruction loss value is determined based on target audio data and synthetic audio data corresponding to each sample audio data, a semantic loss value is determined based on a first semantic feature corresponding to each sample audio data and a second semantic feature corresponding to each sample audio data, a quantification loss value is determined based on a content feature corresponding to each sample audio data and a discretized content feature corresponding to each sample audio data, a comparative learning loss value is determined based on a discretized content feature corresponding to each sample audio data and a second semantic feature corresponding to each sample audio data, and a vocoder loss value in the original timbre conversion model is determined. An overall loss value is determined based on the reconstruction loss value, the semantic loss value, the quantification loss value, the comparative learning loss value, and the vocoder loss value. The parameter values ​​of each parameter in the original timbre conversion model are adjusted based on the overall loss value.

[0107] For example, the overall loss value can be determined by the reconstruction loss value, the semantic loss value, the quantification loss value, the comparative training loss value, and the vocoder loss value, as shown in the following formula:

number

[0108] Here, total_loss denotes the total loss value, recon_loss denotes the reconstruction loss value, KL_loss denotes the semantic loss value, vq_loss denotes the quantification loss value, cpc_loss denotes the comparative training loss value, and decoder_loss denotes the vocoder loss value.

[0109] Note that when the vocoder is a HiFiGAN vocoder, the decoder_loss may include a feature matching loss value (fm_loss) and a generative adversarial loss value (adv_loss).

[0110] Since several sample audio data are included in training the original timbre transformation model, the above steps are carried out for each sample audio data until a preset convergence condition is met.

[0111] Here, the preset convergence condition is satisfied may be that the total loss value determined by the current iteration is less than the preset loss threshold, or the number of iterations for training the original tone conversion model reaches the preset maximum number of iterations, etc. In specific implementation, it can be flexibly set, and no specific limitation is provided here.

[0112] In one possible embodiment, when training the original timbre transformation model, the sample audio data is divided into training samples and test samples, and the original timbre transformation model is first trained based on the training samples, and the reliability of the already trained timbre transformation model is verified based on the test samples.

[0113] After obtaining the trained timbre conversion model, the timbre features corresponding to each speaker in the sample set can be determined based on the timbre conversion model and each sample audio data in the sample set. Then, the timbre features corresponding to each speaker and the target indicator corresponding to each speaker are correspondingly stored. In this way, when the trained timbre conversion model is used to subsequently synthesize synthetic audio data of any one speaker in the sample set, the timbre features of the target indicator of the speaker can be directly determined according to the correspondence between the target indicator and the timbre features, and the timbre features and timbre conversion model of the speaker can be used to obtain synthetic audio data of the speaker, which is advantageous to improving the efficiency of timbre conversion. It can be understood that the timbre of any one speaker of sample audio data in the sample set is the timbre supported by the trained timbre conversion model.

[0114] Example 2: In the embodiment of the present application, a tone conversion method is also provided. FIG. 3 is a schematic diagram of a tone conversion process provided in the embodiment of the present application, the process includes: S301: The method includes obtaining source audio data and timbre characteristics of a target speaker.

[0115] The tone conversion method provided in the embodiment of the present application is applied to an electronic device, which may be, for example, a smart device such as a robot, or a server, where the electronic device performing tone conversion in the embodiment of the present application may be the same as or different from the electronic device performing the tone conversion model training.

[0116] In one possible embodiment, in the process of training the tone conversion model, an offline manner is generally adopted to train the tone conversion model, so that after a trained tone conversion model is obtained, the tone conversion model can be deployed in an electronic device that performs tone conversion, thereby facilitating the electronic device that performs tone conversion to perform tone conversion using the tone conversion model.

[0117] The specific process of training the tone conversion model has already been described in the above embodiment, and the overlapping parts will not be mentioned here.

[0118] In one possible embodiment, when the trained timbre conversion model includes a semantic extraction network, the semantic extraction network is mainly used to supervise the semantic features extracted by the timbre removal network, and therefore, when the trained timbre conversion model is deployed in an electronic device performing timbre conversion, other networks than the semantic extraction network in the timbre conversion model can be deployed in the electronic device performing timbre conversion, thereby reducing the amount of parameters included in the timbre conversion model and reducing the cost of data transfer and the pressure on the internal storage space of the electronic device performing timbre conversion.

[0119] When tone conversion is required, the user can input a synthesis request on the smart device, and the smart device can be controlled according to the synthesis request to synthesize audio data for a target speaker to provide a certain pronunciation content. Here, there are many specific ways to input the synthesis request, for example, the input synthesis request can be input by inputting speech information, or input by operating a virtual button displayed on the display of the smart device, etc., and can be flexibly set according to needs in the specific implementation process, and no specific limitations are provided here. After the smart device receives the synthesis request, it can send the synthesis request, source audio data, and information on the target speaker to an electronic device that performs tone conversion.

[0120] Here, the target speaker refers to the speaker to which the timbre of the synthesized audio data obtained by the timbre conversion technology belongs, and the information of the target speaker includes the target speaker's object identifier or the target speaker's audio data, and the source audio data is arranged to provide semantic information but not audio data of timbre information.

[0121] The source audio data and the target speaker's audio data may be recorded by the user on a smart device, or may be stored in advance in the audio data on the smart device.

[0122] In a possible application scenario, when a user determines a speaker of any one of the sample audio data in the sample set as a target speaker, the information of the target speaker includes a target indicator of the target speaker. For example, a smart device outputs target indicators of multiple speakers, and a user can select multiple target indicators output by the smart device. When the smart device detects the target indicator selected by the user, the smart device determines the selected target indicator as the information of the target speaker, and then the smart device can send a synthesis request, the target indicator of the target speaker, and the source audio data to an electronic device that performs timbre conversion.

[0123] Here, the target marker can be represented in a format such as a number, a string of characters, or in other formats, and any format that uniquely identifies the speaker can be arranged as an embodiment of the present application.

[0124] The sample audio data in the sample set are arranged to train a tone conversion model, and it can be understood that the tone of any one of the speakers in the sample audio data in the sample set is the tone supported by the trained tone conversion model.

[0125] In another possible application scenario, when a user determines a speaker other than the sample set as a target speaker, that is, does not determine a speaker of any one of the sample audio data in the sample set as a target speaker, the information of the target speaker includes the audio data of the target speaker. For example, when a smart device outputs target indicators of multiple speakers, and the smart device does not detect a target indicator selected by the user or detects a trigger operation of adding a target speaker input by the user, the smart device can prompt the user to input audio data of the target speaker, and determine the audio data input by the user as the information of the target speaker. Then, the smart device can send a synthesis request, the audio data of the target speaker, and the source audio data to an electronic device that performs tone conversion.

[0126] After receiving the synthesis request, the information of the target speaker and the source audio data, the electronic device performing timbre conversion can determine the timbre features of the target speaker according to the information of the target speaker, and then obtain audio data in which the target speaker provides pronunciation content of the source audio data according to the timbre features of the target speaker and the source audio data, i.e. obtain the synthesized audio data of the target speaker.

[0127] In one possible embodiment, obtaining the timbre characteristics of the target speaker comprises: acquiring information of the target speaker; When determining that the information of the target speaker is a target marker, determining a timbre feature corresponding to the target marker of the target speaker according to a correspondence relationship between the target marker and the timbre feature to be stored; When determining that the information of the target speaker is audio data, obtaining timbre features of the audio data through a timbre extraction network in the timbre conversion model; Includes.

[0128] In an embodiment of the present application, determining the timbre characteristics of the target speaker based on the information of the target speaker includes the following two situations:

[0129] Situation 1: when the information of the target speaker is a target marker of the target speaker, the target speaker indicates that the target speaker is the speaker of any one of the sample audio data in the sample set, so that the tonal features corresponding to the target marker of the target speaker are determined according to the correspondence between the stored target markers and tonal features, and the determined tonal features are determined as the tonal features of the target speaker.

[0130] Here, the timbre transformation model is trained based on sample audio data in the sample set.

[0131] Situation 2: when the information of the target speaker is audio data, it is impossible to determine whether the target speaker is the speaker of any of the sample audio data in the sample set, so the timbre extraction network in the timbre conversion model obtains the timbre features of the audio data, and determines the obtained timbre features as the timbre features of the target speaker.

[0132] S302: A timbre extraction network in a pre-trained timbre conversion model acquires timbre features of the source audio data, and a timbre removal network in the timbre conversion model acquires semantic features based on the timbre features and a linear spectrum corresponding to the source audio data, where the semantic features are features related to semantic information in the source audio data and are unrelated to the speaker's timbre; and a vocoder in the timbre conversion model acquires synthetic audio data based on the semantic features and the timbre features of the target speaker.

[0133] Based on the above embodiment, after obtaining the source audio data and the tonal features of the target speaker, the source audio data and the tonal features of the target speaker can be input into a pre-trained tonal transformation model, so that the tonal transformation model determines synthesized audio data according to the source audio data and the tonal features of the target speaker.

[0134] In a specific implementation, a timbre extraction network in a pre-trained timbre transformation model is used to obtain timbre features of source audio data; a timbre removal network in the timbre transformation model is used to obtain semantic features according to the timbre features and the linear spectrum corresponding to the source audio data, where the semantic features are unrelated to the speaker timbre in the source audio data and are related to semantic information; and a vocoder in the timbre transformation model is used to obtain synthetic audio data according to the semantic features and the timbre features of the target speaker.

[0135] Here, the process of obtaining a linear spectrum corresponding to the source audio data has been described in the above embodiment, and no specific limitations are made here.

[0136] In one possible embodiment, before inputting source audio data into a pre-trained timbre transformation model, the mel-cepstrum of the source audio data is obtained and input into the timbre transformation model instead of the source audio data, thereby reducing the amount of computation required by the timbre transformation model, improving the efficiency of the timbre transformation, and facilitating the timbre transformation performed by the timbre transformation model.

[0137] 4 is a schematic diagram of a tone conversion model structure provided in the embodiment of the present application. Hereinafter, the tone conversion method provided in the embodiment of the present application will be described in conjunction with FIG.

[0138] First, the source audio data and the target speaker information are obtained.

[0139] When the information of the target speaker is the target marker of the target speaker, it indicates that the target speaker is a speaker of any one of the sample audio data in the sample set, and when the target speaker is currently trying to obtain audio data in which the target speaker provides the pronunciation content in any one of the audio data, the timbre feature corresponding to the target marker of the target speaker is determined according to the correspondence between the target marker and the timbre feature stored, and the determined timbre feature is determined as the timbre feature of the target speaker. According to the information of the target speaker, an any-to-many timbre conversion manner can be subsequently realized, that is, the timbre feature of the speaker of any one of the sample audio data in the sample set can be selected, and the audio data in which the speaker provides the pronunciation content can be obtained according to the timbre feature and the source audio data of any pronunciation content through the pre-trained timbre conversion model.

[0140] When the target speaker information is audio data, it indicates that it is impossible to determine whether the target speaker is the speaker of any one of the sample audio data in the sample set, and currently, when trying to obtain audio data in which any one of the target speakers produces pronunciation content in any one of the audio data, the tone feature of the audio data is obtained through the tone extraction network in the tone conversion model, and the obtained tone feature is determined as the tone feature of the target speaker. According to the target speaker information, a many-to-many tone conversion manner can be subsequently realized, that is, the pre-trained tone conversion model can obtain synthetic audio data that meets the tone feature and pronunciation content according to the tone feature of any audio data and the source audio data of any pronunciation content.

[0141] The source audio data (Reference-wav) and the timbre features of the target speaker were input into a pre-trained timbre transformation model.

[0142] The timbre extraction network (speaker encoder as shown in Figure 4) in the timbre conversion model processes the Mel spectrum of the source audio data to obtain timbre features corresponding to the source audio data, i.e., tone_vector as shown in Figure 4.

[0143] The posterior encoder included in the timbre removal network in the timbre conversion model, as shown in FIG. 4, derives a semantic hidden vector in the source audio data, as shown in FIG. 4, based on the timbre features corresponding to the source audio data and the linear spectrum corresponding to the source audio data, as shown in FIG. sq can be obtained.

[0144] The enhancement sub-network (flow) included in the subtraction tone network in the tone transformation model obtains an enhanced hidden vector based on the hidden vector, that is, determines the semantic feature.

[0145] Using a vocoder (decoder) as shown in FIG. 4, synthetic audio data is obtained based on the semantic features and the tonal features of the target speaker, as well as the speaker inner embedding as shown in FIG. 4, i.e., a raw waveform as shown in FIG. 4 is obtained.

[0146] Example 3: In an embodiment of the present application, a tone transformation model training device is provided. FIG. 5 is a schematic diagram of the structure of the tone transformation model training device provided in the embodiment of the present application, the structure includes: an acquisition unit 51, a processing unit 52 and a training unit 53; The acquiring unit 51 is configured to acquire a sample set, where the sample set includes sample audio data of different speakers, each of the sample audio data corresponds to a target audio data, and the target audio data and the sample audio data have the same semantic information; The processing unit 52 is configured to, for any one of the sample audio data, obtain a first timbre feature of the sample audio data through a timbre extraction network in an original timbre transformation model, obtain a first semantic feature according to the first timbre feature and a linear spectrum corresponding to the sample audio data through a timbre removal network in the original timbre transformation model, where the first semantic feature is a feature related to semantic information in the sample audio data that is independent of a speaker's timbre, and obtain a synthetic audio data according to the first semantic feature and a second timbre feature of a target audio data corresponding to the sample audio data through a vocoder in the original timbre transformation model; The training unit 53 is arranged to obtain a trained timbre transformation model by training the original timbre transformation model according to the target audio data and the synthetic audio data respectively corresponding to each of the sample audio data.

[0147] The principle by which the above-mentioned tone transformation model training device solves the problem is similar to that of the tone transformation model training device method, so the implementation and corresponding beneficial effects of the above-mentioned tone transformation model training device can refer to the implementation and beneficial effects of the method, and no mention will be made of the overlapping parts.

[0148] Example 4: In the embodiment of the present application, a tone conversion device is provided. FIG. 6 is a schematic diagram of the structure of the tone conversion device provided in the embodiment of the present application, which includes an acquisition module 61 and a synthesis module 62; The acquisition module 61 is configured to acquire source audio data and timbre characteristics of a target speaker; The synthesis module 62 is configured to obtain timbre features of the source audio data by a timbre extraction network in a pre-trained timbre transformation model, obtain semantic features based on the timbre features and a linear spectrum corresponding to the source audio data by a timbre removal network in the timbre transformation model, where the semantic features are features in the source audio data that are unrelated to speaker timbre and relate to semantic information, and obtain synthetic audio data based on the semantic features and the timbre features of the target speaker by a vocoder in the timbre transformation model. The principle by which the above-mentioned tone conversion device solves the problem is similar to that of the tone conversion device method, so the implementation of the above-mentioned tone conversion device can refer to the implementation of the method, and no overlapping parts will be mentioned.

[0149] Example 5: FIG. 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, the electronic device includes a processor 71, a communication interface 72, a memory 73, and a communication bus 74, where the processor 71, the communication interface 72, and the memory 73 communicate with each other via the communication bus 74; The memory 73 stores a computer program which, when executed by the processor 71, causes the processor 71 to perform the following steps: Obtain a sample set, where the sample set includes sample audio data of different speakers, each of the sample audio data corresponds to a target audio data, and the target audio data and the sample audio data have the same semantic information; For any one of the sample audio data, a timbre extraction network in an original timbre conversion model obtains a first timbre feature of the sample audio data, and a timbre removal network in the original timbre conversion model obtains a first semantic feature based on the first timbre feature and a linear spectrum corresponding to the sample audio data, where the first semantic feature is a feature related to semantic information and is unrelated to a speaker's timbre in the sample audio data, and a vocoder in the original timbre conversion model obtains synthetic audio data based on the first semantic feature and a second timbre feature of target audio data corresponding to the sample audio data; The original timbre transformation model is trained based on the target audio data and the synthesized audio data corresponding to each of the sample audio data to obtain a trained timbre transformation model.

[0150] The principle by which the above electronic device solves the problem is similar to that of the tone transformation model training method, so the implementation of the above electronic device can refer to Example 1 of the method, and the overlapping parts will not be mentioned, as they are the same as the corresponding beneficial effects, and will not be mentioned here.

[0151] Example 6: FIG. 8 is a schematic diagram of the structure of another electronic device provided in an embodiment of the present application, the electronic device includes a processor 81, a communication interface 82, a memory 83, and a communication bus 84, where the processor 81, the communication interface 82, and the memory 83 communicate with each other via the communication bus 84; The memory 83 stores a computer program which, when executed by the processor 81, causes the processor 81 to perform the following steps: Obtaining source audio data and timbre characteristics of a target speaker; A timbre extraction network in a pre-trained timbre transformation model acquires timbre features of the source audio data, and a timbre removal network in the timbre transformation model acquires semantic features based on the timbre features and a linear spectrum corresponding to the source audio data, where the semantic features are features related to semantic information in the source audio data and are unrelated to speaker timbre, and a vocoder in the timbre transformation model acquires synthetic audio data based on the semantic features and the timbre features of the target speaker.

[0152] The principle by which the above electronic device solves the problem is similar to that of the tone conversion method, so the implementation of the above electronic device can refer to Example 2 of the method, and no overlapping parts will be mentioned.

[0153] The communication bus referred to by the electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, the bus is shown in the figure with only one bold line, but this does not indicate that there is only one bus or one type of bus. The communication interface 82 is arranged to allow the electronic device to communicate with other devices. The memory may include a memory Random Access Memory (RAM) and may also include a Non-Volatile Memory (NVM), such as at least one disk memory. As an option, the memory may be at least one storage device located remotely from the processor.

[0154] The processor may be a general-purpose processor including a central processor, a network processor (NP), etc., or may be a digital signal processing (DSP), an application specific integrated circuit, a field programmable gate display or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0155] Example 7: Based on the above embodiments, the embodiments of the present application also provide a computer-readable storage medium, which stores a computer program executed by a processor. When the program is run on the processor, the program performs the following steps: Obtain a sample set, where the sample set includes sample audio data of different speakers, each of the sample audio data corresponds to a target audio data, and the target audio data and the sample audio data have the same semantic information; For any one of the sample audio data, a timbre extraction network in an original timbre conversion model obtains a first timbre feature of the sample audio data, and a timbre removal network in the original timbre conversion model obtains a first semantic feature based on the first timbre feature and a linear spectrum corresponding to the sample audio data, where the first semantic feature is a feature related to semantic information and is unrelated to a speaker's timbre in the sample audio data, and a vocoder in the original timbre conversion model obtains synthetic audio data based on the first semantic feature and a second timbre feature of target audio data corresponding to the sample audio data; The original timbre transformation model is trained based on the target audio data and the synthesized audio data corresponding to each of the sample audio data to obtain a trained timbre transformation model.

[0156] The principle by which the computer-readable storage medium solves the problem and the beneficial effects obtained accordingly are similar to those of the tone transformation model training method in the above embodiments, so that the specific implementation can refer to the implementation of the tone transformation model training method.

[0157] Example 8: Based on the above embodiments, the embodiments of the present application also provide a computer-readable storage medium, in which a computer program executed by a processor is stored. When the program is run on the processor, the following steps are realized when the processor is made to execute the program: Obtaining source audio data and timbre characteristics of a target speaker; A timbre extraction network in a pre-trained timbre transformation model acquires timbre features of the source audio data, and a timbre removal network in the timbre transformation model acquires semantic features based on the timbre features and a linear spectrum corresponding to the source audio data, where the semantic features are features related to semantic information in the source audio data and are unrelated to speaker timbre, and a vocoder in the timbre transformation model acquires synthetic audio data based on the semantic features and the timbre features of the target speaker.

[0158] The principle by which the computer-readable storage medium solves the problem is similar to that of the tone conversion method in the above embodiment, so that the specific implementation can refer to the implementation of the tone conversion method.

[0159] It will be apparent to those skilled in the art that the embodiments of the present application may be a method, a system, or a computer program product. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. In addition, the present application may take the form of a computer program product embodied in one or more computer usable storage media (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) that contain computer usable program code.

[0160] An embodiment of the present application also provides a computer program product, which includes a computer program, and when the computer program is executed, the steps of the tone conversion model training method as described above or the steps of the tone conversion method as described above can be realized.

[0161] The present application will be described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products of the present application. It will be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate an apparatus, whereby the commands executed by the processor of the computer or other programmable data processing device generate an apparatus for implementing the function specified in a flow or flows in the flowcharts and / or a block or blocks in the block diagrams.

[0162] These computer program instructions may be stored in a computer readable memory that can guide a computer or other programmable data processing device to operate in a particular manner, thereby causing the commands stored in the computer readable memory to generate a product that includes a command apparatus that implements the functions specified in a flow or flows of the flowcharts and / or a block or blocks of the block diagrams.

[0163] These computer program commands may also be loaded into a computer or other programmable data processing device to generate a computer-implemented process by causing the computer or other programmable device to perform a series of operational steps, whereby the commands executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams. [Explanation of symbols]

[0164] 51 Acquired Units 52 Processing Unit 53 Training Unit 61 Acquisition Module 62 Synthesis Module 71 Processor 72 Communication Interface 73 Memory 74 Communication Bus 81 Processor 82 Communication Interface 83 Memory 84 Communication Bus

Claims

1. Obtain a sample set, where the sample set includes sample audio data of different speakers, each of the sample audio data corresponds to a target audio data, and the target audio data and the sample audio data have the same semantic information; For any one of the sample audio data, a timbre extraction network in an original timbre conversion model obtains a first timbre feature of the sample audio data, and a timbre removal network in the original timbre conversion model obtains a first semantic feature based on the first timbre feature and a linear spectrum corresponding to the sample audio data, where the first semantic feature is a feature related to semantic information and is unrelated to a speaker's timbre in the sample audio data; and a vocoder in the original timbre conversion model obtains synthetic audio data based on the first semantic feature and a second timbre feature of target audio data corresponding to the sample audio data; training the original timbre transformation model based on target audio data and synthesized audio data corresponding to each of the sample audio data, respectively, to obtain a trained timbre transformation model; Including, A method for training a tone transformation model.

2. The target audio data corresponding to the sample audio data includes at least one of the sample audio data, sample audio data different from a speaker of the sample audio data, and non-sample audio data different from a speaker of the sample audio data. The method of claim 1.

3. Obtaining a second timbre feature of target audio data corresponding to the sample audio data includes: if the target audio data is the sample audio data, determining a first timbre feature of the sample audio data as the second timbre feature; If the target audio data is not the sample audio data, obtaining a second timbre feature of the target audio data through a timbre extraction network in the original timbre transformation model; Including, The method of claim 2.

4. obtaining a first semantic feature based on the first timbre feature and a linear spectrum corresponding to the sample audio data by a subtraction timbre network in the original timbre transformation model, Obtaining a hidden vector of semantic information in the sample audio data according to the first tonal feature and a linear spectrum corresponding to the sample audio data by a post-encoder in the subtraction tonal network; obtaining the first semantic feature based on the hidden vector by an enhancement sub-network in the subtraction timbre network; Including, The method of claim 1.

5. The method further includes, for any one of the sample audio data, obtaining a second semantic feature based on the sample audio data by a semantic extraction network in the original timbre transformation model; training the original timbre transformation model based on target audio data and synthesized audio data respectively corresponding to each of the sample audio data; training the original timbre transformation model based on target audio data corresponding to each of the sample audio data, synthetic audio data corresponding to each of the sample audio data, first semantic features corresponding to each of the sample audio data, and second semantic features corresponding to each of the sample audio data, to obtain a trained timbre transformation model; The method according to claim 4.

6. obtaining second semantic features based on the sample audio data by a semantic extraction network in the original timbre transformation model; obtaining content features based on the sample audio data by a first content sub-network in the semantic extraction network; Obtaining discretized content features based on the content features by a second content sub-network in the semantic extraction network; obtaining the second semantic features based on the discretized content features by a third content sub-network in the semantic extraction network; Including, The method according to claim 5.

7. Training the original timbre conversion model based on target audio data corresponding to each of the sample audio data, synthetic audio data corresponding to each of the sample audio data, first semantic features corresponding to each of the sample audio data, and second semantic features corresponding to each of the sample audio data, determining a reconstruction loss value based on target audio data corresponding to each of the sample audio data and synthesized audio data corresponding to each of the sample audio data; determining a semantic loss value based on a first semantic feature corresponding to each of the sample audio data and a second semantic feature corresponding to each of the sample audio data; determining an overall loss value according to the reconstruction loss value and the semantic loss value; adjusting parameter values ​​of the parameters in the original timbre conversion model according to the total loss value to obtain a trained timbre conversion model; Including, The method according to claim 5 or 6.

8. The method further includes obtaining, by a post-encoder in the subtraction tonal network, a mean vector and a variance vector of the hidden vector based on the first tonal feature and a linear spectrum corresponding to the sample audio data; determining a semantic loss value based on a first semantic feature corresponding to each of the sample audio data and a second semantic feature corresponding to each of the sample audio data, determining a semantic loss value based on the first semantic feature, the second semantic feature, a mean vector and a variance vector respectively corresponding to each of the sample audio data; 8. The method according to any one of claims 1 to 7.

9. Training the original timbre conversion model based on target audio data corresponding to each of the sample audio data, synthetic audio data corresponding to each of the sample audio data, first semantic features corresponding to each of the sample audio data, and second semantic features corresponding to each of the sample audio data, determining a quantification loss value based on content features corresponding to each of the sample audio data and content features of the discretization corresponding to each of the sample audio data; and determining a comparative training loss value based on discretized content features corresponding to each of the sample audio data and second semantic features corresponding to each of the sample audio data; training the original timbre transformation model based on target audio data corresponding to each of the sample audio data and synthetic audio data corresponding to each of the sample audio data, first semantic features corresponding to each of the sample audio data and second semantic features corresponding to each of the sample audio data, the quantification loss value and the comparative learning loss value to obtain a trained timbre transformation model; Including, The method according to claim 6.

10. After obtaining the trained timbre transformation model, the method comprises: determining timbre features corresponding to the different speakers based on the timbre transformation model and each of the sample audio data in a sample set; Correspondingly storing target markers and tone features corresponding to the different speakers; Further comprising:

10. The method according to any one of claims 1 to 9.

11. Obtaining source audio data and timbre characteristics of a target speaker; Acquire timbre features of the source audio data by a timbre extraction network in a pre-trained timbre transformation model, acquire semantic features based on the timbre features and a linear spectrum corresponding to the source audio data by a timbre removal network in the timbre transformation model, where the semantic features are features related to semantic information in the source audio data that are unrelated to speaker timbre, and acquire synthetic audio data based on the semantic features and the timbre features of the target speaker by a vocoder in the timbre transformation model; Including, Tone conversion method.

12. The acquiring of the timbre characteristics of the target speaker includes: acquiring information of the target speaker; When determining that the information of the target speaker is a target marker, determining a timbre feature corresponding to the target marker of the target speaker according to a correspondence relationship between the target marker and the timbre feature to be stored; When determining that the information of the target speaker is audio data, obtaining timbre features of the audio data through a timbre extraction network in the timbre conversion model; Including, The method of claim 11.

13. An acquisition unit, a processing unit, and a training unit, The acquiring unit is configured to acquire a sample set, where the sample set includes sample audio data of different speakers, each of the sample audio data corresponds to a target audio data, and the target audio data and the sample audio data have the same semantic information; The processing unit is configured to, for any one of the sample audio data, obtain a first timbre feature of the sample audio data through a timbre extraction network in an original timbre conversion model, obtain a first semantic feature based on the first timbre feature and a linear spectrum corresponding to the sample audio data through a timbre removal network in the original timbre conversion model, where the first semantic feature is a feature related to semantic information and is unrelated to a speaker's timbre in the sample audio data; and obtain a synthetic audio data based on the first semantic feature and a second timbre feature of a target audio data corresponding to the sample audio data through a vocoder in the original timbre conversion model; The training unit is configured to: train the original timbre transformation model according to target audio data and synthetic audio data respectively corresponding to each of the sample audio data, thereby obtaining a trained timbre transformation model; A training device for tone transformation models.

14. An acquisition module and a synthesis module, The acquisition module is configured to acquire source audio data and timbre features of a target speaker; The synthesis module is configured to obtain timbre features of the source audio data by a timbre extraction network in a pre-trained timbre transformation model, and obtain semantic features according to the timbre features and a linear spectrum corresponding to the source audio data by a timbre removal network in the timbre transformation model, where the semantic features are features related to semantic information in the source audio data that are independent of speaker timbre, and are configured to obtain synthetic audio data according to the semantic features and the timbre features of the target speaker by a vocoder in the timbre transformation model. Tone conversion device.

15. The system includes at least a processor and a memory, The processor is arranged to implement the steps of the method for training a timbre transformation model according to any one of claims 1 to 10 or the steps of the method for timbre transformation according to any one of claims 11 to 12 when executing a computer program stored in a memory. Electronic devices.

16. A computer program is stored in the When the computer program is executed by a processor, the computer program causes the steps of the tone conversion model training method according to any one of claims 1 to 10 or the steps of the tone conversion method according to any one of claims 11 to 12 to be realized. A computer-readable storage medium.

Citation Information

Patent Citations

  • Prior distribution calculation apparatus, voice recognition apparatus, prior distribution calculation method, voice recognition method, and program

    JP2013178343A

  • Voice conversion device, voice conversion method and program

    JP2019109306A

  • Speech recognition method and device

    JP2019528476A

  • Generating a voice model for a user

    US20210142782A1

  • Acoustic model learning method, voice recognition method, acoustic model learning device, voice recognition device, acoustic model learning program, and voice recognition program

    WO2017135148A1