Speech conversion method, speech model training method, device, equipment and medium

Abstract

The present application is suitable for the field of speech conversion, and particularly relates to a speech conversion method, a speech model training method, a device, equipment and a medium. In the embodiment of the present application, sample speech waveform is input to a content encoder to obtain content features, reference speech waveform is input to a speech encoder to obtain speaker features, and the content features and the speaker features are input to a generator for N times of upsampling processing and residual connection processing to obtain N audio waveforms with preset frequencies increasing in turn. Through a step-by-step method of gradually adding upsampling at the generator end and gradually adding downsampling at the discriminator end, the training speed of the speech conversion model is improved, and the stability and accuracy of the audio waveform are improved. Discrimination results and fundamental frequencies are obtained through the discriminator and the fundamental frequency extraction model to supervise the training of the speech conversion model, ensuring the content, timbre and tone invariability of the generated speech waveform and improving the accuracy of the speech conversion model.

Description

TECHNICAL FIELD

[0001] The present application is suitable for the field of speech conversion, and particularly relates to a speech conversion method, a speech model training method, an apparatus, a device and a medium. BACKGROUND

[0002] At present, with the development of artificial intelligence technology, speech conversion technology has been gradually applied to intelligent question answering, voice interaction and other application scenarios. The existing speech conversion technology mainly relies on a deep neural network model for speech conversion, but the deep neural network model needs a large number of labels corresponding to the speech samples, which greatly increases the data preparation cost.

[0003] To solve the above problems, the existing technology proposes a speech conversion method based on an adversarial generative model, which can reduce the data preparation cost, but when processing high-quality speech generation tasks, the speech conversion method directly uses the discriminative loss of the adversarial generative model for model training, and simultaneously outputs audio waveforms at a fixed frequency, which reduces the stability and fidelity of the audio waveforms, resulting in poor speech conversion effect.

[0004] Therefore, in the field of speech conversion technology, how to improve the accuracy of the speech conversion model has become a problem to be solved. SUMMARY

[0005] Therefore, the embodiments of the present application provide a speech conversion method, a speech model training method, an apparatus, a device and a medium to solve the problem of low accuracy of the existing speech conversion model.

[0006] In a first aspect, the embodiments of the present application provide a speech model training method, which comprises:

[0007] obtaining an initial speech conversion model, the speech conversion model comprising a content encoder, a speech encoder and a generator, inputting a sample speech waveform into the content encoder to obtain a content feature, inputting a reference speech waveform of a reference speaker into the speech encoder to obtain a speaker feature;

[0008] inputting the content feature and the speaker feature into the generator, performing N times of upsampling processing and residual connection processing by using the generator to obtain N audio waveforms with a preset frequency increasing in turn, wherein N is a positive integer;

[0009] inputting the N audio waveforms into a trained discriminator respectively to obtain N discrimination results, and inputting the N audio waveforms into a trained fundamental frequency extraction model respectively to obtain N generated fundamental frequencies;

[0010] According to the N discrimination results and the N generated fundamental frequencies, N generated losses corresponding to the N preset frequencies are calculated, and the speech conversion model is trained according to the N preset frequencies and the N generated losses to obtain a trained speech conversion model.

[0011] In a second aspect, an embodiment of the present application provides a speech conversion method, which comprises:

[0012] The trained speech conversion model comprises a trained content encoder, a trained speech encoder and a trained generator.

[0013] A to-be-processed speech waveform and a target reference speech waveform are obtained, the to-be-processed speech waveform is input into the trained content encoder to obtain target content features, and the target reference speech waveform is input into the trained speech encoder to obtain target speaker features.

[0014] The target content features and the target speaker features are input into the trained generator to obtain N audio waveforms with preset frequencies successively increased.

[0015] The audio waveform corresponding to the highest preset frequency is determined as a target audio waveform.

[0016] In a third aspect, an embodiment of the present application provides a speech model training device, which comprises:

[0017] A feature extraction module is configured to obtain an initial speech conversion model, the speech conversion model comprising a content encoder, a speech encoder and a generator, input a sample speech waveform into the content encoder to obtain content features, and input a reference speech waveform of a reference speaker into the speech encoder to obtain speaker features.

[0018] A waveform generation module is configured to input the content features and the speaker features into the generator, perform N times of upsampling processing and residual connection processing by using the generator, and obtain N audio waveforms with preset frequencies successively increased, wherein N is a positive integer.

[0019] A waveform processing module is configured to input the N audio waveforms into a trained discriminator respectively to obtain N discrimination results, and input the N audio waveforms into a trained fundamental frequency extraction model respectively to obtain N generated fundamental frequencies.

[0020] The model training module is configured to calculate N generation losses corresponding to the N preset frequencies respectively according to the N discrimination results and the N generated fundamental frequencies, and train the speech conversion model according to the N preset frequencies and the N generation losses to obtain a trained speech conversion model.

[0021] In a fourth aspect, an embodiment of the present application provides a computer device, which comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the speech model training method according to the first aspect when executing the computer program.

[0022] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the speech model training method according to the first aspect.

[0023] Compared with the prior art, the embodiment of the present application has the beneficial effects that: the sample speech waveform is input into the content encoder to obtain the content feature, the reference speech waveform of the reference speaker is input into the speech encoder to obtain the speaker feature, and the content feature and the speaker feature are input into the generator, the N times of upsampling processing and the residual connection processing are performed on the generator, N audio waveforms with sequentially increasing preset frequencies are obtained, then the N audio waveforms are input into the trained discriminator to obtain N discrimination results, the N audio waveforms are input into the trained fundamental frequency extraction model to obtain N generated fundamental frequencies, N generation losses corresponding to the N preset frequencies are calculated according to the N discrimination results and the N generated fundamental frequencies, the speech conversion model is trained according to the N preset frequencies and the N generation losses, and a trained speech conversion model is obtained; the step-by-step method of gradually adding upsampling at the generator end and gradually adding downsampling at the discriminator end improves the training speed of the speech conversion model, improves the stability and accuracy of the audio waveform, and obtains the discrimination results and the fundamental frequencies through the discriminator and the fundamental frequency extraction model, which are used to calculate the loss function to supervise the training of the speech conversion model, improve the fitting precision of the speech conversion model, and further ensure the content and timbre invariance of the speech waveform, thereby improving the accuracy of the speech conversion model. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1 is an application environment schematic diagram of a speech model training method provided by an embodiment of the present application;

[0026] Figure 2 is a flow schematic diagram of a speech model training method provided by an embodiment of the present application;

[0027] Figure 3 is a structure schematic diagram of a speech conversion model in an embodiment of the present application;

[0028] Figure 4 is a flow schematic diagram of a speech conversion method provided by an embodiment of the present application;

[0029] Figure 5 is a structure schematic diagram of a speech model training device provided by an embodiment of the present application;

[0030] Figure 6 is a structure schematic diagram of a generator and a discriminator in an embodiment of the present application;

[0031] Figure 7 is a structure schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0032] In the following description, for the purposes of explanation and not limitation, specific details are set forth, such as particular sequences of steps, techniques, etc. in order to provide a thorough understanding of the embodiments of the application. However, it will be apparent to those skilled in the art that the application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, and circuits are omitted so as not to obscure the description of the application with unnecessary detail.

[0033] It should be understood that the term "comprises" when used in this specification and the appended claims indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0034] It should also be understood that the term "and / or" when used in this specification and the appended claims indicates that one or more of the associated listed items can be present, and, further, that one or more of such associated listed items can be realized.

[0035] As used in the description of the application and the appended claims, the term "if' can be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the described condition or event]" or "in response to detecting [the described condition or event]," depending on the context.

[0036] In addition, in the description of the application and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0037] In the description of the application, the reference "one embodiment" or "some embodiments" and the like means that the specific features, structures or characteristics described in connection with the embodiment are included in one or more embodiments of the application. Therefore, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in further some embodiments" and the like appearing in different places in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "include", "contain", "have" and their variants mean "include but not limited to", unless otherwise specifically emphasized.

[0038] The embodiments of the application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0039] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0040] It should be understood that the size of the serial number of each step in the following embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the application.

[0041] In order to illustrate the technical solutions of the application, the following specific embodiments are described. In order to illustrate the technical solutions of the application, the following specific embodiments are described.

[0042] The speech model training method provided by the embodiment one of the present application can be applied in the application environment as shown in Figure 1 The client can be a computer device such as a palm computer, a desktop computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cloud computer device, a personal digital assistant (PDA), etc. The server can be an independent server or a server cluster composed of multiple servers.

[0043] Referring to Figure 2 is a flowchart of the speech model training method provided by the embodiment one of the present application, which can be applied in the client in Figure 1 The speech model training method can include the following steps:

[0044] In step S201, an initial speech conversion model is obtained, which includes a content encoder, a speech encoder and a generator. A sample speech waveform is input into the content encoder to obtain content features, and a reference speech waveform of a reference speaker is input into the speech encoder to obtain speaker features.

[0045] The speech conversion model is used to convert an input speech into a target speech while keeping the speech content unchanged. In this embodiment, the speech conversion model includes a content encoder, a speech encoder and a generator. The content encoder is used to extract features from an input speech waveform to obtain content features, which are used to represent content information in the speech waveform. The speech encoder is used to extract features from the input speech waveform to obtain speaker features, which are used to represent timbre information and pitch information of a speaker corresponding to the speech waveform.

[0046] In the training process of the speech conversion model, a large number of sample speech waveforms and reference speech waveforms of reference speakers are used as training samples. The sample speech waveforms are obtained by converting sample speeches into waveforms, and the reference speech waveforms are obtained by converting speeches of the reference speakers into waveforms.

[0047] Specifically, the sample speech waveform is input into the content encoder to obtain the content features, and the reference speech waveform of the reference speaker is input into the speech encoder to obtain the speaker features, so as to ensure that the sample speech is converted into the target speech while keeping the speech content unchanged.

[0048] Optionally, inputting the reference speech waveform of the reference speaker into the speech encoder to obtain the speaker features includes:

[0049] inputting the reference speech waveform into a speech encoder to obtain a mean of the reference speech waveform and a variance of the reference speech waveform;

[0050] determining a reference speech waveform distribution according to the mean of the reference speech waveform and the variance of the reference speech waveform;

[0051] sampling the reference speech waveform distribution to determine a speaker feature as a sampling result.

[0052] The speaker feature is obtained by feature extraction on the reference speech waveform. In this embodiment, the reference speech waveform distribution of the reference speech waveform is first obtained, and the speaker feature is extracted by sampling the reference speech waveform distribution.

[0053] Specifically, the reference speech waveform is input into the speech encoder to obtain the mean of the reference speech waveform and the variance of the reference speech waveform, and further, the reference speech waveform distribution is determined according to the mean of the reference speech waveform and the variance of the reference speech waveform. For example, according to the speech encoder, the mean of the reference speech waveform is μ, and the variance of the reference speech waveform is σ 2 The corresponding reference speech waveform distribution is (μ, σ 2 ), and further, the reference speech waveform distribution is sampled to determine the speaker feature as a sampling result, which is used to represent the timbre information and the pitch information of the corresponding reference speaker, and is used as the timbre and pitch basis of the speech conversion, thereby improving the fidelity of the timbre and pitch in the speech conversion model.

[0054] The above steps of obtaining the initial speech conversion model include the steps of inputting the sample speech waveform into the content encoder to obtain the content feature, inputting the reference speech waveform of the reference speaker into the speech encoder to obtain the speaker feature, obtaining the content feature representing the content information in the sample speech waveform, and obtaining the speaker feature representing the timbre and pitch information of the corresponding reference speaker, thereby improving the content fidelity and the timbre and pitch fidelity in the speech conversion model.

[0055] In step S202, the content feature and the speaker feature are input into the generator, and the generator is used for N times of upsampling processing and residual connection processing to obtain N audio waveforms with a preset frequency increasing in turn.

[0056] The generator is used for outputting the audio waveform corresponding to the preset frequency according to the input content feature and speaker feature, and the audio waveform is the speech waveform corresponding to the target speech.

[0057] In the process that the speech conversion model outputs an audio waveform, when a preset frequency is high, the resolution of the audio waveform is also high, and the audio waveform contains more details; when the preset frequency is low, the resolution of the audio waveform is also low, and the audio waveform contains fewer details. Generally, the higher the resolution of the audio waveform, the better the quality of the audio waveform. However, because the stability of the speech conversion model in directly outputting a high-resolution audio waveform is poor, in the training process of the speech conversion model, in order to improve the stability of the audio waveform, the preset frequency is increased in multiple times of upsampling processing and residual connection processing, and N audio waveforms with preset frequencies that are successively increased are obtained. Specifically, first, a smaller preset frequency is taken as a generation target of a generator, the content feature and the speaker feature are input into the generator, and a low-frequency audio waveform is obtained. Because the low-frequency audio waveform has high stability, the accuracy of the low-frequency audio waveform can be assumed to be high.

[0058] Then, the size of the preset frequency is increased, the content feature, the speaker feature, and the low-frequency audio waveform are input into the generator, and a higher-frequency audio waveform is obtained.

[0059] Because the high-frequency audio waveform is further output on the basis of the low-frequency audio waveform, the high-frequency audio waveform can retain the information of the low-frequency audio waveform, and degradation of the audio waveform with the increase of the preset frequency is avoided, where the degradation is that the accuracy of the audio waveform is lower and lower with the increase of the preset frequency.

[0060] In the implementation process, the implementer can set multiple different preset frequencies, and iterate in the order of the preset frequencies from small to large, so as to obtain an audio waveform with higher resolution. For example, the preset frequencies can be set as 500 Hz, 1 KHz, 2 KHz, 4 KHz, and the like in the order from small to large.

[0061] Optionally, the content feature and the speaker feature are input into the generator, and the generator is used for N times of upsampling processing and residual connection processing, to obtain N audio waveforms with preset frequencies that are successively increased.

[0062] The generator comprises an upsampling module, a residual connection module, and a decoder.

[0063] The content feature and the speaker feature are input into the generator, the content feature is first upsampling processed according to the upsampling module and a first preset frequency, to obtain a first sampling result, the first sampling result and the speaker feature are input into the residual connection module, to obtain a first fusion feature, and the first fusion feature is input into the decoder, to obtain a first audio waveform corresponding to the first preset frequency.

[0064] input the content feature, the speaker feature and the first audio waveform to the generator, perform the second time upsampling on the content feature and the first audio waveform according to the upsampling module and the second preset frequency, obtain a second sampling result, input the second sampling result and the speaker feature to the residual connection module, obtain a second fusion feature, input the second fusion feature to the decoder, and obtain a second audio waveform corresponding to the second preset frequency;

[0065] By analogy, input the content feature, the speaker feature and the i-th audio waveform to the generator, perform the i+1 time upsampling on the content feature and the i-th audio waveform according to the upsampling module and the i-th preset frequency, obtain an i+1 sampling result, input the i+1 sampling result and the speaker feature to the residual connection module, obtain an i+1 fusion feature, input the i+1 fusion feature to the decoder, and obtain an i+1 audio waveform corresponding to the i+1 preset frequency.

[0066] The upsampling module is configured to sample the content feature and the audio waveform obtained after the last time upsampling and residual connection processing according to a preset frequency, and obtain a sampling result; the residual connection module is configured to add and fuse the sampling result and the speaker feature, and obtain a fusion feature; and the decoder is configured to decode the fusion feature, and generate an audio waveform.

[0067] Specifically, input the content feature, the speaker feature and the i-th audio waveform obtained after the i time sampling processing and residual connection processing to the generator, perform the i+1 time upsampling on the content feature and the i-th audio waveform according to the upsampling module and the i-th preset frequency, obtain an i+1 sampling result, input the i+1 sampling result and the speaker feature to the residual connection module, obtain an i+1 fusion feature, input the i+1 fusion feature to the decoder, and obtain an i+1 audio waveform corresponding to the i+1 preset frequency, wherein i = 1, 2, …, N, and the i-th audio waveform is zero when i = 0; with the increase of the sampling times, N audio waveforms with preset frequencies increasing in turn are obtained.

[0068] The above step of inputting the content feature and the speaker feature to the generator, performing the N time upsampling processing and the residual connection processing by using the generator, and obtaining the N audio waveforms with preset frequencies increasing in turn is performed by the step-by-step generation manner, and with the increase of the sampling times, the corresponding preset frequency gradually increases, and the audio waveform obtained after the last time sampling processing and residual connection processing is sampled at a higher frequency. On the basis of generating the audio waveform of a low frequency, the audio waveform of a high frequency is gradually output, the information of the audio waveform of each frequency is effectively reserved, the generation accuracy of the audio waveform is improved, and then the accuracy of the speech conversion model is improved.

[0069] Referring to Figure 3is a structural schematic diagram of a voice conversion model of the embodiment, and the voice conversion model comprises a content encoder, a speaker encoder and a generator, wherein the generator comprises an up-sampling module, a residual connection module and a decoder.

[0070] The content encoder is configured to perform feature extraction on the input sample voice waveform Y and output a content feature X, the speaker encoder is configured to perform feature extraction on the input reference voice waveform K and output a speaker feature Z, the up-sampling module is configured to sample the content feature X and the audio waveform G obtained after the last up-sampling processing and residual connection processing to obtain a sampling result C, the residual connection module is configured to add and fuse the sampling result C and the speaker feature Z to obtain a fusion feature R, and the decoder is configured to decode the input fusion feature R to obtain the audio waveform G.

[0071] In step S203, the N audio waveforms are respectively input into the trained discriminator to obtain N discrimination results, and the N audio waveforms are respectively input into the trained fundamental frequency extraction model to obtain N generated fundamental frequencies.

[0072] The trained discriminator and the up-sampling module in the generator have a mirror network structure, including the same number of sampling times and preset frequencies. The trained discriminator is configured to perform N times of down-sampling processing on the corresponding N audio waveforms in the order from large to small according to the N preset frequencies, to determine whether each audio waveform is a waveform meeting the speaker feature. The output category of the discriminator is binary classification, wherein one category is that the audio waveform meets the speaker feature, and the other category is that the audio waveform does not meet the speaker feature.

[0073] The trained fundamental frequency extraction model can be used to extract the fundamental frequency information of the waveform to obtain the generated fundamental frequency, so as to determine whether the fundamental frequency of the audio waveform is different from the fundamental frequency of the sample voice waveform, thereby serving as the basis for calculating the generation loss and the basis for training the voice conversion model.

[0074] The above steps of inputting the N audio waveforms into the trained discriminator to obtain N discrimination results and inputting the N audio waveforms into the trained fundamental frequency extraction model to obtain N generated fundamental frequencies, through the discriminator and the fundamental frequency extraction model, the discrimination results and the fundamental frequencies are obtained, which facilitates subsequent calculation of the loss function to supervise the training of the voice conversion model, improves the fitting precision of the voice conversion model, and further improves the accuracy of the voice conversion model.

[0075] Optionally, the N audio waveforms are respectively input into the trained discriminator to obtain N discrimination results, including:

[0076] The real waveform of the sample voice waveform is obtained, and the real waveform is down-sampled according to each preset frequency to obtain a real sampling waveform corresponding to each preset frequency.

[0077] The real sampling waveform and the audio waveform corresponding to each preset frequency are input to the trained discriminator to obtain a discrimination result corresponding to each preset frequency.

[0078] The real waveform can be an ideal waveform of the sample speech waveform after speech conversion, and the frequency of the real waveform should be greater than or equal to the maximum frequency in the preset frequency. The real sampling waveform obtained by downsampling can be embedding information of the audio waveform corresponding to the frequency.

[0079] Specifically, the real sampling waveform and the audio waveform are input to the trained discriminator to obtain a discrimination result, which is used to represent whether the real sampling waveform and the audio waveform are consistent.

[0080] In this embodiment, the real sampling waveform is used as the embedding information input of the trained discriminator, which can obtain a more accurate discrimination result, is conducive to the accurate fitting of the subsequent speech interaction model, and improves the accuracy of the trained speech interaction model.

[0081] In step S204, N generation losses respectively corresponding to N preset frequencies are calculated according to N discrimination results and N generated fundamental frequencies. The speech conversion model is trained based on the N preset frequencies and the N generation losses to obtain a trained speech conversion model.

[0082] The generation loss can be used to supervise the training of the speech conversion model. The training process can use the gradient descent method, that is, for any parameter in the speech conversion model, the partial derivative of the generation loss with respect to the parameter is calculated, so that the direction of parameter update is provided according to the obtained partial derivative. It should be noted that the parameter update uses the back propagation algorithm.

[0083] The above step of calculating N generation losses respectively corresponding to N preset frequencies according to N discrimination results and N generated fundamental frequencies, and training the speech conversion model based on the N preset frequencies and the N generation losses to obtain a trained speech conversion model, trains the speech conversion model through a pre-designed loss function, so that the training process can adjust the parameters according to specific constraints, and improves the accuracy of the trained speech conversion model.

[0084] Optionally, calculating N generation losses respectively corresponding to N preset frequencies according to N discrimination results and N generated fundamental frequencies includes:

[0085] The sample speech waveform is input to the trained fundamental frequency extraction model to obtain a sample fundamental frequency.

[0086] N fundamental frequency losses respectively corresponding to N preset frequencies are calculated according to N generated fundamental frequencies, a sample fundamental frequency, and a preset fundamental frequency loss function.

[0087] According to the fundamental frequency loss, the N discrimination results and the preset generation loss function corresponding to the N preset frequencies respectively, N generation losses corresponding to the N preset frequencies are calculated.

[0088] Wherein, the sample fundamental frequency can refer to the fundamental frequency of the sample speech waveform, and the fundamental frequency loss function can refer to the mean square error loss function. The N fundamental frequency losses are calculated by substituting the sample fundamental frequency and the N generated fundamental frequencies into the fundamental frequency loss function respectively.

[0089] The discrimination result can be calculated by using the cross-entropy loss function to obtain the discrimination loss. Then, for any preset frequency, the discrimination loss corresponding to the preset frequency is added to the fundamental frequency loss corresponding to the preset frequency to obtain the generation loss corresponding to the preset frequency.

[0090] In this embodiment, the difference between the sample fundamental frequency and the generated fundamental frequency is used as the fundamental frequency loss to supervise the speech conversion model to learn the fundamental frequency information, which can make the training process of the speech conversion model converge quickly, improve the efficiency of the speech conversion model training process, and also improve the accuracy of the trained speech conversion model.

[0091] Optionally, according to the N generated fundamental frequencies, the sample fundamental frequency and the preset fundamental frequency loss function, N fundamental frequency losses corresponding to the N preset frequencies are calculated, including:

[0092] N first losses between the sample fundamental frequency and the N generated fundamental frequencies are calculated respectively;

[0093] According to the preset frequency corresponding to each generated fundamental frequency, the generation order of each generated fundamental frequency is determined, and the adjacent fundamental frequency corresponding to each generated fundamental frequency is determined according to the generation order.

[0094] N second losses between the N generated fundamental frequencies and the corresponding adjacent fundamental frequencies are calculated respectively.

[0095] Each first loss and each second loss are substituted into the preset fundamental frequency loss function to calculate N fundamental frequency losses corresponding to the N preset frequencies respectively.

[0096] Wherein, the first loss can be used to represent the difference between the sample fundamental frequency and the generated fundamental frequency, and the second loss can be used to represent the difference between the generated fundamental frequency and its adjacent fundamental frequency.

[0097] Specifically, the first loss can be used to supervise the speech conversion model to learn the sample fundamental frequency information, and the second loss can be used to supervise the speech conversion model to learn the adjacent fundamental frequency information, thereby providing more accurate difference information and enabling more targeted parameter adjustment.

[0098] For example, the i-th (i=1, 2, …, N) generated fundamental frequency corresponding to the i-th (i=1, 2, …, N) preset frequency is denoted as Let the sample fundamental frequency be denoted as The first loss is used to represent the difference between the sample fundamental frequency and each generated fundamental frequency, and is calculated by the sample fundamental frequency The i-th generated fundamental frequency The i-th first loss corresponding to the i-th preset frequency can be calculated as follows:

[0099]

[0100] In the formula, l i1 is the i-th first loss, is the sample fundamental frequency, is the i-th generated fundamental frequency corresponding to the i-th preset frequency.

[0101] Then, first, the i-th generated fundamental frequency corresponding to the adjacent fundamental frequency is the i+1-th generated fundamental frequency according to the generation order, and the second loss is used to represent the difference between the generated fundamental frequency and its adjacent fundamental frequency, and is calculated by the i-th generated fundamental frequency The i+1-th generated fundamental frequency The i-th second loss corresponding to the i-th preset frequency can be calculated as follows:

[0102]

[0103] In the formula, l i2 is the i-th second loss, is the i-th generated fundamental frequency corresponding to the i-th preset frequency, is the adjacent fundamental frequency of the i-th generated fundamental frequency.

[0104] Then, the i-th first loss and the i-th second loss are substituted into the preset fundamental frequency loss function to calculate the i-th fundamental frequency loss corresponding to the i-th preset frequency, wherein the preset fundamental frequency loss function is as follows:

[0105] l i = a1l i1 + a2l i2

[0106] In the formula, l i is the i-th fundamental frequency loss, a1 is the weight of the i-th first loss, l i1 is the i-th first loss, a2 is the weight of the i-th second loss, l i2 is the i-th second loss.

[0107] The weights α1 for the first loss and α2 for the second loss can be set according to actual conditions. In this embodiment, α1 = 0.6 and α2 = 0.4 are set according to actual conditions. This embodiment supervises the training of the speech conversion model by generating the difference between the fundamental frequency and its adjacent fundamental frequencies, so as to ensure that the fundamental frequency of the audio waveform remains consistent under different preset frequencies. During training, the parameters that need to be adjusted can be determined more quickly, thereby improving the accuracy of the trained speech conversion model.

[0108] Optionally, the speech conversion model is trained based on N preset frequencies and N generation losses to obtain a trained speech conversion model, including:

[0109] According to the preset weight mapping function, N preset frequencies are mapped to corresponding N loss weights;

[0110] Multiply the N generation losses and their corresponding N loss weights to obtain N multiplication results. Add the N multiplication results to obtain the model loss. Based on the model loss, train the speech conversion model until the model loss converges to obtain the trained speech conversion model.

[0111] Among them, the loss weight can characterize the degree of influence of the generation loss at each preset frequency on the model loss.

[0112] Specifically, due to the different preset frequencies, the difficulty of generating the audio waveform corresponding to the preset frequency is also different. The higher the preset frequency, the more detailed information the audio waveform contains, and the more the model needs to pay attention to it during training in order to learn enough detailed information. Therefore, the weight mapping function is a positive correlation function, and the higher the preset frequency, the greater the corresponding loss weight.

[0113] For example, the preset mapping function can be to normalize N preset frequencies, and use the normalization result corresponding to each preset frequency as the corresponding loss weight. Specifically, the i-th preset frequency is denoted as f. i Then the weight loss for the i-th preset frequency is:

[0114]

[0115] In the formula, q i The weight loss is the weight for the i-th preset frequency, where N is the number of preset frequencies, and f i This is the i-th preset frequency.

[0116] Then, each preset frequency and its corresponding weight loss are multiplied together, and the N multiplication results are summed to obtain the model loss. Specifically, the generation loss corresponding to the i-th preset frequency is denoted as L. i Then the model loss is:

[0117]

[0118] In the formula, L is the model loss, q i For the weight loss of the i-th preset frequency, L i This is the generation loss corresponding to the i-th preset frequency.

[0119] In this embodiment, different weights are assigned to the generation loss of different preset frequencies, so that the speech conversion model pays more attention to detailed information during the training process, which can effectively improve the speech conversion model's ability to learn detailed information and improve the accuracy of the trained speech conversion model.

[0120] In this embodiment of the invention, a sample speech waveform is input into a content encoder to obtain content features, a reference speaker's speech waveform is input into a speech encoder to obtain speaker features, and the content features and speaker features are input into a generator. The generator performs N upsampling and residual connection processing to obtain N audio waveforms with sequentially increasing preset frequencies. These N audio waveforms are then input into a trained discriminator to obtain N discrimination results. The N audio waveforms are also input into a trained fundamental frequency extraction model to obtain N generated fundamental frequencies. Finally, based on the N discrimination results and the N generated fundamental frequencies, N values ​​corresponding to the N preset frequencies are calculated. The speech conversion model is trained using N preset frequencies and N generation losses to obtain a trained speech conversion model. By progressively adding upsampling at the generator end and downsampling at the discriminator end, the training speed of the speech conversion model is improved, while the stability and accuracy of the audio waveform are also improved. The discrimination result and fundamental frequency are obtained through the discriminator and fundamental frequency extraction model, which are used to calculate the loss function to supervise the training of the speech conversion model, thereby improving the fitting accuracy of the speech conversion model. This ensures the invariance of the content, timbre and pitch of the speech waveform and improves the accuracy of the speech conversion model.

[0121] Corresponding to the speech model training method in Example 1 above, Figure 4 A flowchart illustrating the speech conversion method provided in Embodiment 2 of the present invention is given. The speech conversion method includes:

[0122] Step S401: Obtain the trained speech conversion model, which includes a trained content encoder, a trained speech encoder, and a trained generator.

[0123] The trained content encoder is used to extract features from the input speech waveform to obtain content features, which represent the content information in the speech waveform. The trained speech encoder is used to extract features from the input speech waveform to obtain speaker features, which represent the timbre and pitch information of the speaker corresponding to the speech waveform.

[0124] Step S402: Obtain the speech waveform to be processed and the target reference speech waveform. Input the speech waveform to be processed into the trained content encoder to obtain the target content features. Input the target reference speech waveform into the trained speech encoder to obtain the target speaker features.

[0125] The target reference speech waveform is used to provide timbre and pitch information during speech conversion.

[0126] Specifically, the speech waveform to be processed is input into the trained content encoder to obtain target content features, which are used to represent the content information in the speech waveform to be processed; the target reference speech waveform is input into the trained speech encoder to obtain target speaker features, which are used to represent the timbre and pitch information of the speaker corresponding to the target reference speech waveform.

[0127] Step S403: Input the target content features and target speaker features into the trained generator to obtain N audio waveforms with preset frequencies increasing sequentially.

[0128] The trained generator includes a trained upsampling module, a trained residual connection module, and a trained decoder. The target content features and target speaker features are input into the trained generator, which then performs N upsampling operations, N residual connection operations, and N decoding operations based on N sequentially increasing preset frequencies, resulting in an audio waveform with N sequentially increasing preset frequencies.

[0129] Step S404: Determine the audio waveform corresponding to the highest preset frequency as the target audio waveform.

[0130] Among them, the audio waveform corresponding to the highest preset frequency has the highest resolution, and the high-frequency audio waveform is gradually generated based on the low-frequency audio waveform, effectively preserving the information of the audio waveform at each frequency. Therefore, among the N audio waveforms, the audio waveform corresponding to the highest preset frequency is determined as the target audio waveform.

[0131] In this embodiment, a trained speech conversion model is used to process the speech waveform to be processed and the target reference speech waveform to obtain N audio waveforms with successively increasing preset frequencies. The audio waveform corresponding to the highest preset frequency is then determined as the target audio waveform. Since the target audio waveform effectively retains the information of the audio waveforms at each frequency, the accuracy of speech conversion is effectively improved.

[0132] Corresponding to the speech model training method in Example 1 above, Figure 5 A structural block diagram of the speech model training device provided in Embodiment 3 of the present invention is given. For ease of explanation, only the parts related to the embodiments of the present invention are shown.

[0133] See Figure 5 The speech model training device includes:

[0134] The feature extraction module 51 is used to obtain an initial speech conversion model, which includes a content encoder, a speech encoder, and a generator. The sample speech waveform is input to the content encoder to obtain content features, and the reference speech waveform of the reference speaker is input to the speech encoder to obtain speaker features.

[0135] The waveform generation module 52 is used to input content features and speaker features into the generator, and use the generator to perform N upsampling processes and residual connection processes to obtain N audio waveforms with successively increasing preset frequencies.

[0136] The waveform processing module 53 is used to input N audio waveforms into the trained discriminator to obtain N discrimination results, and to input N audio waveforms into the trained fundamental frequency extraction model to obtain N generated fundamental frequencies.

[0137] The model training module 54 is used to calculate N generation losses corresponding to N preset frequencies based on N discrimination results and N generation fundamental frequencies. The speech conversion model is trained based on the N preset frequencies and N generation losses to obtain the trained speech conversion model.

[0138] In this embodiment, the generator includes N upsampling submodules, N residual connection submodules, and N decoders, and the discriminator includes N downsampling submodules and N discrimination submodules.

[0139] Wherein, the output of the j-th upsampling submodule is connected to the input of the j-th residual connection submodule, the output of the j-th residual connection submodule is connected to the input of the j-th decoder, the output of the j-th decoder is connected to the input of the (j+1)-th upsampling submodule and the input of the j-th downsampling module, and the output of the j-th downsampling module is connected to the input of the j-th discriminant submodule, where j = 1, 2, ..., N-1.

[0140] The content feature input is connected to the input terminals of each upsampling submodule, the speaker feature input is connected to the input terminals of each residual connection submodule, the real waveform input is connected to the input terminals of each downsampling module, and the output terminals of each discrimination submodule are connected to the discrimination results.

[0141] For example, by stacking N upsampling submodules, N residual connection submodules, and N decoders at the generator end, and stacking N downsampling submodules and N discriminator submodules at the discriminator end, see [reference]. Figure 6The diagram below is a schematic of the generator and discriminator model structure in this embodiment. Taking N=6 as an example, the generator is divided into 6 layers. Each layer is equipped with an upsampling submodule, a residual connection submodule, and a decoder. After processing by the first layer upsampling submodule, residual connection submodule, and decoder, an audio waveform with a frequency of 500Hz is obtained. After processing by the second layer upsampling submodule, residual connection submodule, and decoder, an audio waveform with a frequency of 1KHz is obtained. And so on. After processing by the third, fourth, fifth, and sixth layers upsampling submodule, residual connection submodule, and decoder, audio waveforms with frequencies of 2KHz, 4KHz, 8KHz, and 16KHz are obtained, respectively.

[0142] Correspondingly, the discriminator is divided into 6 layers. Each layer has a downsampling submodule and a discrimination submodule, and the real waveform is input as the basis for discrimination. After processing by the first layer sampling submodule and discrimination submodule, the discrimination result of the 16KHz audio waveform is obtained. After processing by the second layer sampling submodule and discrimination submodule, the discrimination result of the 8KHz audio waveform is obtained. And so on. After processing by the third, fourth, fifth and sixth layer sampling submodules and discrimination submodules, the discrimination results of the 4KHz audio waveform, 2KHz audio waveform, 1KHz audio waveform and 500Hz audio waveform are obtained respectively.

[0143] This embodiment uses a preset frequency stepping method to continuously optimize the training of content features and speaker features during the training process of the above model. Based on the generation of low-frequency audio waveforms, it gradually outputs high-frequency audio waveforms, effectively preserving the information of audio waveforms at each frequency. It learns both the large-scale features and the fine features of audio waveforms, improving the accuracy of audio waveform generation. Furthermore, it distinguishes each audio waveform as the training basis for the speech conversion model, thereby improving the accuracy of the speech conversion model.

[0144] Optionally, the feature extraction module 51 mentioned above includes:

[0145] The waveform feature extraction submodule is used to input the reference speech waveform into the speech encoder to obtain the mean and variance of the reference speech waveform.

[0146] The waveform distribution determination submodule is used to determine the distribution of the reference speech waveform based on the mean and variance of the reference speech waveform.

[0147] The waveform distribution sampling submodule is used to sample the distribution of the reference speech waveform and determine the sampling result as speaker features.

[0148] Optionally, the waveform generation module 52 includes:

[0149] The generator structure determines the submodule, since the generator includes an upsampling module, a residual connection module, and a decoder;

[0150] The first waveform processing submodule is used to input content features and speaker features into the generator, perform the first upsampling of the content features according to the upsampling module and the first preset frequency to obtain the first sampling result, input the first sampling result and speaker features into the residual connection module to obtain the first fusion feature, input the first fusion feature into the decoder to obtain the first audio waveform corresponding to the first preset frequency;

[0151] The second waveform processing submodule is used to input the content features, speaker features and the first audio waveform into the generator, perform a second upsampling on the content features and the first audio waveform according to the upsampling module and the second preset frequency to obtain the second sampling result, input the second sampling result and speaker features into the residual connection module to obtain the second fusion feature, input the second fusion feature into the decoder to obtain the second audio waveform corresponding to the second preset frequency;

[0152] The third waveform processing submodule is used to input the content features, speaker features, and the i-th audio waveform into the generator, and perform the (i+1)-th upsampling on the content features and the i-th audio waveform according to the upsampling module and the i-th preset frequency to obtain the (i+1)-th sampling result. The (i+1)-th sampling result and the speaker features are input into the residual connection module to obtain the (i+1)-th fused feature. The (i+1)-th fused feature is input into the decoder to obtain the (i+1)-th audio waveform corresponding to the (i+1)-th preset frequency, where i = 1, 2, ..., N. As the number of sampling increases, N audio waveforms with successively increasing preset frequencies are obtained.

[0153] Optionally, the waveform processing module 53 includes:

[0154] The waveform downsampling submodule is used to acquire the real waveform of the sample speech waveform, and downsample the real waveform according to each preset frequency to obtain the real sampled waveform corresponding to each preset frequency.

[0155] The waveform discrimination submodule is used to input the real sampled waveforms and audio waveforms corresponding to each preset frequency into the trained discriminator to obtain the discrimination results corresponding to each preset frequency.

[0156] Optionally, the above model training module 54 includes:

[0157] The fundamental frequency extraction submodule is used to input the sample speech waveform into the trained fundamental frequency extraction model to obtain the sample fundamental frequency;

[0158] The fundamental frequency loss calculation submodule is used to calculate N fundamental frequency losses corresponding to N preset frequencies based on N generated fundamental frequencies, sample fundamental frequencies, and preset fundamental frequency loss functions.

[0159] The generation loss calculation submodule is used to calculate N generation losses corresponding to N preset frequencies based on the fundamental frequency loss corresponding to N preset frequencies, N discrimination results, and preset generation loss functions.

[0160] Optionally, the aforementioned fundamental frequency loss calculation submodule includes:

[0161] The first loss calculation unit is used to calculate the N first losses between the sample fundamental frequency and the N generated fundamental frequencies respectively;

[0162] The adjacent fundamental frequency determination unit is used to determine the generation order of each generated fundamental frequency according to the preset frequency corresponding to each generated fundamental frequency, and to determine the adjacent fundamental frequencies corresponding to each generated fundamental frequency according to the generation order.

[0163] The second loss calculation unit is used to calculate the N second losses between the N generated fundamental frequencies and their corresponding adjacent fundamental frequencies;

[0164] The fundamental frequency loss calculation unit is used to substitute each first loss and each second loss into a preset fundamental frequency loss function to calculate N fundamental frequency losses corresponding to N preset frequencies.

[0165] Optionally, the above model training module 54 includes:

[0166] The loss weight mapping submodule is used to map N preset frequencies to corresponding N loss weights according to a preset weight mapping function.

[0167] The model training submodule is used to multiply N generation losses with their corresponding N loss weights to obtain N multiplication results. The N multiplication results are then added together to obtain the model loss. Based on the model loss, the speech conversion model is trained until the model loss converges, resulting in a trained speech conversion model.

[0168] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0169] Figure 7 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 7 As shown, the computer device of this embodiment includes: at least one processor ( Figure 7Only one is shown in the diagram), a memory, and a computer program stored in the memory and capable of running on at least one processor, which, when executed by the processor, implements the steps in any of the above-described embodiments of the speech model training methods.

[0170] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 7 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0171] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0172] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of a computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0173] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0174] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0175] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0176] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0177] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0178] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0179] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A speech model training method, characterized in that, The voice model training method comprises: obtaining an initial voice conversion model, the voice conversion model comprising a content encoder, a voice encoder and a generator, inputting a sample voice waveform into the content encoder to obtain content features, inputting a reference voice waveform of a reference speaker into the voice encoder to obtain speaker features; inputting the content features and the speaker features into the generator, performing N times of upsampling processing and residual connection processing by using the generator to obtain N audio waveforms with sequentially increased preset frequencies, wherein N is a positive integer; inputting the N audio waveforms into a trained discriminator respectively to obtain N discrimination results, and inputting the N audio waveforms into a trained fundamental frequency extraction model respectively to obtain N generated fundamental frequencies; based on the N discrimination results and the N generated fundamental frequencies, N generation losses respectively corresponding to the N preset frequencies are calculated, and the voice conversion model is trained based on the N preset frequencies and the N generation losses to obtain a trained voice conversion model. 2.The voice model training method of claim 1, wherein, The inputting of the content features and the speaker features into the generator, and the performing of N times of upsampling processing and residual connection processing by using the generator to obtain N audio waveforms with sequentially increased preset frequencies comprises: The generator comprises an upsampling module, a residual connection module and a decoder; the content features are subjected to first-time upsampling based on the upsampling module and a first preset frequency to obtain first sampling results, the first sampling results and the speaker features are inputted into the residual connection module to obtain first fusion features, and the first fusion features are inputted into the decoder to obtain a first audio waveform corresponding to the first preset frequency; the content features and the first audio waveform are subjected to second-time upsampling based on the upsampling module and a second preset frequency to obtain second sampling results, the second sampling results and the speaker features are inputted into the residual connection module to obtain second fusion features, and the second fusion features are inputted into the decoder to obtain a second audio waveform corresponding to the second preset frequency; by analogy, the content features, the speaker features and an i-th audio waveform are inputted into the generator, the content features and the i-th audio waveform are subjected to (i+1)th-time upsampling based on the upsampling module and an i-th preset frequency to obtain (i+1)th sampling results, the (i+1)th sampling results and the speaker features are inputted into the residual connection module to obtain (i+1)th fusion features, and the (i+1)th fusion features are inputted into the decoder to obtain an (i+1)th audio waveform corresponding to the (i+1)th preset frequency, wherein i=1, 2, …, N, and N audio waveforms with sequentially increased preset frequencies are obtained with the increase of the sampling times. 3.The voice model training method of claim 1, wherein, The inputting of the N audio waveforms into a trained discriminator respectively to obtain N discrimination results comprises: obtaining a real waveform of the sample speech waveform, down-sampling the real waveform according to each of the preset frequencies to obtain a real sample waveform corresponding to each of the preset frequencies; inputting the real sample waveform corresponding to each of the preset frequencies and the audio waveform into the trained discriminator to obtain a discrimination result corresponding to each of the preset frequencies. 4.The voice model training method of claim 1, wherein, The N generation losses corresponding to the N preset frequencies are calculated according to the N discrimination results and the N generated fundamental frequencies, including: inputting the sample speech waveform into the trained fundamental frequency extraction model to obtain a sample fundamental frequency; calculating N fundamental frequency losses corresponding to the N preset frequencies according to the N generated fundamental frequencies, the sample fundamental frequency and a preset fundamental frequency loss function; calculating N generation losses corresponding to the N preset frequencies according to the fundamental frequency losses corresponding to the N preset frequencies, the N discrimination results and a preset generation loss function. 5.The voice model training method of claim 4, wherein, The N fundamental frequency losses corresponding to the N preset frequencies are calculated according to the N generated fundamental frequencies, the sample fundamental frequency and a preset fundamental frequency loss function, including: calculating N first losses between the sample fundamental frequency and the N generated fundamental frequencies respectively; determining a generation order of each of the generated fundamental frequencies according to the preset frequency corresponding to each of the generated fundamental frequencies, and determining a neighboring fundamental frequency corresponding to each of the generated fundamental frequencies according to the generation order; calculating N second losses between the N generated fundamental frequencies and the corresponding neighboring fundamental frequencies respectively; substituting each first loss and each second loss into the preset fundamental frequency loss function to calculate N fundamental frequency losses corresponding to the N preset frequencies respectively. 6.The voice model training method of claim 4, wherein, The trained speech conversion model is obtained by training the speech conversion model according to the N preset frequencies and the N generation losses, including: mapping the N preset frequencies to corresponding N loss weights according to a preset weight mapping function; multiplying the N generation losses and the corresponding N loss weights to obtain N multiplication results, adding the N multiplication results to obtain a model loss, and training the speech conversion model according to the model loss until the model loss converges to obtain the trained speech conversion model.

7. A voice conversion method characterized by, The speech conversion method includes: obtaining the trained speech conversion model according to any one of claims 1-6, the trained speech conversion model including a trained content encoder, a trained speech encoder and a trained generator; obtaining a to-be-processed speech waveform and a target reference speech waveform, inputting the to-be-processed speech waveform into the trained content encoder to obtain a target content feature, and inputting the target reference speech waveform into the trained speech encoder to obtain a target speaker feature; inputting the target content feature and the target speaker feature into the trained generator to obtain an audio waveform with N preset frequencies sequentially increasing; determining the audio waveform corresponding to the highest preset frequency as a target audio waveform.

8. A speech model training apparatus, characterized by comprising: The speech model training device includes: The feature extraction module is configured to obtain an initial speech conversion model, the speech conversion model comprising a content encoder, a speaker encoder and a generator, input a sample speech waveform into the content encoder to obtain content features, and input a reference speech waveform of a reference speaker into the speaker encoder to obtain speaker features; The waveform generation module is configured to input the content features and the speaker features into the generator, perform N times of upsampling processing and residual connection processing by using the generator, and obtain N audio waveforms with a preset frequency sequentially increasing, wherein N is a positive integer. The waveform processing module is configured to input the N audio waveforms into a trained discriminator respectively to obtain N discrimination results, and input the N audio waveforms into a trained fundamental frequency extraction model respectively to obtain N generated fundamental frequencies. The model training module is configured to calculate N generated losses corresponding to the N preset frequencies respectively according to the N discrimination results and the N generated fundamental frequencies, train the speech conversion model according to the N preset frequencies and the N generated losses to obtain a trained speech conversion model.

9. A computer device, comprising: The computer device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the speech model training method according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the speech model training method according to any one of claims 1 to 6.

Images