Updating method and application method of sound conversion model

Abstract

The present application discloses a method for updating a sound conversion model and an application method, which relates to the field of computer technology. The solution does not require a lot of time and cost to collect sample audio of a new timbre. The method includes: step A: calling the sound conversion model to perform feature processing on the first audio signal and the second audio signal of the target timbre to obtain a third audio signal; step B: calling the sound conversion model to perform feature processing on the first audio signal and the third audio signal to obtain a reconstructed first audio signal; step C: determining whether the proximity parameter between the first audio signal and the reconstructed first audio signal meets the termination condition; step D: if the termination condition is not met, the parameters of the sound conversion model are updated based on the proximity parameter, and steps A to C are repeated after the update until it is determined that the termination condition is met; if the termination condition is met, the last updated sound conversion model is obtained.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an updating method and an application method of a sound conversion model. Background Art

[0002] At present, voice conversion models trained based on voice conversion (VC) algorithms can achieve the conversion of timbre without changing the semantic content. For example, voice conversion models trained based on VC algorithms can convert the voice of cartoon character A into the audio of cartoon character B, but the rhythmic information (such as pauses, stress, speaking speed, etc.) in the original voice cannot be flexibly adjusted.

[0003] Since it is impossible to collect audio of all possible timbres during the training of the sound conversion model, it is difficult for the sound conversion model to realize the conversion function of any timbre. In addition, the training audio data requires that the original timbre and the target timbre content are consistent or there are restrictions on the semantic scope of the training data. The amount of training data is large and the cost is high.

[0004] In the prior art, when a conversion function of a new target timbre needs to be added, a sample audio of the new timbre needs to be collected to re-update the sound conversion model so that the updated sound conversion model supports the conversion function of the new timbre. However, in the prior art, when updating the sound conversion model, a lot of time and cost are required to collect more sample audios of the new timbre. Summary of the invention

[0005] The present application provides a method for updating a sound conversion model and an application method thereof. The solution does not require a large amount of time and cost to collect sample audio of a new timbre, and can perform timbre conversion on an input audio signal to generate audio with a target timbre.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a method for updating a sound conversion model, comprising: step A: calling the sound conversion model to perform feature processing on a first audio signal and a second audio signal to obtain a third audio signal; the second audio signal is an audio signal of a target timbre; step B: calling the sound conversion model to perform feature processing on the first audio signal and the third audio signal to obtain a reconstructed first audio signal; step C: determining whether a proximity parameter between the first audio signal and the reconstructed first audio signal satisfies a termination condition; step D: if it is determined that the proximity parameter does not satisfy the termination condition, updating the parameters of the sound conversion model based on the proximity parameter, and repeating steps A to C after the update until it is determined that the proximity parameter satisfies the termination condition; if it is determined that the proximity parameter satisfies the termination condition, obtaining the last updated sound conversion model.

[0008] In the technical solution provided by the present application, in order to realize the function of converting the input timbre into the target timbre, the sound conversion model can be called to perform feature processing on the first audio signal and the second audio signal corresponding to the target timbre to obtain a third audio signal, and then the first audio signal can be reconstructed based on the obtained third audio signal and the first audio signal, and then the parameters of the sound conversion model can be updated based on the proximity parameter between the first audio signal and the reconstructed first audio signal, and the above steps are repeatedly performed until the proximity parameter satisfies the termination condition, and when it is determined that the proximity parameter satisfies the termination condition, the last updated sound conversion model is obtained. When the proximity parameter between the first audio signal and the reconstructed first audio signal satisfies the termination condition, it indicates that the current sound conversion model can well realize the reconstruction of the first audio signal, that is, the current sound conversion model has a good ability to capture various features (including the target timbre) in the sound conversion process, and can realize the conversion function of the target timbre. It can be seen that in the technical solution provided by the present application, by reconstructing the first audio signal and updating the sound conversion model based on the proximity parameter between the first audio signal and the reconstructed first audio signal, it is possible to achieve timbre conversion of the input audio signal and generate audio with a target timbre. At the same time, it is possible to achieve a conversion function of converting the timbre of the input audio signal into a newly added target timbre. In this way, there is no need to spend a lot of time and cost to collect more sample audios of new timbres.

[0009] In the embodiment of the present application, it is possible to achieve targeted adjustments to the rhythmic characteristics of the output sound according to user needs while converting the timbre, thereby further enhancing the richness and expressiveness of the sound in the live broadcast scene.

[0010] Optionally, in a possible design, the sound conversion model is composed of a preset sound recognition model, a first encoder, a second encoder and a decoder, and the above-mentioned “calling the sound conversion model to perform feature processing on the first audio signal and the second audio signal to obtain a third audio signal” may include: extracting a first semantic content of the first audio signal through the preset sound recognition model, and extracting at least one first prosodic feature vector of the first audio signal through the first encoder, and extracting a second timbre feature vector of the second audio signal through the second encoder; performing audio synthesis processing on the first semantic content, at least one first prosodic feature vector and the second timbre feature vector through the decoder to obtain the third audio signal;

[0011] Calling a sound conversion model to perform feature processing on the first audio signal and the third audio signal to obtain a reconstructed first audio signal may include: extracting the second semantic content of the third audio signal through a preset sound recognition model, extracting at least one second prosodic feature vector of the third audio signal through a first encoder, and extracting the first timbre feature vector of the first audio signal through a second encoder; performing audio synthesis processing on the second semantic content, at least one second prosodic feature vector and the first timbre feature vector through a decoder to obtain a reconstructed first audio signal.

[0012] Optionally, in another possible design, the proximity parameter includes at least a first loss value between the first audio signal and the reconstructed first audio signal, and the above-mentioned "determining whether the proximity parameter between the first audio signal and the reconstructed first audio signal satisfies the termination condition" may include: determining whether the first loss value reaches a first preset condition; if the first loss value reaches the first preset condition, determining that the proximity parameter satisfies the termination condition.

[0013] Optionally, in another possible design, the proximity parameter further includes a second loss value of the first semantic content and the second semantic content, a third loss value of at least one first prosodic feature vector and at least one second prosodic feature vector, and a fourth loss value of the first timbre feature vector and the third timbre feature vector; the third timbre feature vector is obtained by calling the second encoder to perform feature extraction on the reconstructed first audio signal;

[0014] The method provided in this application may also include:

[0015] Determining whether the first loss value reaches a first preset condition, determining whether the second loss value reaches a second preset condition, determining whether the third loss value reaches a third preset condition, and determining whether the fourth loss value reaches a fourth preset condition;

[0016] If the first loss value reaches the first preset condition, the second loss value reaches the second preset condition, the third loss value reaches the third preset condition, and the fourth loss value reaches the fourth preset condition, it is determined that the proximity parameter meets the termination condition.

[0017] Optionally, in another possible design, the second encoder is a variational autoencoder, and the sound conversion model updating method provided in the present application may also include: Step E: calling the variational autoencoder to perform feature processing on the first audio signal to obtain a fourth timbre feature vector in the high-dimensional Gaussian space, and calculating the fifth loss value of the fourth timbre feature vector in the high-dimensional Gaussian space and the standard high-dimensional Gaussian probability distribution through KL divergence; Step F: determining whether the fifth loss value satisfies the fifth preset condition; Step G: when it is determined that the fifth loss value does not meet the fifth preset condition, updating the parameters of the variational autoencoder based on the fifth loss value, and repeating steps E to F after the update until the fifth loss value meets the fifth preset condition; when it is determined that the fifth loss value meets the fifth preset condition, obtaining the last updated variational autoencoder, and updating the sound conversion model based on the updated variational autoencoder.

[0018] Optionally, in another possible design, at least one first prosodic feature vector is used to characterize at least one of the fundamental frequency, volume, speaking rate and pauses of the first audio signal, and at least one second prosodic feature vector is used to characterize at least one of the fundamental frequency, volume, speaking rate and pauses of the third audio signal.

[0019] Optionally, in another possible design, updating the parameters of the sound conversion model based on the proximity parameter may include: updating the parameters of the sound conversion model based on the proximity parameter in combination with the gradient descent rule, the back propagation algorithm and the chain rule.

[0020] In a second aspect, the present application provides an application method of a sound conversion model, comprising: obtaining an input audio signal; calling a preset sound recognition model in the sound conversion model that was last updated to extract a third semantic content from the input audio signal; calling a first encoder in the sound conversion model that was last updated to extract at least one fourth prosodic feature vector from the input audio signal, and obtaining at least one adjusted fourth prosodic feature vector based on adjustment information; obtaining a reference audio signal of a target timbre, calling a second encoder in the sound conversion model that was last updated to extract a fifth timbre feature vector of the reference audio signal; calling a decoder in the sound conversion model that was last updated to perform audio synthesis processing on the third semantic content, the adjusted at least one fourth prosodic feature vector, and the fifth timbre feature vector to obtain a target audio signal.

[0021] In a third aspect, the present application provides a device for updating a sound conversion model, comprising: a processing module, a determination module, and an updating module;

[0022] Specifically, the processing module is used to execute step A: calling the sound conversion model to perform feature processing on the first audio signal and the second audio signal to obtain a third audio signal; the second audio signal is an audio signal of the target timbre;

[0023] The processing module is further used to execute step B: calling the sound conversion model to perform feature processing on the first audio signal and the third audio signal to obtain a reconstructed first audio signal;

[0024] A determination module, configured to execute step C: determining whether a proximity parameter between the first audio signal and the reconstructed first audio signal obtained by the processing module satisfies a termination condition;

[0025] An updating module is used to execute step D: when the determining module determines that the proximity parameter does not satisfy the termination condition, the parameters of the sound conversion model are updated based on the proximity parameter, and steps A to C are repeated after the update until the determining module determines that the proximity parameter satisfies the termination condition; when the determining module determines that the proximity parameter satisfies the termination condition, the last updated sound conversion model is obtained.

[0026] Optionally, in a possible design, the sound conversion model is composed of a preset sound recognition model, a first encoder, a second encoder and a decoder, and the processing module is specifically used for:

[0027] Extracting a first semantic content of a first audio signal through a preset sound recognition model, extracting at least one first prosodic feature vector of the first audio signal through a first encoder, and extracting a second timbre feature vector of a second audio signal through a second encoder; performing audio synthesis processing on the first semantic content, at least one first prosodic feature vector, and the second timbre feature vector through a decoder to obtain a third audio signal; calling a sound conversion model to perform feature processing on the first audio signal and the third audio signal to obtain a reconstructed first audio signal may include: extracting a second semantic content of the third audio signal through a preset sound recognition model, extracting at least one second prosodic feature vector of the third audio signal through the first encoder, and extracting a first timbre feature vector of the first audio signal through the second encoder; performing audio synthesis processing on the second semantic content, at least one second prosodic feature vector, and the first timbre feature vector through a decoder to obtain a reconstructed first audio signal.

[0028] Optionally, in another possible design, the proximity parameter includes at least a first loss value between the first audio signal and the reconstructed first audio signal, and the determination module is specifically configured to:

[0029] Determine whether the first loss value reaches a first preset condition; if the first loss value reaches the first preset condition, determine whether the proximity parameter satisfies a termination condition.

[0030] Optionally, in another possible design, the proximity parameter further includes a second loss value of the first semantic content and the second semantic content, a third loss value of at least one first prosodic feature vector and at least one second prosodic feature vector, and a fourth loss value of the first timbre feature vector and the third timbre feature vector; the third timbre feature vector is obtained by calling the second encoder to perform feature extraction on the reconstructed first audio signal;

[0031] The determination module is specifically used to: determine whether the first loss value reaches the first preset condition, determine whether the second loss value reaches the second preset condition, determine whether the third loss value reaches the third preset condition, and determine whether the fourth loss value reaches the fourth preset condition;

[0032] If the first loss value reaches the first preset condition, the second loss value reaches the second preset condition, the third loss value reaches the third preset condition, and the fourth loss value reaches the fourth preset condition, it is determined that the proximity parameter meets the termination condition.

[0033] Optionally, in another possible design, the second encoder is a variational autoencoder, and the processing module is also used to execute step E: call the variational autoencoder to perform feature processing on the first audio signal to obtain a fourth timbre feature vector in the high-dimensional Gaussian space, and calculate the fifth loss value of the fourth timbre feature vector in the high-dimensional Gaussian space and the standard high-dimensional Gaussian probability distribution through KL divergence; the determination module is also used to execute step F: determine whether the fifth loss value satisfies the fifth preset condition; the update module is also used to execute step G: when the determination module determines that the fifth loss value does not meet the fifth preset condition, update the parameters of the variational autoencoder based on the fifth loss value, and repeat steps E to F after the update until the determination module determines that the fifth loss value meets the fifth preset condition; when it is determined that the fifth loss value meets the fifth preset condition, obtain the last updated variational autoencoder, and update the sound conversion model based on the updated variational autoencoder.

[0034] Optionally, in another possible design, at least one first prosodic feature vector is used to characterize at least one of the fundamental frequency, volume, speaking rate and pauses of the first audio signal, and at least one second prosodic feature vector is used to characterize at least one of the fundamental frequency, volume, speaking rate and pauses of the third audio signal.

[0035] Optionally, in another possible design, the update module is specifically used to update the parameters of the sound conversion model based on the proximity parameter in combination with the gradient descent rule, the back propagation algorithm and the chain rule.

[0036] In a fourth aspect, the present application provides an application device of a sound conversion model, comprising: an acquisition module, used to acquire an input audio signal; a processing module, used to call a preset sound recognition model in the sound conversion model updated last time, and extract a third semantic content from the input audio signal; the processing module is also used to call the first encoder in the sound conversion model updated last time, extract at least one fourth prosodic feature vector from the input audio signal, and obtain at least one adjusted fourth prosodic feature vector according to adjustment information; the processing module is also used to acquire a reference audio signal of a target timbre through the acquisition module, and call the second encoder in the sound conversion model updated last time to extract a fifth timbre feature vector of the reference audio signal; the processing module is also used to call the decoder in the sound conversion model updated last time to perform audio synthesis processing on the third semantic content, the adjusted at least one fourth prosodic feature vector, and the fifth timbre feature vector to obtain a target audio signal.

[0037] In a fifth aspect, the present application provides an electronic device comprising a memory, a processor, a bus and a communication interface; the memory is used to store computer execution instructions, and the processor is connected to the memory through a bus; when the electronic device is running, the processor executes the computer execution instructions stored in the memory, so that the electronic device executes the updating method of the sound conversion model provided in the first aspect above or executes the application method of the sound conversion model provided in the second aspect above.

[0038] Optionally, the electronic device may be a physical machine for updating or applying the sound conversion model, or may be a part of a device in a physical machine, for example, a chip system in a physical machine. The chip system is used to support the electronic device to implement the functions involved in the first aspect or the second aspect, for example, receiving, sending or processing the data and / or information involved in the method for updating the sound conversion model. The chip system includes a chip and may also include other discrete devices or circuit structures.

[0039] In a sixth aspect, the present application provides a computer-readable storage medium, in which instructions are stored. When a computer executes the instructions, the computer executes a method for updating a sound conversion model as provided in the first aspect or an application method for a sound conversion model as provided in the second aspect.

[0040] In the seventh aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed on a computer, the computer executes the updating method of the sound conversion model provided in the first aspect or executes the application method of the sound conversion model provided in the second aspect.

[0041] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium, wherein the computer-readable storage medium may be packaged together with the processor of the electronic device, or may be packaged separately from the processor of the electronic device, which is not limited in this application.

[0042] The descriptions of the second to seventh aspects of the present application can refer to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second to seventh aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.

[0043] In this application, the names of the above-mentioned devices or functional modules do not constitute limitations. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they are within the scope of the claims of this application and their equivalent technologies.

[0044] These and other aspects of the present application will become more apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a method for updating a sound conversion model provided in an embodiment of the present application;

[0046] Figure 2 A simplified schematic diagram of a process for obtaining a reconstructed first audio signal provided in an embodiment of the present application;

[0047] Figure 3 A flowchart of an application method of a sound conversion model provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of the structure of a device for updating a sound conversion model provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of the structure of an application device of a sound conversion model provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The following is a detailed description of the updating method and application method of the sound conversion model provided in the embodiment of the present application in conjunction with the accompanying drawings.

[0052] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0053] The terms "first" and "second" and the like in the specification and drawings of this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.

[0054] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0055] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0056] In the description of the present application, unless otherwise specified, “plurality” means two or more.

[0057] In addition, the acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0058] At present, the sound conversion model trained based on the VC algorithm can achieve the conversion of timbre without changing the semantic content. For example, the sound conversion model trained based on the VC algorithm can convert the voice of cartoon character A into the voice of cartoon character B, but the rhythmic information (such as pauses, stress, speaking speed, etc.) in the original voice cannot be flexibly adjusted. Since it is impossible to collect audio of all possible timbres during the training of the sound conversion model, it is difficult for the sound conversion model to achieve the conversion function of any timbre, and the training audio data requires the original timbre and the target timbre content to be consistent or the semantic scope of the training data is limited, the amount of training data is large, and the cost is high.

[0059] In the prior art, when a conversion function of a new target timbre needs to be added, a sample audio of the new timbre needs to be collected to re-update the sound conversion model so that the updated sound conversion model supports the conversion function of the new timbre. However, in the prior art, when updating the sound conversion model, a lot of time and cost are required to collect more sample audios of the new timbre.

[0060] In response to the problems existing in the above-mentioned prior art, an embodiment of the present application provides a method for updating a sound conversion model. The method reconstructs the first audio signal through the first audio signal and a second audio signal corresponding to the target timbre, and updates the sound conversion model based on a proximity parameter between the first audio signal and the reconstructed first audio signal. This method can realize the timbre conversion of the input audio signal and generate audio with the target timbre. At the same time, it can realize the conversion function of converting the timbre of the input audio signal into a newly added target timbre, without spending a lot of time and cost to collect more sample audios of the new timbre.

[0061] In the embodiment of the present application, it is possible to achieve targeted adjustments to the rhythmic characteristics of the output sound according to user needs while converting the timbre, thereby further enhancing the richness and expressiveness of the sound in the live broadcast scene.

[0062] The execution subject of the sound conversion model updating method provided in the embodiment of the present application can be a sound conversion model updating device. Exemplarily, the sound conversion model updating device can be a physical machine (such as a server) or a virtual machine (VM) deployed on a physical machine. The sound conversion model updating device can update the sound conversion model by performing feature processing on the audio signal.

[0063] The updating method of the sound conversion model provided by the present application is described below with reference to the accompanying drawings.

[0064] Reference Figure 1 The method for updating the sound conversion model provided in the embodiment of the present application includes S101-S105:

[0065] S101. Call a sound conversion model to perform feature processing on a first audio signal and a second audio signal to obtain a third audio signal.

[0066] The sound conversion model may be a network model with streaming processing capability, which is used to realize the conversion function of the audio signal. For example, the sound conversion model may be a convolutional neural network or a linear mapping network, etc., which is not limited in the embodiments of the present application.

[0067] The first audio signal may be any audio sample in the sample audio library, for example, a 3-minute audio sample of cartoon character A. The second audio signal is an audio signal of a target timbre, which may also be any audio sample in the sample audio library. For example, when the audio conversion function of cartoon character B is to be added to the sound conversion model, the second audio signal may be an audio of the target timbre corresponding to cartoon character B, for example, a 3-minute audio sample of cartoon character B. The third audio signal obtained in the embodiment of the present application has the same semantic content as the first audio signal and the same timbre as the first audio signal.

[0068] It can be understood that the processing process for each pair of audio signals in the embodiment of the present application is the same. In the embodiment of the present application, the processing process of a pair of audio signals (the first audio signal and the second audio signal in the embodiment of the present application are a pair of audio signals) will be used as an example to explain.

[0069] S102: Calling a sound conversion model to perform feature processing on the first audio signal and the third audio signal to obtain a reconstructed first audio signal.

[0070] Optionally, in a possible implementation, the sound conversion model is composed of a preset sound recognition model, a first encoder, a second encoder and a decoder. The embodiment of the present application can obtain the third audio signal in the following manner: extract the first semantic content of the first audio signal through the preset sound recognition model, and extract at least one first prosodic feature vector of the first audio signal through the first encoder, and extract the second timbre feature vector of the second audio signal through the second encoder; perform audio synthesis processing on the first semantic content, at least one first prosodic feature vector and the second timbre feature vector through the decoder to obtain the third audio signal.

[0071] In addition, the embodiment of the present application can obtain the reconstructed first audio signal in the following manner: extract the second semantic content of the third audio signal through a preset sound recognition model, extract at least one second prosodic feature vector of the third audio signal through a first encoder, and extract the first timbre feature vector of the first audio signal through a second encoder; perform audio synthesis processing on the second semantic content, at least one second prosodic feature vector and the first timbre feature vector through a decoder to obtain a reconstructed first audio signal.

[0072] Optionally, at least one first prosodic feature vector is used to characterize at least one of the fundamental frequency, volume, speech rate, and pauses of the first audio signal, and at least one second prosodic feature vector is used to characterize at least one of the fundamental frequency, volume, speech rate, and pauses of the third audio signal. Of course, in practical applications, at least one first prosodic feature vector and at least one second prosodic feature vector may also include other prosodic feature vectors, which are not limited in the embodiments of the present application.

[0073] The preset sound recognition model may be a pre-trained model for extracting semantic content in an audio signal that is unrelated to timbre. Exemplarily, the preset sound recognition model may be a pre-trained automatic speech recognition (ASR) model. The first encoder may be an encoder for extracting prosodic features that are unrelated to timbre, the first encoder may include one or more encoders, an encoder for extracting a prosodic feature, and an encoder may be composed of one or more deep learning network models. For example, the first encoder may include four encoders, each for extracting the fundamental frequency, volume, speech rate, and pause of an audio signal. The second encoder may be an encoder for extracting timbre features, the second encoder may include an encoder, and an encoder may be composed of one or more deep learning network models. The decoder may be a decoder for performing sound synthesis, the decoder may include a decoder, and a decoder may be composed of one or more deep learning network models.

[0074] For example, refer to Figure 2 , the embodiment of the present application provides a simplified flow chart of obtaining a reconstructed first audio signal. Figure 2 As shown, after obtaining the first audio signal S from the sample audio library A After that, S can be extracted through the preset sound recognition model ASR A The first semantic content L A , and through the first encoder M A Extract S A The N first prosodic feature vectors P A1-AN In addition, the second audio signal R of the target timbre is collected B After that, the second encoder M B Extract R B The second timbre feature vector T B , which can then be decoded by M C For L A , P A1-AN and T B Perform audio synthesis processing to obtain a third audio signal S B Afterwards, S can be extracted through ASR BThe second semantic content L B , and through M A Extract S B The N second prosodic feature vectors P B1-BN , and through M B Extract S A The first timbre feature vector T A , which can then be decoded by M C For L B , P B1-BN and T A Perform audio synthesis processing to obtain a reconstructed first audio signal S A′ .

[0075] In a specific embodiment, the embodiment of the present application can use a temporal-spatial encoder to compress sample audio signals of unequal length into a latent space to obtain a timbre feature vector of fixed length, thereby achieving accurate expression of timbre information.

[0076] S103: Determine whether a proximity parameter between the first audio signal and the reconstructed first audio signal satisfies a termination condition.

[0077] If it is determined that the proximity parameter does not satisfy the termination condition, step S104 is executed; if it is determined that the proximity parameter satisfies the termination condition, step S105 is executed.

[0078] Optionally, the proximity parameter includes at least a first loss value between the first audio signal and the reconstructed first audio signal, and determining whether the proximity parameter between the first audio signal and the reconstructed first audio signal satisfies the termination condition may include: determining whether the first loss value reaches a first preset condition; if the first loss value reaches the first preset condition, determining that the proximity parameter satisfies the termination condition.

[0079] The first preset condition may be a condition determined in advance by humans, such as the first loss value being less than a preset value or the number of iterations reaching a preset value. Exemplarily, the first loss value of the first audio signal and the reconstructed first audio signal may be calculated based on a mean square error (MSE) function.

[0080] The semantic content, rhythmic features and timbre features of an audio signal are the key features that constitute an audio signal. In the process of training the sound conversion model, if it can well reconstruct the original first audio signal, it means that the trained sound conversion model has the ability to accurately extract the semantic content, rhythmic features and timbre features. Therefore, in the embodiment of the present application, it can be judged whether the termination condition of the iteration is reached by the first loss value based on the first audio signal and the reconstructed first audio signal.

[0081] S104: Update the parameters of the sound conversion model based on the proximity parameters.

[0082] After step S104, the process returns to step S101.

[0083] Optionally, in the embodiment of the present application, the parameters of the sound conversion model can be updated based on the proximity parameter in combination with the gradient descent rule, the back propagation algorithm and the chain rule.

[0084] For example, based on S A′ and S A Taking the first loss value of the sound conversion model as an example, we can first calculate S based on the MSE function. A′ and S A The first loss value of (the first loss value can be a parameterless loss function), and then the first loss value can be calculated relative to S A′ The partial derivative of S A′ The related function, S A′ It is related to the parameters in the sound conversion model. Then, the back propagation algorithm and the chain rule can be used to update all the parameters in the first encoder, the second encoder, and the decoder in the sound conversion model. Specifically, the gradient A relative to each parameter in the sound conversion model can be calculated, and each parameter in the sound conversion model can be updated according to the gradient of each parameter itself using the gradient descent method (each parameter minus the gradient of the parameter itself = the updated parameter value). Since the gradient descent method is used, S A′ and S A The first loss value will gradually decrease. In the embodiment of the present application, the above process can be continuously iterated until the first loss value reaches the first preset condition.

[0085] Optionally, the proximity parameter may also include a second loss value of the first semantic content and the second semantic content, a third loss value of at least one first prosodic feature vector and at least one second prosodic feature vector, and a fourth loss value of the first timbre feature vector and the third timbre feature vector; the third timbre feature vector is obtained by calling the second encoder to perform feature extraction on the reconstructed first audio signal; the method for updating the sound conversion model provided in the embodiment of the present application may also include: determining whether the first loss value meets the first preset condition, determining whether the second loss value meets the second preset condition, determining whether the third loss value meets the third preset condition, and determining whether the fourth loss value meets the fourth preset condition; if the first loss value meets the first preset condition, the second loss value meets the second preset condition, the third loss value meets the third preset condition, and the fourth loss value meets the fourth preset condition, then it is determined that the proximity parameter meets the termination condition.

[0086] In a possible implementation, the loss value involved in the embodiments of the present application may be a distance function, for example, L1 distance.

[0087] Among them, the third timbre feature vector is obtained by calling the second encoder to extract features from the reconstructed first audio signal; the second loss value, the third loss value and the fourth loss value can all be used to characterize the size of the proximity parameter. The second loss value is specifically used to characterize the gap between the first semantic content and the second semantic content, the third loss value is specifically used to characterize the gap between at least one first rhythmic feature vector and at least one second rhythmic feature vector, and the fourth loss value is specifically used to characterize the gap between the first timbre feature vector and the third timbre feature vector. The second preset condition can be that the second loss value is less than a preset value, the third preset condition can be that the third loss value is less than a preset value, and the fourth preset condition can be that the fourth loss value is less than a preset value.

[0088] It can be understood that the process of updating the parameters of the sound conversion model based on the second loss value, the third loss value and the fourth loss value is the same as the process of updating the parameters of the sound conversion model based on S A′ and S A The process of updating the parameters of the sound conversion model by using the first loss value is similar, and the embodiments of the present application are not repeated here. In addition, since at least one first prosodic feature vector and at least one second prosodic feature vector are multidimensional vectors, the two groups of prosodic feature vectors can be normalized before calculating the third loss value during the calculation process.

[0089] In the embodiment of the present application, the parameters of the sound conversion model are updated based on the second loss value, so that the consistency of the semantic content before and after the sound conversion can be ensured. In addition, the parameters of the sound conversion model are updated based on the third loss value, so that the coherence of the rhythmic features after the sound conversion can be ensured. In addition, the parameters of the sound conversion model are updated based on the fourth loss value, so that the fidelity of the timbre after the sound conversion can be improved.

[0090] Optionally, the embodiment of the present application may further include: extracting at least one fourth prosodic feature vector of the reconstructed first audio signal through a second encoder, and then updating the parameters of the sound conversion model based on the loss values ​​of the at least one first prosodic feature vector and the at least one fourth prosodic feature vector. In this way, the coherence of the prosodic features after the sound conversion can be further ensured.

[0091] Further optionally, the method for updating the sound conversion model of the embodiment of the present application may also include: extracting at least one real first prosodic feature vector of the first audio signal using an audio signal analysis algorithm or a deep learning algorithm, and determining whether the sixth loss value of at least one first prosodic feature vector and at least one real first prosodic feature vector meets a sixth preset condition, and then updating the parameters of the sound conversion model based on the sixth loss value.

[0092] The sixth loss value is specifically used to characterize the difference between at least one first prosodic feature vector and at least one true first prosodic feature vector, and the sixth preset condition may be that the sixth loss value is less than a preset value. Exemplarily, the audio signal analysis algorithm may use a PYIN fundamental frequency extraction algorithm or a logarithmic domain Mel spectrum energy calculation method, which is not limited in the present application embodiment.

[0093] In the prior art, when training a sound conversion model, the training data in the sample audio library is a paired audio signal (i.e., each pair of audio signals includes two sample audios of the same semantic content generated with different timbres). In the updating method of the sound conversion model provided in the embodiment of the present application, the training data in the sample audio library can be a plurality of sample audios of arbitrary semantic content with different timbres. In this way, the difficulty of collecting sample audio can be further reduced, and the efficiency of model training can be improved. In order to further ensure the accuracy of audio conversion using the sound conversion model provided by the present application, the embodiment of the present application can also constrain the authenticity of the intermediate result (i.e., the third audio signal) generated during the training process. Specifically, based on adversarial training, a discriminator can be used to constrain the authenticity of the third audio signal to determine whether the third audio signal meets the characteristics of a real speech signal. The discriminator is used to determine the strength of the authenticity of the third audio signal, and the parameters in the sound conversion model are updated until the third audio signal meets the pre-set conditions.

[0094] Further optionally, the first encoder can be a variational autoencoder (VAE), for example, a vector quantization variational autoencoder (VQ-VAE), which is used to extract different prosodic features in Gaussian space and convert a one-dimensional audio signal in time space into a multi-dimensional prosodic feature vector in time space.

[0095] Further optionally, the second encoder may be a variational autoencoder, and the method for updating the sound conversion model provided in the embodiment of the present application may also include: step E: calling the variational autoencoder to perform feature processing on the first audio signal to obtain a fourth timbre feature vector in a high-dimensional Gaussian space, and calculating the fifth loss value of the fourth timbre feature vector in the high-dimensional Gaussian space and the standard high-dimensional Gaussian probability distribution through the KL divergence (Kullback-Leibler Divergenc); step F: determining whether the fifth loss value satisfies the fifth preset condition; step G: in the case of determining that the fifth loss value does not satisfy the fifth preset condition, updating the parameters of the variational autoencoder based on the fifth loss value, and repeating steps E to F after the update until the fifth loss value satisfies the fifth preset condition; in the case of determining that the fifth loss value satisfies the fifth preset condition, obtaining the last updated variational autoencoder, and updating the sound conversion model based on the updated variational autoencoder. Optionally, the second encoder may also be a spatiotemporal encoder.

[0096] Among them, the variational autoencoder can use the network structure with time series signal processing capability in machine learning to process the input first audio signal to obtain a fixed-size feature vector T corresponding to the first audio signal, and the feature vector T can be regarded as a sampling example in a high-dimensional Gaussian space. The embodiment of the present application can calculate the fifth loss value of the sampling example and the standard high-dimensional Gaussian probability distribution through KL divergence, and then the second encoder in the sound conversion model can be updated by combining the gradient descent rule, the back propagation algorithm and the chain rule.

[0097] The fifth loss value is used to characterize the gap between the feature vector T corresponding to the first audio signal and the standard high-dimensional Gaussian probability distribution. The fifth preset condition may be that the fifth loss value is less than a preset value.

[0098] It is understandable that in practical applications of the embodiments of the present application, calling the variational autoencoder can not only perform feature processing on the first audio signal, but also process other audio signals, such as performing similar feature processing on the second audio signal and the reconstructed first audio signal. The specific processing method can refer to the process of calling the variational autoencoder to process the first audio signal, which will not be repeated here.

[0099] After the variational autoencoder training is completed, the obtained variational autoencoder has the ability to extract timbre features from the audio signal of the target timbre, and the timbre features conform to the Gaussian distribution. The timbre feature vector can be obtained by random sampling of the timbre Gaussian space, thereby generating more audio signals with unknown target timbre. Therefore, through the technical solution provided in the embodiment of the present application, the timbre range that can be supported by the sound conversion model can be increased as much as possible. At the same time, the sound conversion model can be enabled to generate some timbre in the training data that does not exist in the sample training library and quickly generate data of the corresponding timbre through a small number of samples.

[0100] It is understandable that in the embodiment of the present application, the second encoder may be trained separately first, and then the first encoder, the second encoder and the decoder may be trained together. Alternatively, the first encoder, the second encoder and the decoder may be trained together first, and then the second encoder may be trained separately.

[0101] S105: Obtain the sound conversion model updated last time.

[0102] After the last updated sound conversion model is obtained, the sound conversion model can be applied to convert audio signals of other timbres into audio signals of target timbre, that is, the updated sound conversion model supports the conversion function to the new timbre (target timbre).

[0103] Optionally, the prosodic features in the embodiments of the present application may include explicit features and residual features. Among them, explicit features refer to features with clear definitions and calculation methods, such as fundamental frequency, energy, phase, etc.; residual features refer to information that is not clearly defined or calculated, but may interfere with prosodic features. In order to ensure the naturalness and authenticity of the sound after conversion, residual features can be used in the embodiments of the present application to supplement the information that the explicit features cannot express, thereby ensuring the quality of the converted sound. Specifically, for the extraction of explicit feature vectors, the methods that can be used in the embodiments of the present application include but are not limited to audio signal analysis algorithms (such as PYIN fundamental frequency extraction or logarithmic domain Mel spectrum energy calculation) and deep learning algorithms. For the extraction of residual features, the present application can analyze the original speech (first audio signal) signal through the VAE architecture to generate a time series of residual prosodic feature vectors for expressing the residual features of the speech signal at each moment. Further, the residual prosodic feature vector can be constrained to Gaussian space using KL divergence, and the parameters in the first encoder, the second encoder, and the decoder can be adjusted using the gradient descent method and the chain rule according to the KL divergence. In this way, after the sound conversion model is updated, the prosodic feature vector is randomly sampled in this Gaussian space to achieve the purpose of artificially adjusting the prosodic feature vector.

[0104] Compared with the training model method in traditional VC technology, the model updating method provided in the embodiment of the present application does not require paired audio data during training (each pair of data includes two audios with the same content generated in different timbres), and does not make rigid requirements on the semantic range of the audio contained in each timbre, thereby lowering the threshold for data collection and production. The traditional method requires the original timbre and the target timbre, and the content of the two must be consistent. The amount of training data is large, difficult to collect, and the cost is high. In the embodiment of the present application, the training sample database contains N audios with different timbres and arbitrary semantic content.

[0105] In summary of the above description, in the updating method of the sound conversion model provided by the embodiment of the present application, in order to realize the function of converting the input timbre into the target timbre, the sound conversion model can be called to perform feature processing on the first audio signal and the second audio signal corresponding to the target timbre to obtain a third audio signal, and then the first audio signal can be reconstructed based on the obtained third audio signal and the first audio signal, and then the parameters of the sound conversion model can be updated based on the proximity parameter of the first audio signal and the reconstructed first audio signal, and the above steps are repeatedly performed until the proximity parameter satisfies the termination condition, and when it is determined that the proximity parameter satisfies the termination condition, the last updated sound conversion model is obtained. When the proximity parameter of the first audio signal and the reconstructed first audio signal satisfies the termination condition, it means that the current sound conversion model can well realize the reconstruction of the first audio signal, that is, the current sound conversion model has a good capture capability for various features (including the target timbre) in the sound conversion process, and can realize the conversion function of the target timbre. And to evaluate whether the sound conversion model can well capture various features (including the target timbre) in the sound conversion process, in the above cycle, it can be judged by the first audio signal and the second audio signal corresponding to the target timbre. It can be seen that in the embodiment of the present application, by reconstructing the first audio signal and updating the sound conversion model based on the proximity parameter between the first audio signal and the reconstructed first audio signal, it is possible to perform timbre conversion on the input audio signal and generate audio with a target timbre. At the same time, it is possible to realize a conversion function of converting the timbre of the input audio signal into a newly added target timbre. In this way, there is no need to spend a lot of time and cost to collect more sample audios of new timbres.

[0106] Optional, see Figure 3 The embodiment of the present application further provides an application method of a sound conversion model, which can be applied to an application device of a sound conversion model. The method may include S301-S305:

[0107] S301: Acquire an input audio signal.

[0108] Among them, the input audio signal can be obtained through sound collection devices such as microphones, pickups, and mobile phone microphones.

[0109] S302: Calling a preset sound recognition model in the sound conversion model updated last time to extract a third semantic content from the input audio signal.

[0110] The third semantic content can be used as original corpus information for subsequently synthesizing the target audio signal.

[0111] S303: Call the first encoder in the sound conversion model updated last time, extract at least one fourth prosodic feature vector from the input audio signal, and obtain at least one adjusted fourth prosodic feature vector according to the adjustment information.

[0112] The at least one fourth prosodic feature vector may include prosodic feature vectors such as fundamental frequency, speech rate, pauses and volume which are extracted from the input audio signal and are independent of timbre.

[0113] Specifically, the adjustment information may be information set in advance according to actual needs. After extracting at least one fourth prosodic feature vector of the input audio signal through the first encoder, the at least one fourth prosodic feature vector may be linearly adjusted according to needs, for example, the speech speed may be slowed down, the pauses may be strengthened, or the volume may be increased. Through linear adjustment, the change of voice style may be achieved, for example, the low voice style may be transformed into a bright voice style, which may meet the personalized needs of users. In addition, the effect of extracting prosodic features may be improved.

[0114] In one embodiment, the linear transformation characteristics of Gaussian space latent features can be realized by spatial linear distance constraints. Specifically, when the prosodic features are adjusted, the multi-dimensional prosodic feature vector of the temporal space can be linearly transformed in the Gaussian space to finally obtain the adjusted and transformed prosodic feature vector.

[0115] S304: Acquire a reference audio signal of the target timbre, and call the second encoder in the sound conversion model updated last time to extract the fifth timbre feature vector of the reference audio signal.

[0116] Exemplarily, the duration of the reference audio signal of the target timbre may be greater than or equal to 3 seconds.

[0117] S305 , calling the decoder in the sound conversion model updated last time to perform audio synthesis processing on the third semantic content, the adjusted at least one fourth prosody feature vector, and the fifth timbre feature vector to obtain a target audio signal.

[0118] The target audio signal has the semantic content and rhythmic features of the input audio signal and the timbre of the reference audio signal.

[0119] The sound conversion model obtained by the embodiment of the present application can realize the extraction of effective information that can be used for sound conversion from the original sound of various different timbres, and keep the original semantic content unchanged. And it can support timbre conversion in various scenes, including single timbre-single timbre (One-to-One), multi-timbre-single timbre (Many-to-One), single timbre-multi-timbre (One-to-Many) and multi-timbre-multi-timbre (Many-to-Many). In addition, the technical solution provided by the embodiment of the present application can support the adjustment of the rhythmic features of the output audio with the target timbre, so as to further enhance the richness and expressiveness of the sound in various sound conversion scenes (such as live broadcast scenes). Exemplary, the existing sound conversion model can only support the conversion of the happy voice of the animated character A into the happy voice of the animated character B, and the characteristics of the two such as speech speed and rhythm are the same. The sound conversion model obtained by the embodiment of the present application can support the conversion of the happy voice of the animated character A into the melancholy voice of the animated character B, and the characteristics of the two such as speech speed and rhythm are different.

[0120] like Figure 4 As shown, an embodiment of the present application further provides a device for updating a sound conversion model, which may include: a processing module 11, a determination module 21 and an updating module 31.

[0121] The processing module 11 executes S101 and S102 in the above method embodiment, the determining module 21 executes S103 in the above method embodiment, and the updating module 31 executes S104 and S105 in the above method embodiment.

[0122] Specifically, the processing module 11 is used to execute step A: calling the sound conversion model to perform feature processing on the first audio signal and the second audio signal to obtain a third audio signal; the second audio signal is an audio signal of the target timbre;

[0123] The processing module 11 is further used to execute step B: calling the sound conversion model to perform feature processing on the first audio signal and the third audio signal to obtain a reconstructed first audio signal;

[0124] The determination module 21 is used to execute step C: determine whether a proximity parameter between the first audio signal and the reconstructed first audio signal processed by the processing module 11 satisfies a termination condition;

[0125] The updating module 31 is used to execute step D: when the determining module 21 determines that the proximity parameter does not satisfy the termination condition, the parameters of the sound conversion model are updated based on the proximity parameter, and steps A to C are repeated after the update until the determining module 21 determines that the proximity parameter satisfies the termination condition; when the determining module 21 determines that the proximity parameter satisfies the termination condition, the last updated sound conversion model is obtained.

[0126] Optionally, in a possible design, the sound conversion model is composed of a preset sound recognition model, a first encoder, a second encoder and a decoder, and the processing module 11 is specifically used for:

[0127] Extracting a first semantic content of a first audio signal through a preset sound recognition model, extracting at least one first prosodic feature vector of the first audio signal through a first encoder, and extracting a second timbre feature vector of a second audio signal through a second encoder; performing audio synthesis processing on the first semantic content, at least one first prosodic feature vector, and the second timbre feature vector through a decoder to obtain a third audio signal; calling a sound conversion model to perform feature processing on the first audio signal and the third audio signal to obtain a reconstructed first audio signal may include: extracting a second semantic content of the third audio signal through a preset sound recognition model, extracting at least one second prosodic feature vector of the third audio signal through the first encoder, and extracting a first timbre feature vector of the first audio signal through the second encoder; performing audio synthesis processing on the second semantic content, at least one second prosodic feature vector, and the first timbre feature vector through a decoder to obtain a reconstructed first audio signal.

[0128] Optionally, in another possible design, the proximity parameter includes at least a first loss value between the first audio signal and the reconstructed first audio signal, and the determination module 21 is specifically configured to:

[0129] Determine whether the first loss value reaches a first preset condition; if the first loss value reaches the first preset condition, determine whether the proximity parameter satisfies a termination condition.

[0130] Optionally, in another possible design, the proximity parameter further includes a second loss value of the first semantic content and the second semantic content, a third loss value of at least one first prosodic feature vector and at least one second prosodic feature vector, and a fourth loss value of the first timbre feature vector and the third timbre feature vector; the third timbre feature vector is obtained by calling the second encoder to perform feature extraction on the reconstructed first audio signal;

[0131] The determination module 21 is specifically used to: determine whether the first loss value reaches the first preset condition, determine whether the second loss value reaches the second preset condition, determine whether the third loss value reaches the third preset condition, and determine whether the fourth loss value reaches the fourth preset condition;

[0132] If the first loss value reaches the first preset condition, the second loss value reaches the second preset condition, the third loss value reaches the third preset condition, and the fourth loss value reaches the fourth preset condition, it is determined that the proximity parameter meets the termination condition.

[0133] Optionally, in another possible design, the second encoder is a variational autoencoder, and the processing module 11 is also used to execute step E: call the variational autoencoder to perform feature processing on the first audio signal to obtain a fourth timbre feature vector in the high-dimensional Gaussian space, and calculate the fifth loss value of the fourth timbre feature vector in the high-dimensional Gaussian space and the standard high-dimensional Gaussian probability distribution through KL divergence; the determination module 21 is also used to execute step F: determine whether the fifth loss value meets the fifth preset condition; the update module 31 is also used to execute step G: when the determination module 21 determines that the fifth loss value does not meet the fifth preset condition, update the parameters of the variational autoencoder based on the fifth loss value, and repeat steps E to F after the update until the determination module 21 determines that the fifth loss value meets the fifth preset condition; when it is determined that the fifth loss value meets the fifth preset condition, obtain the last updated variational autoencoder, and update the sound conversion model based on the updated variational autoencoder.

[0134] Optionally, in another possible design, at least one first prosodic feature vector is used to characterize at least one of the fundamental frequency, volume, speaking rate and pauses of the first audio signal, and at least one second prosodic feature vector is used to characterize at least one of the fundamental frequency, volume, speaking rate and pauses of the third audio signal.

[0135] Optionally, in another possible design, the updating module 31 is specifically used to update the parameters of the sound conversion model based on the proximity parameter in combination with the gradient descent rule, the back propagation algorithm and the chain rule.

[0136] Optionally, the device for updating the sound conversion model may further include a storage module, and the storage module is used to store program codes of the device for updating the sound conversion model, etc.

[0137] like Figure 5 As shown, the embodiment of the present application also provides an application device of a sound conversion model, which may include: an acquisition module 32 and a processing module 33.

[0138] The acquisition module 32 executes S301 in the above method embodiment, and the processing module 33 executes S302, S303, S304 and S305 in the above method embodiment.

[0139] Specifically, the acquisition module 32 is used to acquire the input audio signal; the processing module 33 is used to call the preset sound recognition model in the sound conversion model updated for the last time to extract the third semantic content from the input audio signal; the processing module 33 is also used to call the first encoder in the sound conversion model updated for the last time to extract at least one fourth prosodic feature vector from the input audio signal, and obtain at least one adjusted fourth prosodic feature vector according to the adjustment information; the processing module 33 is also used to acquire the reference audio signal of the target timbre through the acquisition module 32, and call the second encoder in the sound conversion model updated for the last time to extract the fifth timbre feature vector of the reference audio signal; the processing module 33 is also used to call the decoder in the sound conversion model updated for the last time to perform audio synthesis processing on the third semantic content, the adjusted at least one fourth prosodic feature vector and the fifth timbre feature vector to obtain the target audio signal.

[0140] like Figure 6 As shown, an embodiment of the present application also provides an electronic device, including a memory 41, a processor 42 (42-1 and 42-2), a bus 43 and a communication interface 44; the memory 41 is used to store computer execution instructions, and the processor 42 is connected to the memory 41 through the bus 43; when the electronic device is running, the processor 42 executes the computer execution instructions stored in the memory 41, so that the electronic device executes the sound conversion model update method or the sound conversion model application method provided in the above embodiments.

[0141] In a specific implementation, as an embodiment, the processor 42 may include one or more central processing units (CPUs), such as Figure 6 As an embodiment, the electronic device may include multiple processors 42, such as CPU0 and CPU1. Figure 6 42-1 and processor 42-2 are shown in FIG. Each CPU in these processors 42 may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor 42 here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0142] The memory 41 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 41 may exist independently and be connected to the processor 42 via a bus 43. The memory 41 may also be integrated with the processor 42.

[0143] In a specific implementation, the memory 41 is used to store the data in this application and the computer execution instructions corresponding to the software program for executing this application. The processor 42 can run or execute the software program stored in the memory 41, and call the data stored in the memory 41, and various functions of the electronic device.

[0144] The communication interface 44 uses any transceiver-like device for communicating with other devices or communication networks, such as a control system, a radio access network (RAN), a wireless local area network (WLAN), etc. The communication interface 44 may include a receiving unit to implement a receiving function, and a sending unit to implement a sending function.

[0145] The bus 43 may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. The bus 43 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0146] As an example, combining Figure 4 , the functions implemented by the processing module in the updating device of the sound conversion model are similar to Figure 6 When the updating device for the sound conversion model includes a storage module, the function implemented by the storage module is the same as Figure 6 The memory in implements the same function.

[0147] The explanation of the relevant contents in this embodiment can refer to the above method embodiment, which will not be repeated here.

[0148] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0149] An embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When a computer executes the instructions, the computer executes the sound conversion model updating method or the sound conversion model application method provided in the above embodiment.

[0150] Among them, the computer-readable storage medium, for example, can be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, RAM, ROM, an erasable programmable read only memory (EPROM), a register, a hard disk, an optical fiber, a CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the above, or any other form of computer-readable storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application-specific integrated circuit (ASIC). In an embodiment of the present application, the computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0151] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for updating a sound conversion model, characterized in that: include: Step A: calling a sound conversion model to perform feature processing on a first audio signal and a second audio signal to obtain a third audio signal; the second audio signal is an audio signal of a target timbre; Step B: calling the sound conversion model to perform feature processing on the first audio signal and the third audio signal to obtain a reconstructed first audio signal; Step C: determining whether a proximity parameter between the first audio signal and the reconstructed first audio signal satisfies a termination condition; Step D: When it is determined that the proximity parameter does not satisfy the termination condition, the parameters of the sound conversion model are updated based on the proximity parameter, and steps A to C are repeated after the update until it is determined that the proximity parameter satisfies the termination condition; when it is determined that the proximity parameter satisfies the termination condition, the last updated sound conversion model is obtained.

2. The method according to claim 1, characterized in that The sound conversion model is composed of a preset sound recognition model, a first encoder, a second encoder and a decoder, and the calling of the sound conversion model performs feature processing on the first audio signal and the second audio signal to obtain a third audio signal, including: Extracting a first semantic content of the first audio signal by using the preset sound recognition model, extracting at least one first prosodic feature vector of the first audio signal by using the first encoder, and extracting a second timbre feature vector of the second audio signal by using the second encoder; performing audio synthesis processing on the first semantic content, the at least one first prosodic feature vector, and the second timbre feature vector by using the decoder to obtain the third audio signal; The calling of the sound conversion model to perform feature processing on the first audio signal and the third audio signal to obtain a reconstructed first audio signal includes: extracting a second semantic content of the third audio signal through the preset sound recognition model, extracting at least one second prosodic feature vector of the third audio signal through the first encoder, and extracting a first timbre feature vector of the first audio signal through the second encoder; and performing audio synthesis processing on the second semantic content, the at least one second prosodic feature vector, and the first timbre feature vector through the decoder to obtain the reconstructed first audio signal.

3. The method according to claim 2, characterized in that The proximity parameter includes at least a first loss value between the first audio signal and the reconstructed first audio signal, and determining whether the proximity parameter between the first audio signal and the reconstructed first audio signal satisfies a termination condition includes: Determining whether the first loss value reaches a first preset condition; If the first loss value reaches the first preset condition, it is determined that the proximity parameter satisfies the termination condition.

4. The method according to claim 3, characterized in that The proximity parameter also includes a second loss value of the first semantic content and the second semantic content, a third loss value of the at least one first prosodic feature vector and the at least one second prosodic feature vector, and a fourth loss value of the first timbre feature vector and the third timbre feature vector; the third timbre feature vector is obtained by calling the second encoder to perform feature extraction on the reconstructed first audio signal, and the method further includes: Determining whether the first loss value meets the first preset condition, determining whether the second loss value meets the second preset condition, determining whether the third loss value meets the third preset condition, and determining whether the fourth loss value meets the fourth preset condition; If the first loss value reaches the first preset condition, the second loss value reaches the second preset condition, the third loss value reaches the third preset condition, and the fourth loss value reaches the fourth preset condition, it is determined that the proximity parameter meets the termination condition.

5. The method according to claim 2, characterized in that: The second encoder is a variational autoencoder, and the method further includes: Step E: calling the variational autoencoder to perform feature processing on the first audio signal to obtain a fourth timbre feature vector in a high-dimensional Gaussian space, and calculating a fifth loss value between the fourth timbre feature vector in the high-dimensional Gaussian space and a standard high-dimensional Gaussian probability distribution through KL divergence; Step F: Determine whether the fifth loss value satisfies a fifth preset condition; Step G: When it is determined that the fifth loss value does not satisfy the fifth preset condition, the parameters of the variational autoencoder are updated based on the fifth loss value, and steps E to F are repeated after the update until the fifth loss value satisfies the fifth preset condition; when it is determined that the fifth loss value satisfies the fifth preset condition, the variational autoencoder that was last updated is obtained, and the sound conversion model is updated based on the updated variational autoencoder.

6. The method according to claim 2, characterized in that The at least one first prosodic feature vector is used to characterize at least one of the fundamental frequency, volume, speaking rate and pauses of the first audio signal, and the at least one second prosodic feature vector is used to characterize at least one of the fundamental frequency, volume, speaking rate and pauses of the third audio signal.

7. The method according to any one of claims 1 to 6, characterized in that: The updating of the parameters of the sound conversion model based on the proximity parameter comprises: Based on the proximity parameter, the parameters of the sound conversion model are updated in combination with the gradient descent rule, the back propagation algorithm and the chain rule.

8. An application method of a sound conversion model, characterized in that: include: Get the input audio signal; Calling a preset sound recognition model in the sound conversion model updated last time to extract a third semantic content from the input audio signal; wherein the sound conversion model updated last time is obtained based on the sound conversion model updating method described in claim 1; Calling the first encoder in the sound conversion model updated for the last time, extracting at least one fourth prosodic feature vector from the input audio signal, and acquiring at least one adjusted fourth prosodic feature vector according to the adjustment information; Acquire a reference audio signal of a target timbre, and call the second encoder in the sound conversion model updated for the last time to extract a fifth timbre feature vector of the reference audio signal; The decoder in the sound conversion model updated for the last time is called to perform audio synthesis processing on the third semantic content, the at least one adjusted fourth prosody feature vector, and the fifth timbre feature vector to obtain a target audio signal.

9. A device for updating a sound conversion model, characterized in that: include: The processing module is used to execute step A: calling the sound conversion model to perform feature processing on the first audio signal and the second audio signal to obtain a third audio signal; the second audio signal is an audio signal of a target timbre; The processing module is further used to execute step B: calling the sound conversion model to perform feature processing on the first audio signal and the third audio signal to obtain a reconstructed first audio signal; A determination module, configured to execute step C: determining whether a proximity parameter between the first audio signal and the reconstructed first audio signal processed by the processing module satisfies a termination condition; An updating module is used to execute step D: when the determining module determines that the proximity parameter does not satisfy the termination condition, the parameters of the sound conversion model are updated based on the proximity parameter, and steps A to C are repeated after the update until the determining module determines that the proximity parameter satisfies the termination condition; when the determining module determines that the proximity parameter satisfies the termination condition, the last updated sound conversion model is obtained.

10. An application device of a sound conversion model, characterized in that: include: An acquisition module, used for acquiring an input audio signal; A processing module, used to call a preset sound recognition model in the sound conversion model updated last time, and extract a third semantic content from the input audio signal; wherein the sound conversion model updated last time is obtained based on the sound conversion model updating method described in claim 1; The processing module is further used to call the first encoder in the sound conversion model updated last time, extract at least one fourth prosodic feature vector from the input audio signal, and obtain at least one adjusted fourth prosodic feature vector according to the adjustment information; The processing module is further used to obtain a reference audio signal of the target timbre through the obtaining module, and call the second encoder in the sound conversion model updated for the last time to extract a fifth timbre feature vector of the reference audio signal; The processing module is further used to call the decoder in the sound conversion model updated for the last time to perform audio synthesis processing on the third semantic content, the at least one adjusted fourth prosody feature vector and the fifth timbre feature vector to obtain a target audio signal.

11. An electronic device, characterized in that: It includes a memory, a processor, a bus and a communication interface; the memory is used to store computer-executable instructions, and the processor is connected to the memory via the bus; When the electronic device is running, the processor executes the computer-executable instructions stored in the memory so that the electronic device executes the updating method of the sound conversion model as described in any one of claims 1 to 7 or executes the application method of the sound conversion model as described in claim 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the computer executes the instructions, the computer executes the method for updating the sound conversion model as described in any one of claims 1 to 7 or the method for applying the sound conversion model as described in claim 8.

Images