Animation character voice transformation system and method, storage medium, electronic device

Abstract

This application provides an animated character voice conversion system, method, storage medium, and electronic device. The system uses a character feature encoding module to obtain audio recordings of different animated characters by a target voice actor as character feature samples. A first weight and a first character feature are determined based on the similarity between the character feature vector corresponding to the character feature sample and the audio feature vector corresponding to the audio to be converted. A second weight is determined based on the phoneme similarity between the character feature vector corresponding to the first character feature and the audio feature vector corresponding to the audio to be converted. The character feature corresponding to the audio to be converted is obtained by weighted summation of the first weight and the second weight. The audio feature vector corresponding to the audio to be converted, the target audio feature of the target voice sample, the second personal feature of the target voice sample, and the target character feature are input into a voice conversion module to obtain the target voice. This solves the problem in related technologies where ideal voice conversion cannot be achieved during the dubbing process of animated characters.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to an animation character voice transformation system and method, storage medium, and electronic device. Background Technology

[0002] In recent years, with the advancement of artificial intelligence technology, speech synthesis technology has also developed rapidly. Text-to-Speech (TTS) technology is used to convert text into speech based on the user's selected timbre, tone, identity, emotion, accent, etc. Voice conversion is one of the important applications of speech synthesis; it is the process of converting the speech to be converted into the target speech according to the user's selection. This voice conversion technology is often used for replicating voice actors. Specifically, for characters in certain animated works, the voice actor's voice itself has become one of the important identifiers of that animator. Changing the voice actor, resulting in a change in the voice of the animated character, can seriously affect the viewer's viewing experience. In related technologies, some solutions propose to use voice conversion technology to replicate the original voice actor of an animated character as the target speaker. When the original voice actor is unable to continue voicing the animated character due to illness, job transfer, or other factors, another voice actor can use voice conversion technology to re-voice the animated character using the original voice actor's voice.

[0003] While the above solutions can address the issue of voice acting changes for animated characters to some extent, in practice, a particular voice actor often voices different characters, and the voice acting styles for different characters are often quite different.

[0004] On the one hand, animated characters are often given exaggerated character settings during the production process, which is reflected in their voices, resulting in exaggerated voice styles and strong voice personalities. This means that even if the same voice actor is used to voice different animated characters, they will present very different voice styles due to the influence of their character settings.

[0005] On the other hand, voice actors often artistically adapt their voices to fit the character's backstory. For example, if a character's storyline is tragic, the voice actor might intentionally portray sadness; if a character is comedic, the voice actor might deliberately sound funny. This artificial approach to voice acting not only amplifies the differences in voice styles between various animated characters, but also makes it difficult for voice conversion technologies to accurately replicate these characteristics. Furthermore, different voice actors have varying understandings and expressive abilities regarding the characters, making it difficult for other voice actors to replicate the original voice actor's performance when re-voicing the same character.

[0006] In related technologies, there is no effective solution yet to address the issue of unsatisfactory voice conversion during the replication process of animated characters' voice acting. Summary of the Invention

[0007] This application provides an animated character voice transformation system and method, storage medium, and electronic device to at least solve the problem in the related art that the voice transformation of animated characters cannot be ideally achieved during the dubbing process.

[0008] In one embodiment of this application, an animated character voice transformation system is provided, the system comprising:

[0009] The audio feature encoding module is configured to acquire target speech samples of the target voice actor and extract target audio features of the target speech samples according to the style category of the target speech samples.

[0010] The personal feature encoding module is configured to acquire the first personal feature of the target speech sample and determine the second personal feature of the target speech sample based on the first personal feature; wherein, the first personal feature is used to indicate the static sound characteristics of the target speech sample, and the second personal feature is used to indicate the predicted values ​​of the feature bias and gain of the first personal feature within a preset time period.

[0011] The character feature encoding module is configured to acquire the voice-over audio of the target voice actor for different animated characters as character feature samples, determine a first weight and a first character feature based on the style similarity between the character feature vector corresponding to the character feature sample and the audio feature vector corresponding to the audio to be converted, determine a second weight through the phoneme similarity between the character feature vector corresponding to the first character feature and the audio feature vector corresponding to the audio to be converted, and obtain the target character feature corresponding to the audio to be converted by weighted summation based on the first weight and the second weight; wherein, the style similarity includes at least: timbre, pitch, and tone similarity, the first weight is used to indicate the similarity between the character feature vector corresponding to different character feature samples and the audio feature vector corresponding to the audio to be converted, the first character feature is used to indicate the character feature sample with the highest similarity to the audio to be converted, and the second weight is used to indicate the phoneme similarity between the character feature vector corresponding to the first character feature and the audio feature vector corresponding to the audio to be converted;

[0012] The timbre conversion module is configured to fuse the audio feature vector corresponding to the audio to be converted, the target audio features of the target speech sample, the second personal features, and the target character features, and convert the audio to be converted into target speech that conforms to the style of the target voice actor.

[0013] In one embodiment, the timbre conversion module includes:

[0014] The feature encoding unit is configured to fuse the target audio features with the second personal features and the target role features to obtain joint encoded features; and to extract implicit features based on the target audio features.

[0015] The feature decoding unit is configured to generate a predicted value of the target speech sample based on the joint encoded features and the implicit features;

[0016] An audio prediction unit is configured to train a first loss function based on the predicted value of the target speech sample and the true value of the target speech sample; wherein the first loss function is used to indicate the loss of the predicted value of the target speech sample relative to the true value.

[0017] The alignment prediction unit is configured to align the joint encoded features with the implicit features to calculate an alignment matrix, and train a second loss function based on the alignment matrix. The second loss function is used to indicate the dispersion between the joint features and the implicit features.

[0018] The discrimination unit is configured to determine the discrimination result based on the difference between the predicted value and the true value of the target speech sample, and to train a third loss function based on the discrimination result;

[0019] The timbre conversion module is further configured to obtain a trained timbre conversion module based on the first loss function, the second loss function, and the third loss function, and to convert the audio to be converted into target speech that conforms to the style of the target voice actor through the trained timbre conversion module.

[0020] In one embodiment, the role feature encoding module includes:

[0021] The character feature extraction unit is configured to acquire the voice-over audio of the target voice actor for different animated characters as character feature samples, and extract the character feature vector corresponding to the character feature samples.

[0022] The character feature encoding unit is configured to select character feature vectors from the character feature vectors corresponding to the character feature samples that have a similarity requirement with the audio to be converted, and encode them to obtain the character features corresponding to the audio to be converted.

[0023] In one embodiment, the role feature encoding unit includes:

[0024] In the first attention layer, the audio feature vector corresponding to the audio to be converted is used as the query object, and the multiple character feature vectors extracted by the character feature extraction unit are used as the query key. Based on the similarity between the audio feature vector corresponding to the audio to be converted and the multiple character feature vectors, the first weight and the first character feature are determined.

[0025] The second attention layer uses the character feature vector corresponding to the first character feature as the query object and the audio feature vector corresponding to the audio to be converted as the query key. The second weight is determined based on the phoneme similarity between the audio feature vector corresponding to the audio to be converted and the character feature vector corresponding to the first character feature.

[0026] In one embodiment, the personal feature encoding module includes:

[0027] The spectrum processing unit, consisting of a fully connected layer, is configured to extract the Mel spectrum features of the target speech sample and convert them into a feature sequence.

[0028] The temporal processing unit includes a gated convolutional layer and a residual layer, configured to acquire the temporal information of the feature sequence;

[0029] An attention unit is configured to extract the first personal features corresponding to the feature sequence within a plurality of first preset durations based on the temporal information of the feature sequence, and to average the plurality of first personal features extracted within the plurality of first preset durations within a second preset duration to obtain a style vector, wherein the second preset duration includes the plurality of first preset durations.

[0030] The style adaptive unit is configured to predict the feature bias and gain of the first personal feature based on the style vector, and use the predicted feature bias and gain of the first personal feature as the second personal feature of the target speech sample.

[0031] In one embodiment, the feature encoding unit is further configured to fuse the target audio features extracted by the trained audio feature encoding module with the second personal features extracted by the trained style encoding module to obtain the joint encoded features, and extract the corresponding implicit features based on the real Mel spectrum features of the target audio features; the feature decoding unit is further configured to determine the estimated Mel spectrum features corresponding to the target audio features based on the estimated value of the generated target speech sample; the audio estimation unit is further configured to train the first loss function based on the real Mel spectrum features of the target audio features and the estimated Mel spectrum features corresponding to the target audio features; wherein the first loss function is used to indicate the loss of the estimated Mel spectrum features corresponding to the target audio features relative to the estimated Mel spectrum features corresponding to the target audio features.

[0032] In one embodiment, the discrimination unit is further configured to perform adversarial training on the feature encoding unit, the feature decoding unit, the audio prediction unit, and the alignment prediction unit, respectively, and train the third loss function based on the result of the adversarial training.

[0033] According to another embodiment of this application, a method for voice transformation of animated characters is also provided, comprising: acquiring audio to be converted and target voice samples of a target voice actor; identifying the style category of the target voice sample through an audio feature encoding module, and extracting target audio features of the target voice sample according to the style category of the target voice sample; wherein the target audio features include text features, prosodic features, and timbre features of the target voice sample; extracting the corresponding audio feature vector of the audio to be converted through the audio feature encoding module, wherein the corresponding audio feature vector of the audio to be converted includes text features, prosodic features, and timbre features of the audio to be converted; acquiring a first personal feature of the target voice sample through a personal feature encoding module, and determining a second personal feature of the target voice sample according to the first personal feature; wherein the first personal feature is used to indicate the static sound characteristics of the target voice sample, and the second personal feature is used to indicate the predicted values ​​of feature bias and gain of the first personal feature within a preset duration; acquiring the voice-over audio of the target voice actor for different animated characters as a character feature sample through a character feature encoding module, and based on the character feature vector corresponding to the character feature sample and the audio to be converted... The similarity of the corresponding audio feature vectors determines the first weight and the first character feature. The phoneme similarity between the character feature vector corresponding to the first character feature and the audio feature vector corresponding to the audio to be converted determines the second weight. The character feature corresponding to the audio to be converted is obtained by weighted summation based on the first weight and the second weight. The first weight is used to indicate the similarity between the character feature vector corresponding to different character feature samples and the audio feature vector corresponding to the audio to be converted. The first character feature is used to indicate the character feature sample with the highest similarity to the audio to be converted. The second weight is used to indicate the phoneme similarity between the character feature vector corresponding to the first character feature and the audio feature vector corresponding to the audio to be converted. The audio feature vector corresponding to the audio to be converted, the target audio feature of the target speech sample, the second personal feature of the target speech sample, and the target character feature are input into the timbre conversion module. The timbre conversion module performs fusion mapping to obtain joint encoded features. After standard streaming operation on the joint encoded features, the decoding is performed to obtain the target speech feature corresponding to the speaking style of the target voice actor. The original speech is converted based on the target speech feature to obtain the target speech.

[0034] In one embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to execute the steps of any of the above method embodiments at runtime.

[0035] In one embodiment of this application, an electronic device is also provided, including a memory and a processor, characterized in that the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0036] The animation character voice transformation system provided in this application includes an audio feature encoding module, a personal feature encoding module, a character feature encoding module, and a timbre conversion module. It acquires the audio to be converted and target voice samples from a target voice actor. The audio feature encoding module identifies the style category of the target voice sample and extracts target audio features based on the style category. The personal feature encoding module acquires the first personal feature of the target voice sample and determines the second personal feature based on the first personal feature. The character feature encoding module acquires the voice-over audio of the target voice actor for different animation characters as character feature samples, and then converts the audio based on the character feature vector corresponding to the character feature sample and the audio feature vector corresponding to the audio to be converted. The similarity determines the first weight and the first character feature. The second weight is determined by the phoneme similarity between the character feature vector corresponding to the first character feature and the audio feature vector corresponding to the audio to be converted. The character feature corresponding to the audio to be converted is obtained by weighted summation based on the first weight and the second weight. The audio feature vector corresponding to the audio to be converted, the target audio feature of the target speech sample, the second personal feature of the target speech sample, and the target character feature are input into the timbre conversion module to obtain the target speech. This solves the problem that the voice conversion of the dubbing of animated characters cannot be ideally achieved in the replication process in related technologies. It can effectively identify the speaking style of the target voice actor and convert the original speech to be converted according to the speaking style of the target voice actor, effectively replicating the timbre of the target voice actor. Attached Figure Description

[0037] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0038] Figure 1 This is a hardware structure block diagram of a mobile terminal for an animated character voice transformation method according to an embodiment of this application;

[0039] Figure 2 This is a structural block diagram of an optional animation character voice transformation system according to an embodiment of this application;

[0040] Figure 3 This is a flowchart of an optional animated character voice transformation method according to an embodiment of this application;

[0041] Figure 4 This is a schematic diagram of the training process of an optional audio feature encoding module according to an embodiment of this application;

[0042] Figure 5 This is a schematic diagram of the structure of an optional timbre conversion module according to an embodiment of this application;

[0043] Figure 6 This is a schematic diagram of an optional electronic device structure according to an embodiment of this application. Detailed Implementation

[0044] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0045] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0046] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for voice transformation of animated characters according to an embodiment of this application. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0047] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer programs corresponding to the training methods of the speech conversion model, clustering model, audio feature encoding module, style encoding module, or timbre conversion module in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thus implementing the methods described above. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0048] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0049] Figure 2 This is a structural block diagram of an optional animation character voice transformation system according to an embodiment of this application, such as... Figure 2 As shown, the system includes:

[0050] The audio feature encoding module 202 is configured to acquire target speech samples of the target voice actor and extract target audio features of the target speech samples according to the style category of the target speech samples.

[0051] The personal feature encoding module 204 is configured to acquire the first personal feature of the target speech sample and determine the second personal feature of the target speech sample based on the first personal feature; wherein, the first personal feature is used to indicate the static sound characteristics of the target speech sample, and the second personal feature is used to indicate the predicted values ​​of the feature bias and gain of the first personal feature within a preset time period.

[0052] The character feature encoding module 206 is configured to acquire the voice-over audio of the target voice actor for different animated characters as character feature samples, determine a first weight and a first character feature based on the style similarity between the character feature vector corresponding to the character feature sample and the audio feature vector corresponding to the audio to be converted, determine a second weight through the phoneme similarity between the character feature vector corresponding to the first character feature and the audio feature vector corresponding to the audio to be converted, and obtain the target character feature corresponding to the audio to be converted by weighted summation based on the first weight and the second weight; wherein, the style similarity includes at least: timbre, pitch, and tone similarity, the first weight is used to indicate the similarity between the character feature vector corresponding to different character feature samples and the audio feature vector corresponding to the audio to be converted, the first character feature is used to indicate the character feature sample with the highest similarity to the audio to be converted, and the second weight is used to indicate the phoneme similarity between the character feature vector corresponding to the first character feature and the audio feature vector corresponding to the audio to be converted;

[0053] The timbre conversion module 208 is configured to fuse the audio feature vector corresponding to the audio to be converted, the target audio feature of the target speech sample, the second personal feature and the target role feature, and convert the audio to be converted into a target speech that conforms to the style of the target voice actor.

[0054] In one embodiment, the timbre conversion module includes:

[0055] The feature encoding unit is configured to fuse the target audio features with the second personal features and the target role features to obtain joint encoded features; and to extract implicit features based on the target audio features.

[0056] The feature decoding unit is configured to generate a predicted value of the target speech sample based on the joint encoded features and the implicit features;

[0057] An audio prediction unit is configured to train a first loss function based on the predicted value of the target speech sample and the true value of the target speech sample; wherein the first loss function is used to indicate the loss of the predicted value of the target speech sample relative to the true value.

[0058] The alignment prediction unit is configured to align the joint encoded features with the implicit features to calculate an alignment matrix, and train a second loss function based on the alignment matrix. The second loss function is used to indicate the dispersion between the joint features and the implicit features.

[0059] The discrimination unit is configured to determine the discrimination result based on the difference between the predicted value and the true value of the target speech sample, and to train a third loss function based on the discrimination result;

[0060] The timbre conversion module is further configured to obtain a trained timbre conversion module based on the first loss function, the second loss function, and the third loss function, and to convert the audio to be converted into target speech that conforms to the style of the target voice actor through the trained timbre conversion module.

[0061] In one embodiment, the role feature encoding module includes:

[0062] The character feature extraction unit is configured to acquire the voice-over audio of the target voice actor for different animated characters as character feature samples, and extract the character feature vector corresponding to the character feature samples.

[0063] The character feature encoding unit is configured to select character feature vectors from the character feature vectors corresponding to the character feature samples that have a similarity requirement with the audio to be converted, and encode them to obtain the character features corresponding to the audio to be converted.

[0064] In one embodiment, the role feature encoding unit includes:

[0065] In the first attention layer, the audio feature vector corresponding to the audio to be converted is used as the query object, and the multiple character feature vectors extracted by the character feature extraction unit are used as the query key. Based on the similarity between the audio feature vector corresponding to the audio to be converted and the multiple character feature vectors, the first weight and the first character feature are determined.

[0066] The second attention layer uses the character feature vector corresponding to the first character feature as the query object and the audio feature vector corresponding to the audio to be converted as the query key. The second weight is determined based on the phoneme similarity between the audio feature vector corresponding to the audio to be converted and the character feature vector corresponding to the first character feature.

[0067] In one embodiment, the personal feature encoding module includes:

[0068] The spectrum processing unit, consisting of a fully connected layer, is configured to extract the Mel spectrum features of the target speech sample and convert them into a feature sequence.

[0069] The temporal processing unit includes a gated convolutional layer and a residual layer, configured to acquire the temporal information of the feature sequence;

[0070] An attention unit is configured to extract the first personal features corresponding to the feature sequence within a plurality of first preset durations based on the temporal information of the feature sequence, and to average the plurality of first personal features extracted within the plurality of first preset durations within a second preset duration to obtain a style vector, wherein the second preset duration includes the plurality of first preset durations.

[0071] The style adaptive unit is configured to predict the feature bias and gain of the first personal feature based on the style vector, and use the predicted feature bias and gain of the first personal feature as the second personal feature of the target speech sample.

[0072] In one embodiment, the feature encoding unit is further configured to fuse the target audio features extracted by the trained audio feature encoding module with the second personal features extracted by the trained style encoding module to obtain the joint encoded features, and extract the corresponding implicit features based on the real Mel spectrum features of the target audio features; the feature decoding unit is further configured to determine the estimated Mel spectrum features corresponding to the target audio features based on the estimated value of the generated target speech sample; the audio estimation unit is further configured to train the first loss function based on the real Mel spectrum features of the target audio features and the estimated Mel spectrum features corresponding to the target audio features; wherein the first loss function is used to indicate the loss of the estimated Mel spectrum features corresponding to the target audio features relative to the estimated Mel spectrum features corresponding to the target audio features.

[0073] In one embodiment, the discrimination unit is further configured to perform adversarial training on the feature encoding unit, the feature decoding unit, the audio prediction unit, and the alignment prediction unit, respectively, and train the third loss function based on the result of the adversarial training.

[0074] Figure 3 This is a flowchart of an optional animated character voice transformation method according to an embodiment of this application. According to another embodiment of this application, an animated character voice transformation method is also provided, including:

[0075] Step S302: Obtain the audio to be converted and the target voice sample of the target voice actor;

[0076] Step S304: Identify the style category of the target speech sample through the audio feature encoding module, and extract the target audio features of the target speech sample according to the style category of the target speech sample; wherein, the target audio features include the text features, prosodic features and timbre features of the target speech sample;

[0077] Step S306: Extract the corresponding audio feature vector of the audio to be converted through the audio feature encoding module, wherein the corresponding audio feature vector of the audio to be converted includes the text features, prosodic features and timbre features of the audio to be converted;

[0078] Step S308: Obtain the first personal feature of the target speech sample through the personal feature encoding module, and determine the second personal feature of the target speech sample based on the first personal feature; wherein, the first personal feature is used to indicate the static sound characteristics of the target speech sample, and the second personal feature is used to indicate the predicted values ​​of the feature bias and gain of the first personal feature within a preset time period.

[0079] Step S310: Obtain the voice-over audio of the target voice actor for different animated characters as character feature samples through the character feature encoding module. Determine a first weight and a first character feature based on the similarity between the character feature vector corresponding to the character feature sample and the audio feature vector corresponding to the audio to be converted. Determine a second weight based on the phoneme similarity between the character feature vector corresponding to the first character feature and the audio feature vector corresponding to the audio to be converted. Obtain the character feature corresponding to the audio to be converted by weighted summation based on the first weight and the second weight. Wherein, the first weight is used to indicate the similarity between the character feature vector corresponding to different character feature samples and the audio feature vector corresponding to the audio to be converted, the first character feature is used to indicate the character feature sample with the highest similarity to the audio to be converted, and the second weight is used to indicate the phoneme similarity between the character feature vector corresponding to the first character feature and the audio feature vector corresponding to the audio to be converted.

[0080] Step S312: Input the audio feature vector corresponding to the audio to be converted, the target audio feature of the target speech sample, the second personal feature of the target speech sample, and the target role feature into the timbre conversion module. The timbre conversion module performs fusion mapping to obtain joint coding features. After performing standard streaming operation on the joint coding features, decode them to obtain target speech features corresponding to the speaking style of the target voice actor. Based on the target speech features, convert the original speech to obtain the target speech.

[0081] It should be noted that during the inference and usage phases, the audio feature encoding module needs to identify the target speech sample and extract features based on the style type of the target speech sample. Different style types can use different encoding methods to extract the target audio features of the target speech sample, thereby improving the targeting of the sound transformation.

[0082] It should be noted that, for an audio clip, during processing, some features are explicitly known to indicate the timbre and rhythm of the audio; these features are referred to as audio features. Other features are explicitly known to indicate personal style; these features are referred to as personal features. However, some features are unknown to the indicated object before training; their potential representation of timbre, rhythm, personal style, etc., can only be inferred from the model training results. These features are collectively defined as implicit features here.

[0083] It should be noted that the third loss function is equivalent to the loss function in adversarial training, the feature encoding unit, the feature decoding unit, the audio prediction unit, and the alignment prediction unit are equivalent to the generator in a generative adversarial network, and the discriminator unit is equivalent to the discriminator.

[0084] It should be noted that the main method for extracting audio features in related technologies is through large-scale self-supervised models such as SoftHubert, Hubbert, and WAV2Vec2.0. This allows for the acquisition of richer features and knowledge. However, from the perspective of the final voice transformation result, the timbre of the target speech cannot accurately reflect the timbre characteristics or individuality of the target speaker's voice. Specifically, because the aforementioned audio feature extraction methods do not specifically address the timbre characteristics of the target speaker, if the target speaker's timbre is extreme, such as that of certain singers with distinctive voices, the above audio feature extraction methods will lead to a certain degree of "averaging" of the target speaker's timbre during training. This means the target speaker's timbre will become similar to that of other speakers, thus failing to fully represent the target speaker's timbre. Consequently, the final voice transformation result will not achieve the desired effect. Therefore, the method of directly extracting audio features through large-scale self-supervised models in related technologies still has certain shortcomings in style transformation.

[0085] To address the aforementioned shortcomings, this application embodiment uses a pre-trained clustering model to perform clustering processing during audio feature extraction in the audio feature encoding module. By clustering the features, the quality of the features and the effect of sound conversion are improved, as detailed below.

[0086] The first step is to pre-set two clustering models, namely the first clustering model and the second clustering model described above. These two clustering models can be k-means models, but are not limited to them (K-means clustering, hierarchical clustering (SPSS), and DBSCAN clustering (MATLAB) are all acceptable). Both the first and second clustering models include feature extractors for feature extraction during the training process. The feature extractors for the first and second clustering models can be any two of the three types of models mentioned above: softhubert, hubert, and wav2vec2.0. For example, the first clustering model can use hubert as its feature extractor, and the second clustering model can use wav2vec2.0. The first and second clustering models can use different feature extractors.

[0087] The second step involves pre-training the first and second clustering models using general training data. This general training data can be based on LibriSpeech-960 and AISHELL-3 datasets, specifically acquiring speech samples from 200 speakers for each model, resulting in 200 clusters. The purpose of this training is to enable the two clustering models to classify speech samples from different speakers. Each model employs a different feature extractor, allowing for clustering from different dimensions. Different feature extraction methods have advantages in timbre, prosody, and speech rhythm; therefore, combining these different feature extraction methods can enrich the subsequent training and inference processes of the voice transformation system.

[0088] During the training of the clustering model, the cluster categories need to be manually encoded. For example, after the first clustering model clusters the speech sample data and obtains different categories, they can be assigned ID1.1, ID1.2, ..., ID1.9, and so on. Similarly, after the second clustering model clusters the speech sample data and obtains different categories, they can be assigned ID2.1, ID2.2, ..., ID2.9, and so on. Here, the purpose of category encoding is to give each category after clustering by the clustering model a unique identifier for differentiation, so as to facilitate category mapping and encoding during the subsequent training of the voice transformation system.

[0089] It should be noted that the clustering model involved in this application embodiment is trained in an unsupervised manner, that is, it is clustered solely based on the similarity of samples without relying on labels. The first training samples themselves do not have category labels. After clustering, category labels are manually assigned to sample sets of different types. Furthermore, the same category label does not necessarily correspond to the same speaker, but rather to speech with similar styles. For example, the first training sample includes 100 speech samples from different speakers. Through the clustering model, these can be divided into 10 sample sets, each corresponding to a certain style type, such as a hoarse male voice, a clear female voice, etc., and then these 10 sample sets are manually assigned corresponding category labels.

[0090] It should be noted that the aforementioned first and second clustering models do not participate in the construction of the audio feature encoding module; they only provide category encodings during the training phase of the audio feature encoding module. Similarly, for the trained audio feature encoding module, the first and second clustering models do not participate in the inference work during the actual sound conversion process. Correspondingly, these clustering models also do not participate in the subsequent inference part.

[0091] In one embodiment, training the initial audio feature encoding module according to the loss function of the initial audio feature encoding module includes:

[0092] The initial audio feature encoding module is trained using the following loss function formula:

[0093]

[0094] Where p represents probability, i represents the identifier of the cluster category, and s t Let d represent the target vector, and e represent the probability that the current feature vector satisfies the target vector and belongs to class i. i Let sim(.,.) represent the trainable feature vectors for the current category i, τ represent the calculation of cosine similarity, K represent the number of clusters, and e represent the number of clusters. k This represents the trainable feature vectors for cluster category k.

[0095] It should be noted that the audio feature encoding module in this application embodiment may include the following three units:

[0096] Backbone network: Based on the Hubert network structure, it is used to extract and encode audio features.

[0097] Category mapping unit (linear projection): Consists of a mapping layer. This category mapping unit is used for category coding, that is, the category code corresponding to the audio feature map extracted by the feature coding unit.

[0098] Speech units: These consist of an embedding layer. During the training of the audio feature coding module, the speech units assign the category codes of the aforementioned clustering model to the audio features extracted by the feature coding units.

[0099] During training, the feature encoding units and category mapping units are first initialized, that is, some parameters of the Hubert network and mapping layers are randomly initialized. After initialization, the model is still trained using general training data. The general model data used for training here can be the same as or different from the training samples of the aforementioned clustering model; there are no restrictions on this.

[0100] Figure 4 This is a schematic diagram illustrating the training process of an optional audio feature encoding module according to an embodiment of this application, as shown below. Figure 4 As shown, in one embodiment, training the initial audio feature encoding module to convergence based on the true category labels output by the first clustering model and the second clustering model and the predicted category labels output by the initial audio feature encoding module includes:

[0101] The feature encoding unit and category mapping unit of the initial audio feature encoding module are initialized.

[0102] The feature encoding unit extracts and encodes the audio features of the third training sample, and the category mapping unit maps the audio features extracted by the feature encoding unit to the corresponding category code to obtain the predicted category label corresponding to the third training sample.

[0103] The predicted category label is output by the category encoding unit of the initial audio feature encoding module. Based on the predicted category label and the true category label output by the first clustering model and the second clustering model, the initial audio feature encoding module is trained until convergence.

[0104] In one embodiment, training the initial audio feature encoding module to convergence based on the predicted category labels and the true category labels output by the first clustering model and the second clustering model includes:

[0105] The loss function of the initial audio feature encoding module is updated based on the minimized first average cross-entropy between the true class label and the predicted class label output by the first clustering model, and the training parameters of the initial audio encoding module are also updated.

[0106] It should be noted that, during the training of the audio feature encoding module, in addition to the normal training and parameter updates of the feature encoding units, this embodiment also trains the audio encoding module based on the predicted class encoding of the corresponding class encoding of the training speech samples and the true class encoding ID obtained from the aforementioned clustering model. Specifically, the average cross-entropy between the true class encoding of Hubert's k-means (first clustering model) and the predicted class encoding of the audio encoding module is minimized, while the average cross-entropy between the true class encoding of wav2vec's k-means (second clustering model) and the predicted class encoding of the audio encoding module is also minimized. Based on this, the aforementioned loss is updated, and the parameters of the audio encoding module are updated accordingly. Through the above training method, the ability of the audio encoding module to classify timbre categories is further enhanced.

[0107] Another embodiment of this application provides an improvement to the timbre conversion module, which mainly includes combining the personal characteristics of the style coding module during the timbre conversion process to improve the timbre conversion effect.

[0108] To address the issue that the style transfer effect of directly extracting audio features using large-scale self-supervised models in related technologies still has certain shortcomings, one embodiment of this application is based on a pre-trained clustering model. During the audio feature extraction process in the audio feature encoding module, clustering is performed on the features. By clustering the features, the extraction effect of personalized styles of some target speakers with strong personalities can be significantly improved, thereby improving the effect of subsequent voice conversion.

[0109] Since the clustering of features used in the aforementioned embodiments enables the extracted features for the target speaker to better represent the speaker's voice individuality, in order to further utilize the voice characteristics of the target speaker highlighted by the above features in the subsequent voice conversion process, this application embodiment further improves the timbre conversion part in the subsequent voice conversion process, so that it can better use personal features to enhance the timbre conversion effect.

[0110] It should be noted that in the embodiments of this application, the scheme for personal characteristics in the timbre conversion process is based on the aforementioned clustering of features. That is, only through the clustered features can the style encoding module and the relevant schemes in the timbre conversion module be used to further encode the style and perform subsequent timbre conversion.

[0111] It should be noted that the personal feature encoding module focuses on traditional style encoding approaches, emphasizing the extraction of the target speaker's own vocal characteristics, such as vocal rhythm; while the character encoding module, based on the content the voice actor needs to dub, extracts the voice actor's personalized expression of the animated character during dubbing and performs voice conversion. Through the cooperation of these two encoding modules, the overall style is learned.

[0112] The character encoding module includes a character feature extraction unit, which consists of a self-supervised model pre-trained using general data, such as SoftHubert, Hubert, or WAV2Vec2.0. Based on this, audio recordings of original voice actors for different animated characters are pre-collected as character feature samples; these multiple character feature samples are defined as the character feature set. Multiple character feature samples from the character feature set are then input into the self-supervised model to extract corresponding style features (e.g., SSL features).

[0113] The character feature encoding unit is used to select appropriate character features from the above style features and encode them to obtain the corresponding character feature information. The character feature encoding unit is built on a Transformer network. This invention sets up two attention layers on the basis of the basic Transformer network.

[0114] In this invention, a character feature set is first constructed to obtain as many original voice actors' voice acting for different animated characters as possible. Then, through a first attention layer, based on the similarity between different character feature samples in the character feature set and the audio to be processed, the character feature samples most similar to the audio to be processed are selected. Furthermore, through a second attention layer, the first and the audio to be processed are processed, so that the model can further calculate the corresponding character features based on the similarity between the two, so that the audio to be processed can better replicate the character features of the first style.

[0115] By first applying the attention mechanism to the entire set of character features, no subsequent output is required during this process; only the corresponding first style is labeled. Therefore, even if the target style samples in the character feature set are very large, the computational load is relatively limited, effectively controlling the computational burden. Based on this, the attention mechanism is then applied to the first style and the audio to be processed, significantly improving computational efficiency while accurately acquiring the corresponding character features. It is important to note that when calculating the attention mechanism in the first and second attention layers, the query object and query key are interchanged. This allows for the calculation of attention between features from different dimensions, further improving the final performance while controlling the overall model parameters.

[0116] During the overall model training process, the training samples used are the original voice actors' voiceovers of a certain animated character, A. If the aforementioned character feature encoding module is not used, then in the subsequent model inference process, the current voice actor's voice conversion is based solely on voiceover A. Theoretically, if voiceover A perfectly matches the content that the current voice actor needs to dub, a good dubbing effect can indeed be achieved, that is, the original voice actor's dubbing style and personalized expression can be transferred. However, in practice, it is almost impossible to find dubbing samples that perfectly match the content that the current voice actor needs to dub. This is mainly due to two aspects:

[0117] 1) A voice actor frequently performs voice acting for different animated characters, making it difficult to label the styles of different voice acting audio and build a database. Therefore, it is impossible to directly find the original voice acting audio that matches the style of the current voice acting content.

[0118] 2) Although the current dubbing content may be the same animated character as the original voice actor, the plot and lines are different, which may result in different dubbing styles. Therefore, simply converting the voice using the original dubbing audio will not produce ideal results.

[0119] By introducing the aforementioned character feature encoding module, based on the preceding analysis, in practice, only the original dubbing audio needs to be directly used to construct a character feature set to achieve the relevant calculations. This not only solves the problem in the aforementioned background technology of being unable to effectively replicate personalized style expressions, but also effectively reduces time and manpower costs, making the entire solution feasible. It should be noted that the aforementioned character feature encoding module is trained independently of the overall model in this invention. That is, during the overall model training process, character features can be disregarded, and the original dubbing audio related to the current dubbing content, such as dubbing A mentioned above, can be directly used as samples to train the model. In this process, the character feature encoding module does not participate in the training. However, during the model inference process, on the one hand, the current dubbing content completes the transfer of rhythm and other styles based on dubbing A, and on the other hand, character features can be learned based on other original dubbing audio using the character feature encoding module.

[0120] During the calculation process of the character feature encoding module, the first style it selects may not be voice A, and may not even be the same animated character represented by voice A. For example, if a voice actor needs to dub a character (A), a segment of the original voice actor's dubbing of character A can be selected to train the overall model. However, the current dubbing content may correspond to different plot scenarios for character A compared to the original dubbing content used in the training process. For instance, in the original dubbing content, character A might be associated with a relatively cheerful plot, while in the current dubbing content, character A might be associated with a relatively tragic plot. This kind of plot variation cannot be covered by the original dubbing content. In this case, the original dubbing content can still be selected to train the overall model. However, during the model inference process, the character feature encoding module may select the original voice actor's dubbing of character B, which fits the setting of a tragic plot. Thus, the model, on the one hand, performs prosody and other dimensions transformation on the current dubbing content based on the original dubbing audio (A), and on the other hand, the character feature encoding module learns the character features of the original voice actor's personalized performance corresponding to the tragic plot based on the original dubbing audio (B). Combining the two, the current dubbing content can be effectively transformed by referring to the original voice actor's voice and personalized performance of the tragic plot.

[0121] Another embodiment of this application provides an improvement to the timbre conversion module, which mainly includes combining the personal characteristics of the style coding module during the timbre conversion process to improve the timbre conversion effect.

[0122] In one embodiment, the personal feature encoding module includes:

[0123] The spectrum processing unit, consisting of a fully connected layer, is configured to extract the Mel spectrum features of the target speech sample and convert them into a feature sequence.

[0124] The temporal processing unit includes a gated convolutional layer and a residual layer, configured to acquire the temporal information of the feature sequence;

[0125] An attention unit is configured to extract the first personal features corresponding to the feature sequence within a plurality of first preset durations based on the temporal information of the feature sequence, and to average the plurality of first personal features extracted within the plurality of first preset durations within a second preset duration to obtain a style vector, wherein the second preset duration includes the plurality of first preset durations.

[0126] The style adaptive unit is configured to predict the feature bias and gain of the first personal feature based on the style vector, and use the predicted feature bias and gain of the first personal feature as the second personal feature of the target speech sample.

[0127] In one embodiment, the feature encoding unit is further configured to fuse the target audio features extracted by the trained audio feature encoding module with the second personal features extracted by the trained style encoding module to obtain the joint encoded features, and extract the corresponding implicit features based on the true Mel spectrum features of the target audio features;

[0128] The feature decoding unit is further configured to determine the estimated Mel spectrum features corresponding to the target audio features based on the estimated value of the generated target speech sample.

[0129] The audio prediction unit is further configured to train the first loss function based on the true Mel spectrum features of the target audio feature and the predicted Mel spectrum features corresponding to the target audio feature; wherein the first loss function is used to indicate the loss of the predicted Mel spectrum features corresponding to the target audio feature relative to the predicted Mel spectrum features corresponding to the target audio feature.

[0130] In one embodiment, the discrimination unit is further configured to perform adversarial training on the feature decoding unit, the audio prediction unit, and the alignment prediction unit, respectively, and train the third loss function based on the result of the adversarial training.

[0131] Another embodiment of this application provides an improvement to the timbre conversion module, which mainly includes combining the personal characteristics of the style coding module during the timbre conversion process to improve the timbre conversion effect.

[0132] Figure 5 This is a schematic diagram of the structure of an optional timbre conversion module according to an embodiment of this application, as shown below. Figure 5As shown, the timbre conversion module includes: a feature encoding unit, a feature decoding unit, an audio prediction unit, and an alignment prediction unit.

[0133] During the timbre conversion process, the audio feature encoding module acquires the target speech sample of the target speaker and extracts the target audio features of the target speech sample according to the style category of the target speech sample; the personal feature encoding module acquires the first personal feature of the target speech sample and determines the second personal feature of the target speech sample according to the first personal feature; wherein, the first personal feature is used to indicate the static sound characteristics of the target speech sample, and the second personal feature is used to indicate the predicted values ​​of the feature bias and gain of the first personal feature within a preset duration.

[0134] Based on the aforementioned processing, the timbre conversion module is trained in the following way.

[0135] The feature encoding unit is configured to fuse the aforementioned target audio features with the second human features to obtain a joint feature encoding; and to extract implicit features based on the target audio features.

[0136] The feature decoding unit is configured to generate a predicted value of the target speech sample based on the joint feature encoding and implicit features;

[0137] An audio prediction unit is configured to train a first loss function based on the predicted value of a target speech sample and the true value of the target speech sample; wherein the first loss function is used to indicate the loss of the predicted value of the target speech sample relative to the true value.

[0138] The alignment prediction unit is configured to align the joint features with the implicit features to calculate the alignment matrix, and train a second loss function based on the alignment matrix. The second loss function is used to indicate the dispersion between the joint features and the implicit features.

[0139] During the training process of the timbre conversion module, an identification unit is also set up, configured to determine the identification result based on the difference between the predicted value and the true value of the target speech sample, and train a third loss function based on the identification result; the timbre conversion module obtains the trained timbre conversion module based on the first loss function, the second loss function, and the third loss function, and converts the speech to be processed into the target speech corresponding to the target speaker through the trained timbre conversion module.

[0140] It should be noted that, for an audio clip, if some features are clearly known during the processing to indicate the timbre and rhythm of the audio, then those features are the aforementioned audio features; if some features are clearly known to indicate personal style, then those features are the aforementioned personal features.

[0141] However, the relationship between some features and the indicated object is unknown before training. It can only be deduced from the model training results that they may represent timbre, rhythm, personal style, etc. These features are uniformly defined as implicit features here.

[0142] The style encoding module processes the speaker's voice style, including timbre and rhythm. For the input speech, the style encoding module outputs corresponding style information (equivalent to the aforementioned "second person feature"), which includes the speaker's identity and rhythm. Specifically, the style encoding module is structured as follows:

[0143] The linear spectrogram (MSG) consists of a fully connected layer and is used to obtain the Mel spectrum of the input speech and convert it into a feature sequence.

[0144] Temporal processing unit: includes a gated convolutional layer and a residual layer, used to obtain temporal information in the feature sequence.

[0145] Attention unit: It is used to extract the personal features corresponding to the feature sequence within a first preset duration based on the temporal information, and repeat the operation. The first preset duration is a short duration at the frame level. The above operation is to extract the corresponding personal features in multiple short durations respectively. On this basis, in the second preset duration, the multiple personal features corresponding to the multiple first preset durations are averaged to obtain the style vector. Generally speaking, the second preset duration is a long duration and includes the first preset duration.

[0146] The style adaptation unit, comprising a normalization layer and a fully connected layer, is used to predict the corresponding feature bias and feature gain based on the aforementioned style vector, and uses these as the style information of the speech. In traditional text-to-audio (TTS) systems, style information is determined directly from the speech. In this invention, the style information adaptively changes according to the style vector, resulting in more accurate style reproduction and less need for training samples.

[0147] The timbre conversion module is trained using pre-recorded speech from the target speaker as training sample data. The specific training process of the timbre conversion module is explained in detail below:

[0148] The loss function of the timbre conversion module consists of four parts: audio reconstruction loss, KL divergence loss, adversarial training loss, and style adaptation loss. The training process of the timbre conversion module is explained below:

[0149] Training the audio reconstruction loss:

[0150] For the input training sample speech x, obtain the corresponding Mel spectrum y. mek (This Mel spectrum refers to the actual Mel spectrum of the training sample speech x); simultaneously, the posterior encoder is based on the aforementioned Mel spectrum y. mel The implicit feature z (a black box, used to better identify the corresponding speaker) is obtained. Finally, the speech x^ is obtained through the decoding module, which is the audio of the target speaker predicted by the timbre conversion module, and then the estimated Mel spectrum y^ is obtained based on this x^. mel Based on this, the L1 loss for audio reconstruction is as follows:

[0151] l r =||y mel -y^ mel ||1

[0152] Training the KL discrete loss:

[0153] The joint encoded feature C obtained by fusing the audio feature codes (specifically including text features, prosodic features, etc.) output by the audio feature encoding module and the personal feature codes output by the style encoding module, and the implicit features z obtained by the target audio through the posterior encoder, are aligned to obtain the alignment matrix A. Then, the KL dispersion is calculated. The KL dispersion loss is:

[0154] l kl =logq θ (z|x)-logq θ (z|c,A)

[0155] It should be noted that the personal feature encoding module is configured to acquire the first personal feature of the target speech sample and determine the second personal feature of the target speech sample based on the first personal feature; wherein, the first personal feature is used to indicate the static sound characteristics of the target speech sample, and the second personal feature is used to indicate the predicted values ​​of the feature bias and gain of the first personal feature within a preset duration.

[0156] It should be noted that in the process of aligning C and z and calculating the alignment matrix A, the embodiments of this application may adopt a unidirectional and continuous method for alignment calculation. Specifically, the log-likelihood function P of C and z needs to be calculated first. For the log-likelihood function P corresponding to the i-th dimension of C and the j-th dimension of z... i，j It can be determined by the log-likelihood function P corresponding to the i-th dimension C and the (j-1)-th dimension z. i，j-1 And the log-likelihood functions P corresponding to the (i-1)th dimension C and the (j-1)th dimension z. i-1，j-1The calculation yields the alignment matrix A. Through the above iterative calculations, given the different dimensions corresponding to P, the optimal alignment path can be obtained through backtracking calculations.

[0157] It should be noted that the audio feature encoding module mentioned above will not participate in the training of the timbre conversion module after the training is completed.

[0158] Adversarial training loss:

[0159] In this embodiment, a discriminator D is added to the above network structure to participate in the training process, and the loss is calculated using the estimated G generated by the decoder and the real audio x.

[0160] l adv (D)=E x，z [(D(x)-1) 2 +D(G(z)) 2 ]

[0161] l adv (G)=E z [(D(G(z))-1) 2 ]

[0162]

[0163] Where T is the number of layers in the discriminator network, and D l To obtain N l The features of the discriminator at layer l.

[0164] The embodiments of this application can achieve two improvements in timbre conversion. First, based on a pre-trained clustering model, clustering is performed during audio feature extraction in the audio feature encoding module, improving feature quality and voice conversion effect. Second, based on clustering, improvements to the timbre conversion model further enhance the final voice conversion effect. Third, the style encoding module can be used to process the speaker's voice style, including timbre, rhythm, etc. During timbre conversion, after obtaining the style information (second person feature) of the target speech sample through the style encoding module, the training process can better target the speaker's speaking style. Then, during timbre conversion, the target audio features are fused with the second person feature to obtain joint encoded features, and further training is performed on the joint encoded features. This effectively integrates the target speaker's style information into the training process, enabling the trained model to effectively recognize the target speaker's speaking style and convert the original speech to be converted according to the target speaker's speaking style, effectively replicating the target speaker's timbre.

[0165] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described animation character voice transformation method is also provided. This electronic device can, but is not limited to, be applied in a server. For example... Figure 6 As shown, the electronic device includes a memory 602 and a processor 604. The memory 602 stores a computer program, and the processor 604 is configured to execute the steps of any of the above method embodiments via the computer program.

[0166] Optionally, in this embodiment, the electronic device may be located in at least one of a plurality of network devices in a computer network.

[0167] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0168] S1, Obtain the audio to be converted and the target voice sample of the target voice actor;

[0169] S2, the style category of the target speech sample is identified by the audio feature encoding module, and the target audio features of the target speech sample are extracted according to the style category of the target speech sample;

[0170] S3, the corresponding audio feature vector of the audio to be converted is extracted through the audio feature encoding module;

[0171] S4, obtain the first personal feature of the target speech sample through the personal feature encoding module, and determine the second personal feature of the target speech sample based on the first personal feature;

[0172] S5, the target voice actor's dubbing audio for different animated characters is obtained through the character feature encoding module as a character feature sample. The first weight and the first character feature are determined based on the similarity between the character feature vector corresponding to the character feature sample and the audio feature vector corresponding to the audio to be converted. The second weight is determined based on the phoneme similarity between the character feature vector corresponding to the first character feature and the audio feature vector corresponding to the audio to be converted. The character feature corresponding to the audio to be converted is obtained by weighted summation based on the first weight and the second weight.

[0173] S6, the audio feature vector corresponding to the audio to be converted, the target audio feature of the target speech sample, the second personal feature of the target speech sample, and the target role feature are input into the timbre conversion module. The timbre conversion module performs fusion mapping to obtain joint coding features. After standard streaming operation on the joint coding features, the timbre conversion module decodes them to obtain target speech features corresponding to the speaking style of the target voice actor. The original speech is then converted based on the target speech features to obtain the target speech.

[0174] Alternatively, as those skilled in the art will understand, Figure 6 The structure shown is for illustrative purposes only. The electronic device can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile internet device (MID), a PAD, and other terminal devices. Figure 6 This does not limit the structure of the aforementioned electronic device. For example, the electronic device may also include components that are more... Figure 6 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 6 The different configurations shown.

[0175] The memory 602 can be used to store software programs and modules, such as the program instructions / modules corresponding to the animation character voice transformation method and device in this embodiment. The processor 604 executes various functional applications and data processing by running the software programs and modules stored in the memory 602, thereby realizing the above-mentioned animation character voice transformation method. The memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 602 may further include memory remotely located relative to the processor 704, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 602 may be used, but is not limited to, to store the program steps of the animation character voice transformation method. As an example, such as Figure 6 As shown, the memory 602 may include, but is not limited to, the animation character voice transformation system, personal feature encoding module, and character feature encoding module in the aforementioned animation character voice transformation system. Furthermore, it may also include, but is not limited to, other module units in the aforementioned speech recognition neural network model, which will not be elaborated upon in this example.

[0176] Optionally, the transmission device 606 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 606 includes a Network Interface Controller (NIC), which can be connected to other network devices and routers via a network cable to communicate with the Internet or a local area network. In one example, the transmission device 706 is a Radio Frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0177] In addition, the above-mentioned electronic device also includes: a display 608 for displaying input and output information for voice conversion; and a connection bus 610 for connecting the various module components in the above-mentioned electronic device.

[0178] Optionally, the storage medium is also configured to store a computer program for performing the steps included in the methods of the above embodiments, which will not be described again in this embodiment.

[0179] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0180] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0181] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0182] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0183] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0184] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0185] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0186] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A voice transformation system for animated characters, characterized in that, The system includes: The audio feature encoding module is configured to acquire target speech samples of the target voice actor and extract target audio features of the target speech samples according to the style category of the target speech samples. The personal feature encoding module is configured to acquire the first personal feature of the target speech sample and determine the second personal feature of the target speech sample based on the first personal feature; wherein, the first personal feature is used to indicate the static sound characteristics of the target speech sample, and the second personal feature is used to indicate the predicted values ​​of the feature bias and gain of the first personal feature within a preset time period. The character feature encoding module is configured to acquire the voice-over audio of the target voice actor for different animated characters as character feature samples, determine a first weight and a first character feature based on the style similarity between the character feature vector corresponding to the character feature sample and the audio feature vector corresponding to the audio to be converted, determine a second weight by the phoneme similarity between the character feature vector corresponding to the first character feature and the audio feature vector corresponding to the audio to be converted, and obtain the target character feature corresponding to the audio to be converted by weighted summation based on the first weight and the second weight; wherein, the style similarity includes at least the similarity of timbre, pitch, and volume; the first weight is used to indicate the similarity between the character feature vector corresponding to different character feature samples and the audio feature vector corresponding to the audio to be converted; the first character feature is used to indicate the character feature sample with the highest similarity to the audio to be converted; and the second weight is used to indicate the phoneme similarity between the character feature vector corresponding to the first character feature and the audio feature vector corresponding to the audio to be converted. The timbre conversion module is configured to fuse the audio feature vector corresponding to the audio to be converted, the target audio features of the target speech sample, the second personal features, and the target character features, and convert the audio to be converted into target speech that conforms to the style of the target voice actor.

2. The animation character voice transformation system as described in claim 1, characterized in that, The timbre conversion module includes: The feature encoding unit is configured to fuse the target audio features with the second personal features and the target role features to obtain joint encoded features; and to extract implicit features based on the target audio features. The feature decoding unit is configured to generate a predicted value of the target speech sample based on the joint encoded features and the implicit features; An audio prediction unit is configured to train a first loss function based on the predicted value of the target speech sample and the true value of the target speech sample; wherein the first loss function is used to indicate the loss of the predicted value of the target speech sample relative to the true value. The alignment prediction unit is configured to align the joint encoded features with the implicit features to calculate an alignment matrix, and train a second loss function based on the alignment matrix. The second loss function is used to indicate the dispersion between the joint features and the implicit features. The discrimination unit is configured to determine the discrimination result based on the difference between the predicted value and the true value of the target speech sample, and to train a third loss function based on the discrimination result; The timbre conversion module is further configured to obtain a trained timbre conversion module based on the first loss function, the second loss function, and the third loss function, and to convert the audio to be converted into target speech that conforms to the style of the target voice actor through the trained timbre conversion module.

3. The animation character voice transformation system as described in claim 2, characterized in that, The role feature encoding module includes: The character feature extraction unit is configured to acquire the voice-over audio of the target voice actor for different animated characters as character feature samples, and extract the character feature vector corresponding to the character feature samples. The character feature encoding unit is configured to select character feature vectors from the character feature vectors corresponding to the character feature samples that meet the style similarity requirements with the audio to be converted, and encode them to obtain the character features corresponding to the audio to be converted.

4. The animation character voice transformation system as described in claim 3, characterized in that, The role feature encoding unit includes: In the first attention layer, the audio feature vector corresponding to the audio to be converted is used as the query object, and the multiple character feature vectors extracted by the character feature extraction unit are used as the query key. Based on the style similarity between the audio feature vector corresponding to the audio to be converted and the multiple character feature vectors, the first weight and the first character feature are determined. The second attention layer uses the character feature vector corresponding to the first character feature as the query object and the audio feature vector corresponding to the audio to be converted as the query key. The second weight is determined based on the phoneme similarity between the audio feature vector corresponding to the audio to be converted and the character feature vector corresponding to the first character feature.

5. The animation character voice transformation system as described in claim 2, characterized in that, The personal feature encoding module includes: The spectrum processing unit, consisting of a fully connected layer, is configured to extract the Mel spectrum features of the target speech sample and convert them into a feature sequence; The temporal processing unit includes a gated convolutional layer and a residual layer, configured to acquire the temporal information of the feature sequence; An attention unit is configured to extract the first personal features corresponding to the feature sequence within a plurality of first preset durations based on the temporal information of the feature sequence, and to average the plurality of first personal features extracted within the plurality of first preset durations within a second preset duration to obtain a style vector, wherein the second preset duration includes the plurality of first preset durations. The style adaptive unit is configured to predict the feature bias and gain of the first personal feature based on the style vector, and use the predicted feature bias and gain of the first personal feature as the second personal feature of the target speech sample.

6. The animation character voice transformation system as described in claim 5, characterized in that, The feature encoding unit is further configured to fuse the target audio features extracted by the trained audio feature encoding module with the second personal features extracted by the trained style encoding module to obtain the joint encoded features, and extract the corresponding implicit features based on the real Mel spectrum features of the target audio features. The feature decoding unit is further configured to determine the estimated Mel spectrum features corresponding to the target audio features based on the estimated value of the generated target speech sample. The audio prediction unit is further configured to train the first loss function based on the true Mel spectrum features of the target audio feature and the predicted Mel spectrum features corresponding to the target audio feature; wherein the first loss function is used to indicate the loss of the predicted Mel spectrum features corresponding to the target audio feature relative to the predicted Mel spectrum features corresponding to the target audio feature.

7. The animation character voice transformation system as described in any one of claims 2 to 6, characterized in that, The discrimination unit is further configured to perform adversarial training on the feature encoding unit, the feature decoding unit, the audio prediction unit, and the alignment prediction unit, respectively, and train the third loss function based on the results of the adversarial training.

8. A method for voice transformation of animated characters, characterized in that, include: Obtain the audio to be converted and the target voice sample of the target voice actor; The style category of the target speech sample is identified by the audio feature encoding module, and the target audio features of the target speech sample are extracted based on the style category of the target speech sample; wherein, the target audio features include the text features, prosodic features and timbre features of the target speech sample; The audio feature encoding module extracts the corresponding audio feature vector of the audio to be converted, wherein the corresponding audio feature vector of the audio to be converted includes the text features, prosodic features and timbre features of the audio to be converted; The first personal feature of the target speech sample is obtained through the personal feature encoding module, and the second personal feature of the target speech sample is determined based on the first personal feature; wherein, the first personal feature is used to indicate the static sound characteristics of the target speech sample, and the second personal feature is used to indicate the predicted values ​​of the feature bias and gain of the first personal feature within a preset time period. The target voice actor's audio recordings for different animated characters are obtained as character feature samples through a character feature encoding module. A first weight and a first character feature are determined based on the similarity between the character feature vector corresponding to the character feature sample and the audio feature vector corresponding to the audio to be converted. A second weight is determined based on the phoneme similarity between the character feature vector corresponding to the first character feature and the audio feature vector corresponding to the audio to be converted. The character feature corresponding to the audio to be converted is obtained by weighted summation of the first weight and the second weight. Specifically, the first weight indicates the similarity between the character feature vectors corresponding to different character feature samples and the audio feature vector corresponding to the audio to be converted; the first character feature indicates the character feature sample with the highest similarity to the audio to be converted; and the second weight indicates the phoneme similarity between the character feature vector corresponding to the first character feature and the audio feature vector corresponding to the audio to be converted. The audio feature vector corresponding to the audio to be converted, the target audio feature of the target speech sample, the second personal feature of the target speech sample, and the target role feature are input into the timbre conversion module. The timbre conversion module performs fusion mapping to obtain joint coding features. After standard streaming operation, the joint coding features are decoded to obtain target speech features corresponding to the speaking style of the target voice actor. The original speech is then converted based on the target speech features to obtain the target speech.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method of claim 8 at runtime.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method of claim 8.

Images