Voice conversion, model training method and device, equipment, storage medium and program

Abstract

Embodiments of the present disclosure provide a voice conversion method and device, model training method and device, storage medium and program. The method comprises: obtaining original voice with a first timbre and an identifier of a second timbre to be converted; processing the original voice to obtain semantic features corresponding to the original voice and style features of at least two levels; the style features are used to indicate expression styles corresponding to the original voice; determining timbre features corresponding to the second timbre according to the identifier of the second timbre; and generating target voice with the second timbre according to the semantic features, the style features of at least two levels and the timbre features. Through the above voice conversion process, the style features of the original voice are retained in the converted target voice, thereby improving the accuracy of the semantic information expressed by the target voice.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a speech conversion, model training method, apparatus, device, storage medium, and program. Background Technology

[0002] Speech conversion is an important aspect of artificial intelligence. It mainly studies how to convert the timbre of speech without changing the semantic content of the speech expression, that is, to convert the timbre of the current speaker into the timbre of another person.

[0003] Typically, speech conversion can be performed as follows: extract semantic features from the original speech, and then fuse these semantic features with the timbre features of the target speaker to obtain the target speech.

[0004] However, in practical applications, it has been found that the accuracy of the semantic information expressed by the target speech obtained after the above conversion is reduced, and it may not be able to accurately restore the semantic information expressed by the original speech. Summary of the Invention

[0005] This disclosure provides a speech conversion and model training method, apparatus, device, storage medium, and program to improve the accuracy of the semantic information expressed by the converted target speech.

[0006] In a first aspect, embodiments of this disclosure provide a speech conversion method, including:

[0007] Obtain the original speech with the first timbre, and the identifier of the second timbre to be converted;

[0008] The original speech is processed to obtain semantic features and at least two levels of style features corresponding to the original speech; the style features are used to indicate the expression style corresponding to the original speech.

[0009] Based on the identifier of the second timbre, determine the timbre characteristics corresponding to the second timbre;

[0010] Based on the semantic features, the style features at least two levels, and the timbre features, a target speech with the second timbre is generated.

[0011] Secondly, embodiments of this disclosure provide a method for training a speech conversion model, comprising:

[0012] Multiple sets of first training samples are obtained. Each set of first training samples includes: a first sample original speech with a first sample timbre, an identifier with a second sample timbre, and a first sample target speech with a second sample timbre. The first sample original speech and the first sample target speech express the same semantics.

[0013] The original speech of the first sample is processed by the speech conversion model to be trained to obtain the first semantic features and at least two levels of first predicted style features corresponding to the original speech of the first sample.

[0014] The speech conversion model determines the timbre features corresponding to the second sample timbre based on the identifier of the second sample timbre.

[0015] The speech conversion model generates a first predicted target speech based on the first semantic feature, the first predicted style features at least two levels, and the timbre features corresponding to the second sample timbre.

[0016] The speech conversion model is trained based on the identifiers of the first predicted target speech, the first sample original speech, the first sample target speech, and the second sample timbre; wherein the training objectives include: minimizing the style difference between the first predicted target speech and the first sample original speech, minimizing the semantic difference between the first predicted target speech and the first sample target speech, and minimizing the timbre difference between the first predicted target speech and the second sample timbre.

[0017] Thirdly, embodiments of this disclosure provide a speech conversion device, including:

[0018] The acquisition module is used to acquire the original speech with the first timbre and the identifier of the second timbre to be converted;

[0019] The feature extraction module is used to process the original speech to obtain semantic features and at least two levels of style features corresponding to the original speech; the style features are used to indicate the expression style corresponding to the original speech.

[0020] The determining module is used to determine the timbre characteristics corresponding to the second timbre based on the identifier of the second timbre;

[0021] The generation module is used to generate target speech with the second timbre based on the semantic features, the style features at least two levels, and the timbre features.

[0022] Fourthly, embodiments of this disclosure provide a training apparatus for a speech conversion model, comprising:

[0023] The acquisition module is used to acquire multiple sets of first training samples. Each set of first training samples includes: a first sample original speech with a first sample timbre, an identifier with a second sample timbre, and a first sample target speech with a second sample timbre. The first sample original speech and the first sample target speech express the same semantics.

[0024] The feature extraction module is used to process the original speech of the first sample through the speech conversion model to be trained, and obtain the first semantic features and at least two levels of first predicted style features corresponding to the original speech of the first sample.

[0025] The determination module is used to determine the timbre features corresponding to the second sample timbre based on the identifier of the second sample timbre through the speech conversion model;

[0026] The generation module is used to generate a first predicted target speech based on the first semantic feature, the first predicted style features at least two levels, and the timbre features corresponding to the second sample timbre through the speech conversion model;

[0027] The training module is used to train the speech conversion model based on the identifiers of the first predicted target speech, the first sample original speech, the first sample target speech, and the second sample timbre; wherein the training objectives include: minimizing the style difference between the first predicted target speech and the first sample original speech, minimizing the semantic difference between the first predicted target speech and the first sample target speech, and minimizing the timbre difference between the first predicted target speech and the second sample timbre.

[0028] Fifthly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0029] The memory stores computer-executed instructions;

[0030] The processor executes the computer execution instructions to implement the method as described in the first aspect, or the method as described in the second aspect.

[0031] In a sixth aspect, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the method described in the first aspect or the method described in the second aspect.

[0032] In a seventh aspect, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect or the method described in the second aspect.

[0033] The speech conversion, model training method, apparatus, device, storage medium, and program provided in this disclosure include: acquiring original speech with a first timbre and an identifier of a second timbre to be converted; processing the original speech to obtain semantic features and at least two levels of style features corresponding to the original speech; the style features are used to indicate the expression style corresponding to the original speech; determining the timbre features corresponding to the second timbre based on the identifier of the second timbre; and generating target speech with the second timbre based on the semantic features, at least two levels of style features, and timbre features. Through the above speech conversion process, the style features of the original speech are preserved in the converted target speech, thereby improving the accuracy of the semantic information expressed by the target speech. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 A schematic diagram of a speech conversion scenario provided by an embodiment of this disclosure;

[0036] Figure 2 A schematic flowchart of a speech conversion method provided in an embodiment of this disclosure;

[0037] Figure 3 A schematic diagram illustrating another speech conversion scenario provided by an embodiment of this disclosure;

[0038] Figure 4 A schematic diagram of the structure of the style feature extraction network provided in the embodiments of this disclosure;

[0039] Figure 5 A flowchart illustrating another speech conversion method provided in this embodiment of the present disclosure;

[0040] Figure 6 This is a schematic diagram of the structure of a speech conversion model provided in an embodiment of the present disclosure;

[0041] Figure 7 A flowchart illustrating a training method for a speech conversion model provided in an embodiment of this disclosure;

[0042] Figure 8 A schematic diagram illustrating the method for determining style loss values ​​provided in embodiments of this disclosure;

[0043] Figure 9 This is a schematic diagram of the structure of another speech conversion model provided in an embodiment of the present disclosure;

[0044] Figure 10 This is a schematic diagram of the structure of a speech conversion device provided in an embodiment of the present disclosure;

[0045] Figure 11 A schematic diagram of the structure of a training device for a speech conversion model provided in an embodiment of this disclosure;

[0046] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0048] To facilitate understanding, let's first combine... Figure 1 The application scenarios of the embodiments of this disclosure will be described.

[0049] Figure 1 This is a schematic diagram illustrating a speech conversion scenario provided by an embodiment of this disclosure. The application scenario includes a speech conversion device. The speech conversion device can be in the form of software and / or hardware. For example... Figure 1 As shown, assume the original speech has a first timbre. The original speech is input into a speech conversion device, which can convert the timbre of the original speech without changing its semantic content, thus obtaining the target speech. The target speech has a second timbre.

[0050] The above-described speech conversion process can be applied to various scenarios such as dubbing and entertainment. The speech conversion process can be performed in real-time or offline; this disclosure does not limit the specific method used.

[0051] Typically, speech conversion devices perform speech conversion by extracting semantic features from the original speech and fusing these semantic features with the timbre features corresponding to the second timbre to obtain the target speech. However, in practical applications, it has been found that the accuracy of the information expressed by the target speech obtained after the above conversion is reduced, and it may not be able to accurately reproduce the semantic information expressed by the original speech.

[0052] Through research on the aforementioned issues, the applicant discovered that, in addition to semantic content, the original speech also includes paralinguistic information such as emotion, intonation, stress, and intonation, which play an important role in semantic expression. In this embodiment, the paralinguistic information such as emotion, intonation, stress, and intonation in the original speech is referred to as the style features of the original speech. In the above speech conversion process, since only the semantic features of the original speech and the timbre features corresponding to the second timbre are fused, without preserving the style features of the original speech, the loss of style features leads to the loss of some semantic information. Therefore, the accuracy of the semantic information expressed by the target speech is low.

[0053] Therefore, embodiments of this disclosure provide a speech conversion method that can preserve the stylistic features of the original speech as much as possible during the speech conversion process, thereby improving the accuracy of the semantic information expressed by the converted target speech.

[0054] The technical solutions provided in this disclosure will be described in detail below with reference to several specific embodiments. These embodiments can be combined with each other, and for the same or similar concepts or processes, further elaboration may not be repeated in some embodiments.

[0055] Figure 2 This is a schematic flowchart illustrating a speech conversion method provided in an embodiment of this disclosure. Figure 2 As shown, the method in this embodiment includes:

[0056] S201: Obtain the original speech with the first timbre and the identifier of the second timbre to be converted.

[0057] For example, the original speech can be a sentence, a word, or a paragraph. In this embodiment, both the first timbre and the second timbre can be the timbre of any object. For example, the first timbre can be the timbre of virtual object 1, and the second timbre can be the timbre of virtual object 2. Optionally, the identifier of the second timbre can be the identifier of the object corresponding to the second timbre.

[0058] S202: Process the original speech to obtain the semantic features and at least two levels of style features corresponding to the original speech; the style features are used to indicate the expression style corresponding to the original speech.

[0059] In this embodiment, speech processing technology can be used to process the original speech to extract semantic and stylistic features. It should be noted that this embodiment does not limit the method used to process the original speech.

[0060] Semantic features are used to indicate the content expressed by the original speech. For example, speech recognition processing can be performed on the original speech to obtain the corresponding text content. Suppose the text content corresponding to the original speech is "The weather is really nice today", then this text content can be used as a semantic feature.

[0061] Stylistic features refer to the expressive style adopted by a speaker when reproducing original speech. Expressive style includes, but is not limited to, styles related to emotion, intonation, stress, and intonation. For example, features that characterize aspects such as emotion, intonation, stress, and intonation can be extracted from original speech as stylistic features.

[0062] In some possible implementations, the style features at the above two levels may include at least two of the following: global style features, local style features, and frame-level style features.

[0063] Among them, global-level style features are used to characterize the overall expressive style of the original speech. For example, assuming the original speech has a duration of 5 seconds, the entire duration of the original speech is considered as a whole, and the characteristics of emotion, intonation, stress, and fluctuation presented by this whole are the global-level style features.

[0064] Local-level style features are used to characterize the expressive style presented by different speech segments in the original speech. For example, assuming the original speech has a duration of 5 seconds, it can be divided into multiple speech segments in the time domain. The characteristics of each speech segment, such as emotion, intonation, stress, and fluctuation, are the local-level style features. It should be noted that this embodiment does not limit the method of dividing speech segments. For example, speech segments can be divided according to a preset duration, or each speech unit can be divided into a speech segment.

[0065] Frame-level style features are used to characterize the expressive style presented by different speech frames in the original speech. For example, assuming the original speech has a duration of 5 seconds, it can be divided into frames in the time domain to obtain multiple speech frames. The features of each speech frame, such as emotion, intonation, stress, and intonation, are the frame-level style features. It should be understood that the duration of a speech frame is shorter than the duration of a speech segment.

[0066] By characterizing the style features of the original speech from at least two of the three levels—global, local, and frame—the accuracy and comprehensiveness of the style features of the original speech are improved.

[0067] It should be understood that the aforementioned global-level style features, local-level style features, and frame-level style features are divided according to the temporal granularity of style presentation. These three levels of style features can be referred to as multi-scale style features. In practical applications, other levels of division can also be used.

[0068] S203: Determine the timbre characteristics corresponding to the second timbre based on the identifier of the second timbre.

[0069] For example, multiple timbres can be pre-learned to obtain timbre features for each timbre, and these timbre features can be stored. In a speech conversion scenario, based on the identifier of the second timbre to be converted, the timbre features corresponding to the second timbre are retrieved from the pre-stored timbre features.

[0070] It should be noted that this disclosure does not limit the storage method for timbre features corresponding to multiple timbres. For example, they can be stored in a database or in a pre-trained speech conversion model. In the latter case, the timbre features corresponding to each timbre can be determined by the parameters in the speech conversion model.

[0071] S204: Generate target speech with a second timbre based on semantic features, style features at least two levels, and timbre features.

[0072] In this embodiment, semantic features, at least two levels of style features, and timbre features can be fused to generate target speech. It should be understood that since the semantic features and at least two levels of style features are derived from the original speech, the target speech retains the semantic and style features of the original speech; while the timbre features represent the characteristics of a second timbre, therefore, the target speech has a second timbre.

[0073] Therefore, compared with the aforementioned speech conversion process, the target speech obtained in this embodiment not only retains the semantic features of the original speech but also its stylistic features. Stylistic features play an auxiliary role in semantic expression; by preserving the stylistic features of the original speech during the speech conversion process, the accuracy of the semantic information expressed by the converted target speech is improved.

[0074] The speech conversion method provided in this embodiment includes: acquiring original speech with a first timbre and an identifier of a second timbre to be converted; processing the original speech to obtain semantic features and at least two levels of style features corresponding to the original speech; the style features are used to indicate the expression style corresponding to the original speech; determining the timbre features corresponding to the second timbre based on the identifier of the second timbre; and generating target speech with the second timbre based on the semantic features, at least two levels of style features, and timbre features. Through the above speech conversion process, the style features of the original speech are preserved in the converted target speech, thereby improving the accuracy of the semantic information expressed by the target speech.

[0075] based on Figure 2 The illustrated embodiment, in some possible implementations, Figure 2 The speech conversion process in the illustrated embodiment can also be implemented by a speech conversion model. The speech conversion model can be a machine learning model trained based on a machine learning algorithm. The following section will discuss this further. Figure 3 Please provide an explanation.

[0076] Figure 3 This is a schematic diagram illustrating another speech conversion scenario provided by an embodiment of this disclosure. For example... Figure 3 As shown, this scenario includes a model training phase and a model usage phase. In the model training phase, a speech conversion model can be obtained by training multiple sets of training samples. Each set of training samples includes: original speech samples with a first sample timbre, a marker for a second sample timbre, and target speech samples with the second sample timbre. The original speech samples and the target speech samples express the same semantics. In this way, the trained speech conversion model can convert the timbre of the original speech to obtain the target speech without changing the semantic information and speaking style of the original speech.

[0077] During the model usage phase, see Figure 3 The original speech with a first timbre and an identifier for a second timbre are input into a speech conversion model. The speech conversion model includes a semantic feature extraction network, a style feature extraction network, and a timbre conversion network. The semantic feature extraction network processes the original speech to obtain semantic features corresponding to the original speech. The style feature extraction network processes the original speech to obtain at least two levels of style features corresponding to the original speech. The timbre conversion network determines the timbre features corresponding to the second timbre based on the identifier for the second timbre, and generates target speech with the second timbre based on the semantic features, the at least two levels of style features, and the timbre features.

[0078] One possible implementation is to deploy a pre-trained Automatic Speech Recognition (ASR) model in a semantic feature extraction network, and then use the ASR model to perform speech recognition processing on the original speech to obtain the semantic features corresponding to the original speech.

[0079] The following is combined with Figure 4 The structure of the style feature extraction grid and the method of style feature extraction are explained in detail.

[0080] Figure 4 This is a schematic diagram of the structure of a style feature extraction network provided in an embodiment of this disclosure. Figure 4 As shown, the style feature extraction grid includes three branch networks, which are used to extract style features at the global level, local level, and frame level, respectively.

[0081] See Figure 4 The first branch network consists of a pre-trained self-supervised learning model and a first reference encoder. The self-supervised learning model can be a VQwav2vec model. The first reference encoder can be a classic modular reference encoder. In one example, the first reference encoder may include six 2D convolutions and a gated recurrent unit (GRU). The last state vector of the GRU can be used as a global-level style feature.

[0082] Based on this branch network, global-level style features can be extracted as follows: Features are extracted from the original speech using a pre-trained self-supervised learning model to obtain the self-supervised learning (SSL) features corresponding to each speech frame in the original speech. See also... Figure 4 Assuming the original speech contains T speech frames, the self-supervised learning model outputs T self-supervised learning features for each speech frame, with each self-supervised learning feature having a dimension of D. SSL Furthermore, the self-supervised learning features corresponding to each speech frame are encoded using a first reference encoder to obtain a dimension D. glo The feature is the style feature at the global level.

[0083] See also Figure 4The second branch network includes an Automatic Speech Recognition (ASR) model and a second reference encoder. The second reference encoder can be obtained by adjusting a classic modular reference encoder and can extract hidden layer representations of the original speech. In one example, the second reference encoder includes 6 layers of 2D convolutions and a unidirectional GRU with a stride of 2*1, the output of which can be used as frame-level hidden representations.

[0084] Based on this branch network, local-level style features can be extracted as follows: Feature extraction is performed on the original speech using a pre-trained ASR model to obtain bottleneck features (BN) corresponding to each speech frame in the original speech. See also... Figure 4 Assuming the original speech contains T speech frames, the ASR model outputs BN features corresponding to each of the T speech frames, with each BN feature having a dimension of D. BN The bottleneck features corresponding to each speech frame are encoded using a second reference encoder to obtain the hidden features of each speech frame. The dimension of the hidden features of each speech frame is D. Seg Furthermore, the local-level style features are obtained by downsampling and upsampling the hidden features of each speech frame. For example, the hidden features can be segmented according to a window size of γ=16 to obtain multiple segments. Downsampling is performed on each segment; for example, the mean of the hidden features of each speech frame in that segment is used as the downsampled hidden feature. This results in T / γ hidden features after downsampling. Then, the T / γ hidden features are upsampled to obtain T hidden features. These T upsampled hidden features serve as the local-level style features.

[0085] See also Figure 4 The third branch network includes a linear mapping layer. Based on this branch network, frame-level style features can be extracted as follows: The fundamental frequency features, articulation features, and energy features corresponding to each speech frame in the original speech are obtained explicitly. Specifically, the fundamental frequency features characterize the logarithmic domain fundamental frequency (LF0) of the corresponding speech frame, the articulation features characterize whether the corresponding speech frame is audible or unvoiced (VUV), and the energy features characterize the average amplitude spectrum of the corresponding speech frame. For example... Figure 4 As shown, assume the original speech contains T speech frames, and the dimension of the fundamental frequency feature corresponding to each speech frame is D. LF0 The dimension of the pronunciation features corresponding to each speech frame is D. VUV The dimension of the energy feature corresponding to each speech frame is D. EFurthermore, the fundamental frequency features, articulation features, and energy features mentioned above are linearly processed through a linear mapping layer, and the linearly processed fundamental frequency features, articulation features, and energy features are then dimensionally concatenated to obtain frame-level style features.

[0086] It should be noted that, Figure 4 The example illustrates the structure of a style feature extraction mesh for obtaining style features at three scales: global, local, and frame. When only two scales of style features are needed, the style feature extraction network can include two branch networks from each scale.

[0087] In this embodiment, by extracting style features at three different scales—global, local, and frame—the accuracy and comprehensiveness of the style features of the original speech are improved.

[0088] Based on any of the above embodiments, the following is combined with Figure 5 and Figure 6 The speech conversion process is explained in detail.

[0089] Figure 5 A flowchart illustrating another speech conversion method provided in this disclosure embodiment is shown below. Figure 5 As shown, the method in this embodiment includes:

[0090] S501: Obtain the original speech with the first timbre and the identifier of the second timbre to be converted.

[0091] S502: Process the original speech to obtain the semantic features and style features at the following three levels: global style features, local style features, and frame style features.

[0092] It should be understood that the extraction methods for semantic and style features can be found in the detailed description of the foregoing embodiments, and will not be repeated here.

[0093] To facilitate understanding, the following will be combined with... Figure 6 Please provide an explanation. Figure 6 This is a schematic diagram of the structure of a speech conversion model provided in an embodiment of this disclosure. Figure 6 As shown, by inputting the original speech into a semantic feature extraction network, the semantic features corresponding to the original speech can be obtained. By inputting the original speech into a style feature extraction network, the style features corresponding to the original speech at three levels can be obtained.

[0094] S503: Determine the timbre characteristics corresponding to the second timbre based on the identifier of the second timbre.

[0095] Combination Figure 6The timbre conversion network includes a timbre feature acquisition unit. The identifier of the second timbre is input into the timbre feature acquisition unit, which can then acquire the corresponding timbre features based on the identifier.

[0096] S504: The semantic features and timbre features are superimposed to obtain the first fusion feature.

[0097] The overlay process can be understood as an addition operation. Since the semantic features and timbre features have the same dimensions, the values ​​of the corresponding dimensions in the semantic features and timbre features are added together to obtain the first fused feature.

[0098] Thus, the first fusion feature includes semantic features from the original speech, as well as timbre features corresponding to the second timbre. Furthermore, the first fusion feature and the aforementioned three levels of style features can be encoded and decoded to obtain the target Mel spectrum feature.

[0099] Optionally, the first fusion feature and the style features of the above three levels are encoded and decoded to obtain the target Mel spectrum features, which can be done using methods S505 to S508.

[0100] S505: The first fusion feature and the style features at the local level are dimensionally concatenated to obtain the second fusion feature.

[0101] For example, assuming the first fused feature has a dimension of M and the local-level style features have a dimension of N, then the second fused feature obtained after dimension concatenation will have a dimension of M+N. Thus, the second fused feature includes semantic features, timbre features, and local-level style features.

[0102] S506: Encode the second fusion feature to obtain the encoded feature.

[0103] Combination Figure 6 The timbre conversion network includes an encoder, which can employ a Conformer model, a type of Transformer model based on convolutional enhancement.

[0104] S507: The coding features, global-level style features, and frame-level style features are concatenated dimensionally to obtain the third fusion feature.

[0105] Thus, the third fusion feature includes not only semantic features and timbre features, but also local-level style features, global-level style features, and frame-level style features.

[0106] In one possible implementation, S507 can be implemented as follows: See Figure 6The encoded features and timbre features output by the encoder are superimposed to obtain the fourth fusion feature. The fourth fusion feature, the global-level style features, and the frame-level style features are then concatenated dimensionally to obtain the third fusion feature. By superimposing timbre features on top of the encoded features, the timbre features are enhanced, making the timbre of the converted target speech closer to the second timbre.

[0107] S508: Decode the third fusion feature to obtain the target Mel spectrum feature.

[0108] Combination Figure 6 The timbre conversion model also includes a decoder, which inputs the third fusion feature into the decoder for decoding to obtain the target Mel spectrum feature.

[0109] Optionally, the first fusion feature and the style features of the above three levels are encoded and decoded to obtain the target Mel spectrum feature. Alternatively, this can be achieved as follows: the first fusion feature is encoded to obtain an encoded feature; the encoded feature, the global-level style feature, the local-level style feature, and the frame-level style feature are then concatenated dimensionally to obtain the fifth fusion feature. The fifth fusion feature is then decoded to obtain the target Mel spectrum feature.

[0110] S509: Generate target speech with a second timbre based on the target Mel spectrum features.

[0111] Combination Figure 6 The speech conversion model can also include a vocoder. For example, the vocoder can be implemented using a linear predictive coding (LPC) network. The target Mel-spectral features output from the decoder are input into the vocoder, which then reconstructs the waveform from these features to obtain the target speech.

[0112] In this embodiment, based on the multi-scale nature of speech style, global, local, and frame-level style features can be extracted from the original speech. Appropriate features are selected for each level, improving the accuracy and comprehensiveness of the style features. Furthermore, the extracted style features from multiple levels are used for speech conversion, ensuring that the converted target speech retains as many of the original speech's style features as possible, thereby improving the accuracy of the semantic information expressed by the target speech.

[0113] The above embodiments describe how to convert and process original speech using a speech conversion model. The training process of the speech conversion model will be introduced below with reference to specific embodiments.

[0114] Figure 7This is a flowchart illustrating a training method for a speech conversion model provided in an embodiment of this disclosure. The method in this embodiment can be executed by a training device. Figure 7 As shown, the method in this embodiment includes:

[0115] S701: Obtain multiple sets of first training samples. Each set of first training samples includes: a first sample original speech with a first sample timbre, an identifier with a second sample timbre, and a first sample target speech with a second sample timbre. The first sample original speech and the first sample target speech express the same semantics.

[0116] For example, the first sample original speech and the first sample target speech are the speech outputs of the first object and the second object, respectively, for the same text content. For instance, the first sample original speech is the speech generated when the first object says "The weather is so nice today," and the first sample target speech is the speech generated when the second object says "The weather is so nice today." Both express the same semantics; the first sample original speech has the timbre of the first object (i.e., the first timbre), and the first sample target speech has the timbre of the second object (i.e., the second timbre).

[0117] S702: The original speech of the first sample is processed by the speech conversion model to be trained to obtain the first semantic feature and at least two levels of first predicted style features corresponding to the original speech of the first sample. The timbre features corresponding to the timbre of the second sample are determined according to the identifier of the timbre of the second sample. The first predicted target speech is generated according to the first semantic feature, the first predicted style features at least two levels, and the timbre features corresponding to the timbre of the second sample.

[0118] Optionally, the first predicted style features at least two levels include one or more of the following: global style features, local style features, and frame-level style features. Specifically, global style features characterize the overall expressive style of the first sample original speech; local style features characterize the expressive style of different speech segments of the first sample original speech; and frame-level style features characterize the expressive style of different speech frames within the first sample original speech.

[0119] In other words, the identifiers of the first sample original speech and the second sample timbre are input into the speech conversion model, and the speech conversion model outputs the first predicted target speech. It should be understood that the processing method of the speech conversion model for the identifiers of the first sample original speech and the second sample timbre can be found in [reference needed]. Figures 4 to 6 The detailed description of the illustrated embodiments is omitted here.

[0120] S703: The speech conversion model is trained based on the identifiers of the first predicted target speech, the first sample original speech, the first sample target speech, and the second sample timbre; wherein, the training objectives include: minimizing the style difference between the first predicted target speech and the first sample original speech, minimizing the semantic difference between the first predicted target speech and the first sample target speech, and minimizing the timbre difference between the first predicted target speech and the second sample timbre.

[0121] In this embodiment, by setting the training objectives as "minimizing the stylistic differences between the first predicted target speech and the first sample original speech, minimizing the semantic differences between the first predicted target speech and the first sample target speech, and minimizing the timbre differences between the first predicted target speech and the second sample timbre," the trained speech conversion model possesses the following capabilities: during speech conversion, it preserves the stylistic features of the original speech as much as possible and makes the timbre of the converted target speech as close as possible to the timbre of the second sample. Thus, when the trained speech conversion model is used to convert the original speech, the accuracy of the semantic information expressed by the converted target speech can be improved.

[0122] In one possible implementation, the speech conversion model can be trained in S703 in the following way:

[0123] (1) Determine the style loss value based on the first predicted target speech and the first sample original speech.

[0124] The style loss value is used to indicate the style difference between the first predicted target speech and the original semantics of the first sample. One possible implementation is to determine the style loss value as follows: process the Mel-spectral features of the first predicted target speech using a pre-trained style classification model to obtain first intermediate features output by each network unit in the style classification model; process the Mel-spectral features of the original speech of the first sample using the style classification model to obtain second intermediate features output by each network unit in the style classification model; and determine the style loss value based on the first intermediate features and the second intermediate features.

[0125] To facilitate understanding, the following will be combined with... Figure 8 Let's illustrate with examples. Figure 8 This is a schematic diagram illustrating the method for determining style loss values ​​provided in embodiments of this disclosure. For example... Figure 8 As shown, a style classification model can be used, which is a pre-trained machine learning model. Assume the style classification model consists of multiple network units, designated as network unit 1 through network unit 4. See also... Figure 8The Mel spectrum features of the first predicted target speech are input into the style classification model. During the processing of the Mel spectrum features of the first predicted target speech by the style classification model, network units 1 to 3 output the first intermediate feature 1 (let's call it h1), the first intermediate feature 2 (let's call it h2), and the first intermediate feature 3 (let's call it h3), respectively. Similarly, the Mel spectrum features of the first sample original speech are input into the style classification model. During the processing of the Mel spectrum features of the first sample original speech by the style classification model, network units 1 to 3 output the second intermediate feature 1 (let's call it h3). ), second intermediate feature 2 (let's assume it's denoted as ) ), second intermediate feature 3 (let's assume it's denoted as ) ).

[0126] Based on the first intermediate feature 1 and the second intermediate feature 1, the intermediate loss value 1 (let's call it L) can be obtained. styie1 Based on the first intermediate feature 2 and the second intermediate feature 2, the intermediate loss value 2 (let's assume it's denoted as L) can be obtained. styie2 Based on the first intermediate feature 3 and the second intermediate feature 3, the intermediate loss value 3 (let's assume it's denoted as L) can be obtained. style3 For example,

[0127]

[0128] Then, the style loss value is obtained based on the intermediate loss value 1, intermediate loss value 2, and intermediate loss value 3. For example, let the style loss value be denoted as L. styie ,but:

[0129] L styie =L styie1 +L styie2 +L styie3

[0130] (2) Determine the semantic loss value based on the first predicted target speech and the first sample target speech.

[0131] The semantic loss value is used to indicate the semantic difference between the first predicted target speech and the first sample target speech. The semantic loss value can also be called the reconstruction loss value.

[0132] In one possible implementation, the semantic loss value can be determined as follows: The semantic loss value is determined based on the Mel-spectral features of the first predicted target speech and the Mel-spectral features of the first sample target speech. For example, suppose the Mel-spectral features of the first predicted target speech are denoted as... The Mel-spectral features of the first sample target speech are denoted as Y, and can be... The mean squared error (MSE) between Y and Y is used as the semantic loss value. Let the semantic loss value be denoted as L. recons ,but:

[0133]

[0134] (3) Determine the timbre loss value based on the identifiers of the first predicted target speech and the second sample timbre.

[0135] The timbre loss value is used to indicate the difference between the timbre of the first predicted target speech and the timbre of the second sample speech.

[0136] In one possible implementation, the timbre loss value can be determined as follows: the Mel spectrum features of the first predicted target speech are processed by a timbre classification model to obtain a first probability, which indicates the probability that the timbre identifier of the first predicted target speech matches the identifier of the second sample timbre; the timbre loss value is determined based on the first probability.

[0137] The timbre classification model can be used to identify the timbre identifier / timbre category / speaker corresponding to speech. For example, the Mel-spectral features of the first predicted target speech are input into the timbre classification model, and the model outputs the probability that the timbre identifier corresponding to the first predicted target speech is an identifier of multiple sample timbres. Let the Mel-spectral features of the first predicted target speech be denoted as... The timbre classification model outputs the probability corresponding to the identifier s1 of sample timbre 1. The probability corresponding to the identifier s2 of sample timbre 2 ..., s n Corresponding probability The timbre loss L can be determined using the following formula. speaker :

[0138]

[0139] Where 's' represents the identifier of the second sample timbre.

[0140] (3) Determine the target loss value based on the style loss value, semantic loss value and timbre loss value.

[0141] For example, suppose the target loss value is denoted as L. total ,but:

[0142] L total =L recons +L speaker +L style =L recons +L speaker +L style1 +L styie2+L style3

[0143] (4) The speech conversion model is trained with the goal of minimizing the target loss value.

[0144] In other words, the training objective is to minimize the aforementioned target loss value, update the parameters of the speech conversion model, and repeat the above training process until the speech conversion model reaches the convergence condition and the training ends.

[0145] Figure 9 This is a schematic diagram illustrating the structure of yet another speech conversion model provided in an embodiment of this disclosure. For example... Figure 9 As shown, in Figure 6 Based on the speech conversion model shown, the speech conversion model in this embodiment further includes a constraint network, which includes a style classification model and a timbre classification model. It should be understood that the constraint network is only used during model training.

[0146] based on Figure 9 The training process of the speech conversion model shown is as follows: The original first sample speech and the identifiers of the second sample timbre are input into the speech conversion model. The original first sample speech is processed by a semantic feature extraction network to obtain the first semantic feature corresponding to the original first sample speech. The original first sample speech is then processed by a style feature extraction grid to obtain the following three levels of style features corresponding to the original first sample speech: global style features, local style features, and frame-level style features. The timbre feature acquisition unit in the timbre conversion network determines the timbre feature corresponding to the second sample timbre based on the identifier of the second sample timbre, and generates the first predicted Mel spectrum feature based on the first semantic feature, the first predicted style feature at the above three levels, and the timbre feature corresponding to the second sample timbre.

[0147] See also Figure 9 In the constrained network, the first predicted Mel-spectral features and the Mel-spectral features of the first sample original speech are input into the style classification model. The style loss value is obtained based on the intermediate features generated during the processing of these two features by the style classification model. The first predicted Mel-spectral features are input into the timbre classification model, and the timbre loss value is determined based on the classification result and the timbre identifier of the second sample. The semantic loss value is determined based on the first predicted Mel-spectral features and the Mel-spectral features of the first sample target speech. Further, a target loss value is determined based on the style loss, semantic loss, and timbre loss values, and the parameters of the speech conversion model are updated with minimizing the target loss value as the training objective. This training process is repeated until the speech conversion model reaches convergence.

[0148] In the aforementioned training process, multiple sets of first training samples are required. Each first training sample needs to include two sample speech with different timbres but the same semantics (i.e., the original first sample speech and the target first sample speech). However, in practical applications, obtaining these first training samples is quite difficult. Therefore, this disclosure also provides a phased training method, including a first training phase and a second training phase.

[0149] First training phase and Figures 7 to 9 The training process shown is the same. The difference is that the timbre of the first sample and the timbre of the second sample in each group of first training samples are the same. That is to say, the first sample original speech and the first sample target speech included in each group of first training samples are the same speech. This reduces the difficulty of obtaining the first training samples.

[0150] After the first training phase, the speech conversion model has the ability to reconstruct the target speech from the original speech, while preserving the stylistic features of the original speech in the target speech.

[0151] In the second training phase, multiple sets of second training samples are acquired. Each set of second training samples includes: original second sample speech with a third sample timbre and an identifier for a fourth sample timbre; the third sample timbre and the fourth sample timbre are different; the original second sample speech is processed by a speech conversion model to obtain the second semantic features and at least two levels of second predicted style features corresponding to the original second sample speech; the timbre features corresponding to the fourth sample timbre are determined based on the identifier of the fourth sample timbre; and the second predicted target speech is generated based on the second semantic features, at least two levels of second predicted style features, and the timbre features corresponding to the fourth sample timbre; the parameters of the speech conversion model are updated based on the second predicted target speech, the original second sample speech, and the identifier of the fourth sample timbre. The update objectives include: minimizing the style difference between the second predicted target speech and the original second sample speech, and minimizing the timbre difference between the second predicted target speech and the fourth sample timbre.

[0152] Specifically, the style loss value is determined based on the second predicted target speech and the second sample original speech. It should be understood that the process for determining the style loss value can be found in [link to documentation]. Figure 7 The detailed description of the illustrated embodiment is omitted here. In some possible implementations, considering that the intermediate features extracted by network unit 1 in the style classification model are strongly correlated with timbre, the intermediate loss value 1 (L) may be disregarded when determining the style loss value in the second training phase. style1 The remaining processes are the same as... Figure 7 Similarly, the style loss value L style as follows:

[0153] L style =Lstyle2 +L style3

[0154] The timbre loss value is determined based on the identifiers of the second predicted target speech and the fourth sample timbre. The specific calculation method is the same as... Figure 7 The illustrated embodiment is similar and will not be described in detail here. That is, the timbre loss value L... speaker as follows:

[0155]

[0156] Where 's' represents the identifier of the fourth sample timbre.

[0157] Based on the style loss value L style and tone loss value L speaker Determine the target loss value L totai ,as follows:

[0158] L totai =L speaker +L style =L speaker +L styie2 +L style3

[0159] By minimizing the aforementioned target loss value, the model parameters of the speech conversion model are updated to fine-tune the model. This enhances the timbre conversion function of the speech conversion model.

[0160] In some possible implementations, the first and second training phases described above can be performed alternately.

[0161] In this embodiment, by adopting a phased training approach, the problem of difficulty in obtaining training samples is avoided, thereby improving the training efficiency of the speech conversion model.

[0162] Figure 10 This is a schematic diagram of a speech conversion device provided in an embodiment of this disclosure. The device can be in software and / or hardware form. Figure 10 As shown, the speech conversion device 1000 provided in this embodiment includes: an acquisition module 1001, a feature extraction module 1002, a determination module 1003, and a generation module 1004. Wherein,

[0163] The acquisition module 1001 is used to acquire the original speech with the first timbre and the identifier of the second timbre to be converted;

[0164] The feature extraction module 1002 is used to process the original speech to obtain semantic features and at least two levels of style features corresponding to the original speech; the style features are used to indicate the expression style corresponding to the original speech.

[0165] The determining module 1003 is used to determine the timbre characteristics corresponding to the second timbre based on the identifier of the second timbre;

[0166] The generation module 1004 is used to generate target speech with the second timbre based on the semantic features, the style features at least two levels, and the timbre features.

[0167] In one possible implementation, the at least two levels of style features include at least two of the following:

[0168] Global-level style features are used to characterize the overall expressive style of the original speech;

[0169] Local-level style features are used to characterize the expressive style presented by different speech segments of the original speech;

[0170] Frame-level style features are used to characterize the expressive style presented by different speech frames in the original speech.

[0171] In one possible implementation, the generation module 1004 is specifically used for:

[0172] The semantic features and the timbre features are superimposed to obtain the first fusion feature;

[0173] The first fused feature and the style features of at least two levels are encoded and decoded to obtain the target Mel spectrum features;

[0174] The target speech is generated based on the target Mel spectrum features.

[0175] In one possible implementation, the at least two levels of style features include: global-level style features, local-level style features, and frame-level style features; the generation module 1004 is specifically used for:

[0176] The first fusion feature and the style feature of the local level are dimensionally concatenated to obtain the second fusion feature;

[0177] The second fused feature is encoded to obtain the encoded feature;

[0178] The encoding features, the global-level style features, and the frame-level style features are concatenated dimensionally to obtain the third fusion feature;

[0179] The third fusion feature is decoded to obtain the target Mel spectrum feature.

[0180] In one possible implementation, the generation module 1004 is specifically used for:

[0181] The encoding feature and the timbre feature are superimposed to obtain the fourth fusion feature;

[0182] The third fusion feature is obtained by dimensionally concatenating the fourth fusion feature, the global-level style feature, and the frame-level style feature.

[0183] In one possible implementation, the feature extraction module 1002 is specifically used to perform at least two of the following:

[0184] The original speech is feature extracted by a pre-trained self-supervised learning model to obtain the self-supervised learning features corresponding to each speech frame in the original speech. The self-supervised learning features corresponding to each speech frame are encoded by a first reference encoder to obtain the global-level style features.

[0185] The original speech is feature extracted by a pre-trained speech recognition model to obtain the bottleneck features corresponding to each speech frame in the original speech. The bottleneck features corresponding to each speech frame are encoded by a second reference encoder to obtain the hidden layer features of each speech frame. The hidden layer features of each speech frame are downsampled and upsampled to obtain the style features of the local level.

[0186] Obtain the fundamental frequency features, articulation features, and energy features corresponding to each speech frame in the original speech, and determine the style features at the frame level based on the fundamental frequency features, articulation features, and energy features corresponding to each speech frame.

[0187] In one possible implementation, the feature extraction module 1002 is specifically used to: process the original speech through a trained speech conversion model to obtain the semantic features and at least two levels of style features corresponding to the original speech;

[0188] The determining module 1003 is specifically used to: determine the timbre features corresponding to the second timbre based on the identifier of the second timbre through the speech conversion model;

[0189] The generation module 1004 is specifically used to: generate target speech with the second timbre by means of the speech conversion model based on the semantic features, the style features at least two levels and the timbre features;

[0190] The speech conversion model is trained using multiple sets of training samples. Each set of training samples includes: original speech samples with a first sample timbre, an identifier for a second sample timbre, and target speech samples with the second sample timbre. The original speech samples and the target speech samples express the same semantics.

[0191] The apparatus provided in this embodiment can be used to execute the speech conversion method provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0192] Figure 11 This is a schematic diagram of a training device for a speech conversion model provided in an embodiment of this disclosure. The device can be in software and / or hardware form. Figure 11 As shown, the speech conversion model training device 1100 provided in this embodiment includes: an acquisition module 1101, a feature extraction module 1102, a determination module 1103, a generation module 1104, and a training module 1105. Among them,

[0193] The acquisition module 1101 is used to acquire multiple sets of first training samples. Each set of first training samples includes: a first sample original speech with a first sample timbre, an identifier with a second sample timbre, and a first sample target speech with a second sample timbre. The first sample original speech and the first sample target speech express the same semantics.

[0194] The feature extraction module 1102 is used to process the original speech of the first sample through the speech conversion model to be trained, and obtain the first semantic features and at least two levels of first predicted style features corresponding to the original speech of the first sample.

[0195] The determining module 1103 is used to determine the timbre features corresponding to the second sample timbre based on the identifier of the second sample timbre through the speech conversion model;

[0196] The generation module 1104 is used to generate a first predicted target speech based on the first semantic feature, the first predicted style features at least two levels and the timbre features corresponding to the second sample timbre through the speech conversion model;

[0197] The training module 1105 is used to train the speech conversion model based on the identifiers of the first predicted target speech, the first sample original speech, the first sample target speech, and the second sample timbre; wherein the training objectives include: minimizing the style difference between the first predicted target speech and the first sample original speech, minimizing the semantic difference between the first predicted target speech and the first sample target speech, and minimizing the timbre difference between the timbre of the first predicted target speech and the second sample timbre.

[0198] In one possible implementation, the training module 1105 is specifically used for:

[0199] Determine the style loss value based on the first predicted target speech and the first sample original speech;

[0200] Determine the semantic loss value based on the first predicted target speech and the first sample target speech;

[0201] Based on the identifiers of the first predicted target speech and the second sample timbre, determine the timbre loss value;

[0202] The target loss value is determined based on the style loss value, the semantic loss value, and the timbre loss value;

[0203] The speech conversion model is trained with the goal of minimizing the target loss value.

[0204] In one possible implementation, the training module 1105 is specifically used for:

[0205] The Mel spectrum features of the first predicted target speech are processed by a pre-trained style classification model to obtain the first intermediate features output by each network unit in the style classification model.

[0206] The style classification model is used to process the Mel spectrum features of the original speech of the first sample to obtain the second intermediate features output by each network unit in the style classification model.

[0207] The style loss value is determined based on the first intermediate feature and the second intermediate feature.

[0208] In one possible implementation, the training module 1105 is specifically used for:

[0209] The semantic loss value is determined based on the Mel spectrum features of the first predicted target speech and the Mel spectrum features of the first sample target speech.

[0210] In one possible implementation, the training module 1105 is specifically used for:

[0211] The Mel spectrum features of the first predicted target speech are processed by a timbre classification model to obtain a first probability. The first probability is used to indicate the probability that the timbre identifier corresponding to the first predicted target speech is a match of the identifier of the second sample timbre.

[0212] The timbre loss value is determined based on the first probability.

[0213] In one possible implementation, the first sample timbre and the second sample timbre are the same timbre; after the speech conversion model reaches a preset convergence condition, the acquisition module 1101 is further configured to: acquire multiple sets of second training samples, each set of second training samples including: the original speech of the second sample with the third sample timbre, and the identifier of the fourth sample timbre; the third sample timbre and the fourth sample timbre are different;

[0214] The feature extraction module 1102 is further configured to: process the original speech of the second sample through the speech conversion model to obtain the second semantic features and at least two levels of second predicted style features corresponding to the original speech of the second sample;

[0215] The determining module 1103 is further configured to: determine the timbre feature corresponding to the fourth sample timbre based on the identifier of the fourth sample timbre;

[0216] The generation module 1104 is further configured to: generate a second predicted target speech based on the second semantic feature, the second predicted style features at least two levels, and the timbre features corresponding to the fourth sample timbre;

[0217] The training module 1105 is further configured to: update the parameters of the speech conversion model based on the identifiers of the second predicted target speech, the second sample original speech, and the fourth sample timbre, wherein the update objectives include: minimizing the style difference between the second predicted target speech and the second sample original speech, and minimizing the timbre difference between the second predicted target speech and the fourth sample timbre.

[0218] In one possible implementation, the first predictive style features of the at least two levels include at least two of the following:

[0219] Global-level style features are used to characterize the overall expressive style of the original speech of the first sample;

[0220] Local-level style features are used to characterize the expressive style presented by different speech segments of the original speech of the first sample;

[0221] Frame-level style features are used to characterize the expressive style presented by different speech frames in the original speech of the first sample.

[0222] The speech conversion model training device provided in this embodiment can be used to execute the speech conversion model training method provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0223] To implement the above embodiments, this disclosure also provides an electronic device.

[0224] refer to Figure 12The diagram illustrates a structural schematic of an electronic device 1200 suitable for implementing embodiments of the present disclosure. The electronic device 1200 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 12 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0225] like Figure 12 As shown, the electronic device 1200 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage device 1208 into a random access memory (RAM) 1203. The RAM 1203 also stores various programs and data required for the operation of the electronic device 1200. The processing unit 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0226] Typically, the following devices can be connected to I / O interface 1205: input devices 1206 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1207 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1208 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1209. Communication device 1209 allows electronic device 1200 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 12 An electronic device 1200 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0227] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1209, or installed from storage device 1208, or installed from ROM 1202. When the computer program is executed by processing device 1201, it performs the functions defined in the methods of embodiments of this disclosure.

[0228] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0229] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0230] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0231] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0232] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0233] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0234] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0235] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0236] In a first aspect, according to one or more embodiments of this disclosure, a speech conversion method is provided, comprising:

[0237] Obtain the original speech with the first timbre, and the identifier of the second timbre to be converted;

[0238] The original speech is processed to obtain semantic features and at least two levels of style features corresponding to the original speech; the style features are used to indicate the expression style corresponding to the original speech.

[0239] Based on the identifier of the second timbre, determine the timbre characteristics corresponding to the second timbre;

[0240] Based on the semantic features, the style features at least two levels, and the timbre features, a target speech with the second timbre is generated.

[0241] According to one or more embodiments of this disclosure, the at least two levels of style features include at least two of the following:

[0242] Global-level style features are used to characterize the overall expressive style of the original speech;

[0243] Local-level style features are used to characterize the expressive style presented by different speech segments of the original speech;

[0244] Frame-level style features are used to characterize the expressive style presented by different speech frames in the original speech.

[0245] According to one or more embodiments of this disclosure, generating target speech having the second timbre based on the semantic features, the at least two levels of style features, and the timbre features includes:

[0246] The semantic features and the timbre features are superimposed to obtain the first fusion feature;

[0247] The first fused feature and the style features of at least two levels are encoded and decoded to obtain the target Mel spectrum features;

[0248] The target speech is generated based on the target Mel spectrum features.

[0249] According to one or more embodiments of this disclosure, the at least two levels of style features include: global-level style features, local-level style features, and frame-level style features; encoding and decoding the first fused feature and the at least two levels of style features to obtain target Mel spectrum features include:

[0250] The first fusion feature and the style feature of the local level are dimensionally concatenated to obtain the second fusion feature;

[0251] The second fused feature is encoded to obtain the encoded feature;

[0252] The encoding features, the global-level style features, and the frame-level style features are concatenated dimensionally to obtain the third fusion feature;

[0253] The third fusion feature is decoded to obtain the target Mel spectrum feature.

[0254] According to one or more embodiments of this disclosure, the coding features, the global-level style features, and the frame-level style features are dimensionally concatenated to obtain a third fusion feature, including:

[0255] The encoding feature and the timbre feature are superimposed to obtain the fourth fusion feature;

[0256] The third fusion feature is obtained by dimensionally concatenating the fourth fusion feature, the global-level style feature, and the frame-level style feature.

[0257] According to one or more embodiments of this disclosure, the original speech is processed to obtain at least two levels of style features corresponding to the original speech, including at least two of the following:

[0258] The original speech is feature extracted by a pre-trained self-supervised learning model to obtain the self-supervised learning features corresponding to each speech frame in the original speech. The self-supervised learning features corresponding to each speech frame are encoded by a first reference encoder to obtain the global-level style features.

[0259] The original speech is feature extracted by a pre-trained speech recognition model to obtain the bottleneck features corresponding to each speech frame in the original speech. The bottleneck features corresponding to each speech frame are encoded by a second reference encoder to obtain the hidden layer features of each speech frame. The hidden layer features of each speech frame are downsampled and upsampled to obtain the style features of the local level.

[0260] Obtain the fundamental frequency features, articulation features, and energy features corresponding to each speech frame in the original speech, and determine the style features at the frame level based on the fundamental frequency features, articulation features, and energy features corresponding to each speech frame.

[0261] According to one or more embodiments of this disclosure, the original speech is processed to obtain semantic features and at least two levels of style features corresponding to the original speech, including:

[0262] The original speech is processed by a trained speech conversion model to obtain the semantic features and at least two levels of style features corresponding to the original speech.

[0263] Based on the identifier of the second timbre, determine the timbre characteristics corresponding to the second timbre, including:

[0264] The speech conversion model determines the timbre characteristics corresponding to the second timbre based on the identifier of the second timbre;

[0265] Based on the semantic features, the at least two levels of style features, and the timbre features, a target speech with the second timbre is generated, including:

[0266] The speech conversion model generates a target speech with the second timbre based on the semantic features, the at least two levels of style features, and the timbre features.

[0267] The speech conversion model is trained using multiple sets of training samples. Each set of training samples includes: original speech samples with a first sample timbre, an identifier for a second sample timbre, and target speech samples with the second sample timbre. The original speech samples and the target speech samples express the same semantics.

[0268] Secondly, according to one or more embodiments of this disclosure, a method for training a speech conversion model is provided, comprising:

[0269] Multiple sets of first training samples are obtained. Each set of first training samples includes: a first sample original speech with a first sample timbre, an identifier with a second sample timbre, and a first sample target speech with a second sample timbre. The first sample original speech and the first sample target speech express the same semantics.

[0270] The original speech of the first sample is processed by the speech conversion model to be trained to obtain the first semantic features and at least two levels of first predicted style features corresponding to the original speech of the first sample.

[0271] The speech conversion model determines the timbre features corresponding to the second sample timbre based on the identifier of the second sample timbre.

[0272] The speech conversion model generates a first predicted target speech based on the first semantic feature, the first predicted style features at least two levels, and the timbre features corresponding to the second sample timbre.

[0273] The speech conversion model is trained based on the identifiers of the first predicted target speech, the first sample original speech, the first sample target speech, and the second sample timbre; wherein the training objectives include: minimizing the style difference between the first predicted target speech and the first sample original speech, minimizing the semantic difference between the first predicted target speech and the first sample target speech, and minimizing the timbre difference between the timbre of the first predicted target speech and the second sample timbre.

[0274] According to one or more embodiments of this disclosure, training the speech conversion model based on the identifiers of the first predicted target speech, the first sample original speech, the first sample target speech, and the second sample timbre includes:

[0275] Determine the style loss value based on the first predicted target speech and the first sample original speech;

[0276] Determine the semantic loss value based on the first predicted target speech and the first sample target speech;

[0277] Based on the identifiers of the first predicted target speech and the second sample timbre, determine the timbre loss value;

[0278] The target loss value is determined based on the style loss value, the semantic loss value, and the timbre loss value;

[0279] The speech conversion model is trained with the goal of minimizing the target loss value.

[0280] According to one or more embodiments of this disclosure, determining a style loss value based on the first predicted target speech and the first sample original speech includes:

[0281] The Mel spectrum features of the first predicted target speech are processed by a pre-trained style classification model to obtain the first intermediate features output by each network unit in the style classification model.

[0282] The style classification model is used to process the Mel spectrum features of the original speech of the first sample to obtain the second intermediate features output by each network unit in the style classification model.

[0283] The style loss value is determined based on the first intermediate feature and the second intermediate feature.

[0284] According to one or more embodiments of this disclosure, determining a semantic loss value based on the first predicted target speech and the first sample target speech includes:

[0285] The semantic loss value is determined based on the Mel spectrum features of the first predicted target speech and the Mel spectrum features of the first sample target speech.

[0286] According to one or more embodiments of this disclosure, determining a timbre loss value based on the identifiers of the first predicted target speech and the second sample timbre includes:

[0287] The Mel spectrum features of the first predicted target speech are processed by a timbre classification model to obtain a first probability. The first probability is used to indicate the probability that the timbre identifier of the first predicted target speech matches the identifier of the second sample timbre.

[0288] The timbre loss value is determined based on the first probability.

[0289] According to one or more embodiments of this disclosure, the first sample timbre and the second sample timbre are the same timbre; after the speech conversion model reaches a preset convergence condition, the method further includes:

[0290] Multiple sets of second training samples are obtained, each set of second training samples includes: original second sample speech with third sample timbre and identifier of fourth sample timbre; the third sample timbre and the fourth sample timbre are different;

[0291] The second sample original speech is processed by the speech conversion model to obtain the second semantic feature and at least two levels of second predicted style features corresponding to the second sample original speech. The timbre feature corresponding to the fourth sample timbre is determined according to the identifier of the fourth sample timbre. The second predicted target speech is generated according to the second semantic feature, the at least two levels of second predicted style features and the timbre feature corresponding to the fourth sample timbre.

[0292] Based on the identifiers of the second predicted target speech, the second sample original speech, and the fourth sample timbre, the parameters of the speech conversion model are updated. The update objectives include: minimizing the style difference between the second predicted target speech and the second sample original speech, and minimizing the timbre difference between the second predicted target speech and the fourth sample timbre.

[0293] According to one or more embodiments of this disclosure, the first predictive style features of the at least two levels include at least two of the following:

[0294] Global-level style features are used to characterize the overall expressive style of the original speech of the first sample;

[0295] Local-level style features are used to characterize the expressive style presented by different speech segments of the original speech of the first sample;

[0296] Frame-level style features are used to characterize the expressive style presented by different speech frames in the original speech of the first sample.

[0297] Thirdly, according to one or more embodiments of this disclosure, a voice conversion device is provided, comprising:

[0298] The acquisition module is used to acquire the original speech with the first timbre and the identifier of the second timbre to be converted;

[0299] The feature extraction module is used to process the original speech to obtain semantic features and at least two levels of style features corresponding to the original speech; the style features are used to indicate the expression style corresponding to the original speech.

[0300] The determining module is used to determine the timbre characteristics corresponding to the second timbre based on the identifier of the second timbre;

[0301] The generation module is used to generate target speech with the second timbre based on the semantic features, the style features at least two levels, and the timbre features.

[0302] According to one or more embodiments of this disclosure, the at least two levels of style features include at least two of the following:

[0303] Global-level style features are used to characterize the overall expressive style of the original speech;

[0304] Local-level style features are used to characterize the expressive style presented by different speech segments of the original speech;

[0305] Frame-level style features are used to characterize the expressive style presented by different speech frames in the original speech.

[0306] According to one or more embodiments of this disclosure, the generation module is specifically used for:

[0307] The semantic features and the timbre features are superimposed to obtain the first fusion feature;

[0308] The first fused feature and the style features of at least two levels are encoded and decoded to obtain the target Mel spectrum features;

[0309] The target speech is generated based on the target Mel spectrum features.

[0310] According to one or more embodiments of this disclosure, the at least two levels of style features include: global-level style features, local-level style features, and frame-level style features; the generation module is specifically used for:

[0311] The first fusion feature and the style feature of the local level are dimensionally concatenated to obtain the second fusion feature;

[0312] The second fused feature is encoded to obtain the encoded feature;

[0313] The encoding features, the global-level style features, and the frame-level style features are concatenated dimensionally to obtain the third fusion feature;

[0314] The third fusion feature is decoded to obtain the target Mel spectrum feature.

[0315] According to one or more embodiments of this disclosure, the generation module is specifically used for:

[0316] The encoding feature and the timbre feature are superimposed to obtain the fourth fusion feature;

[0317] The third fusion feature is obtained by dimensionally concatenating the fourth fusion feature, the global-level style feature, and the frame-level style feature.

[0318] According to one or more embodiments of this disclosure, the feature extraction module is specifically configured to perform at least two of the following:

[0319] The original speech is feature extracted by a pre-trained self-supervised learning model to obtain the self-supervised learning features corresponding to each speech frame in the original speech. The self-supervised learning features corresponding to each speech frame are encoded by a first reference encoder to obtain the global-level style features.

[0320] The original speech is feature extracted by a pre-trained speech recognition model to obtain the bottleneck features corresponding to each speech frame in the original speech. The bottleneck features corresponding to each speech frame are encoded by a second reference encoder to obtain the hidden layer features of each speech frame. The hidden layer features of each speech frame are downsampled and upsampled to obtain the style features of the local level.

[0321] Obtain the fundamental frequency features, articulation features, and energy features corresponding to each speech frame in the original speech, and determine the style features at the frame level based on the fundamental frequency features, articulation features, and energy features corresponding to each speech frame.

[0322] According to one or more embodiments of this disclosure, the feature extraction module is specifically used to: process the original speech using a trained speech conversion model to obtain semantic features and at least two levels of style features corresponding to the original speech;

[0323] The determining module is specifically used to: determine the timbre features corresponding to the second timbre based on the identifier of the second timbre through the speech conversion model;

[0324] The generation module is specifically used to: generate target speech with the second timbre by means of the speech conversion model based on the semantic features, the style features at least two levels and the timbre features;

[0325] The speech conversion model is trained using multiple sets of training samples. Each set of training samples includes: original speech samples with a first sample timbre, an identifier for a second sample timbre, and target speech samples with the second sample timbre. The original speech samples and the target speech samples express the same semantics.

[0326] Fourthly, according to one or more embodiments of this disclosure, a training apparatus for a speech conversion model is provided, comprising:

[0327] The acquisition module is used to acquire multiple sets of first training samples. Each set of first training samples includes: a first sample original speech with a first sample timbre, an identifier with a second sample timbre, and a first sample target speech with a second sample timbre. The first sample original speech and the first sample target speech express the same semantics.

[0328] The feature extraction module is used to process the original speech of the first sample through the speech conversion model to be trained, and obtain the first semantic features and at least two levels of first predicted style features corresponding to the original speech of the first sample.

[0329] The determination module is used to determine the timbre features corresponding to the second sample timbre based on the identifier of the second sample timbre through the speech conversion model;

[0330] The generation module is used to generate a first predicted target speech based on the first semantic feature, the first predicted style features at least two levels, and the timbre features corresponding to the second sample timbre through the speech conversion model;

[0331] The training module is used to train the speech conversion model based on the identifiers of the first predicted target speech, the first sample original speech, the first sample target speech, and the second sample timbre; wherein the training objectives include: minimizing the style difference between the first predicted target speech and the first sample original speech, minimizing the semantic difference between the first predicted target speech and the first sample target speech, and minimizing the timbre difference between the first predicted target speech and the second sample timbre.

[0332] According to one or more embodiments of this disclosure, the training module is specifically used for:

[0333] Determine the style loss value based on the first predicted target speech and the first sample original speech;

[0334] The semantic loss value is determined based on the first predicted target speech and the first sample target speech;

[0335] Based on the identifiers of the first predicted target speech and the second sample timbre, determine the timbre loss value;

[0336] The target loss value is determined based on the style loss value, the semantic loss value, and the timbre loss value;

[0337] The speech conversion model is trained with the goal of minimizing the target loss value.

[0338] According to one or more embodiments of this disclosure, the training module is specifically used for:

[0339] The Mel spectrum features of the first predicted target speech are processed by a pre-trained style classification model to obtain the first intermediate features output by each network unit in the style classification model.

[0340] The style classification model is used to process the Mel spectrum features of the original speech of the first sample to obtain the second intermediate features output by each network unit in the style classification model.

[0341] The style loss value is determined based on the first intermediate feature and the second intermediate feature.

[0342] According to one or more embodiments of this disclosure, the training module is specifically used for:

[0343] The semantic loss value is determined based on the Mel spectrum features of the first predicted target speech and the Mel spectrum features of the first sample target speech.

[0344] According to one or more embodiments of this disclosure, the training module is specifically used for:

[0345] The Mel spectrum features of the first predicted target speech are processed by a timbre classification model to obtain a first probability. The first probability is used to indicate the probability that the timbre identifier corresponding to the first predicted target speech is a match of the identifier of the second sample timbre.

[0346] The timbre loss value is determined based on the first probability.

[0347] According to one or more embodiments of this disclosure, the first sample timbre and the second sample timbre are the same timbre; after the speech conversion model reaches a preset convergence condition, the acquisition module is further configured to: acquire multiple sets of second training samples, each set of second training samples including: a second sample original speech with a third sample timbre, and an identifier of a fourth sample timbre; the third sample timbre and the fourth sample timbre are different;

[0348] The feature extraction module is also used to: process the original speech of the second sample through the speech conversion model to obtain the second semantic features and at least two levels of second predicted style features corresponding to the original speech of the second sample;

[0349] The determining module is also used to: determine the timbre characteristics corresponding to the fourth sample timbre based on the identifier of the fourth sample timbre;

[0350] The generation module is also used to: generate a second predicted target speech based on the second semantic feature, the second predicted style features at least two levels, and the timbre features corresponding to the fourth sample timbre;

[0351] The training module is also used to update the parameters of the speech conversion model based on the identifiers of the second predicted target speech, the second sample original speech, and the fourth sample timbre. The update objectives include minimizing the style difference between the second predicted target speech and the second sample original speech, and minimizing the timbre difference between the second predicted target speech and the fourth sample timbre.

[0352] According to one or more embodiments of this disclosure, the first predictive style features of the at least two levels include at least two of the following:

[0353] Global-level style features are used to characterize the overall expressive style of the original speech of the first sample;

[0354] Local-level style features are used to characterize the expressive style presented by different speech segments of the original speech of the first sample;

[0355] Frame-level style features are used to characterize the expressive style presented by different speech frames in the original speech of the first sample.

[0356] Fifthly, according to one or more embodiments of the present disclosure, an electronic device is provided, including: a processor and a memory;

[0357] The memory stores computer-executed instructions;

[0358] The processor executes the computer execution instructions to implement the method described in the first aspect above or any possible design of the first aspect above, or the method described in the second aspect above or any of the various possible designs of the second aspect above.

[0359] In a sixth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, the computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the method described in the first aspect or any possible design of the first aspect above, or the method described in the second aspect or any possible design of the second aspect above.

[0360] In a seventh aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described in the first aspect or any possible design of the first aspect, or the method described in the second aspect or any possible design of the second aspect.

[0361] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0362] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0363] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A speech conversion method, characterized in that, include: Obtain the original speech with the first timbre, and the identifier of the second timbre to be converted; The original speech is processed to obtain semantic features and at least two levels of style features corresponding to the original speech; The style features are used to indicate the expression style corresponding to the original speech; the style features at least two levels include: global style features, local style features, and frame-level style features; the global style features are used to characterize the overall expression style of the original speech; the local style features are used to characterize the expression style of different speech segments of the original speech; and the frame-level style features are used to characterize the expression style of different speech frames in the original speech. Based on the identifier of the second timbre, determine the timbre characteristics corresponding to the second timbre; Based on the semantic features, the style features at least two levels, and the timbre features, a target speech with the second timbre is generated.

2. The method according to claim 1, characterized in that, Based on the semantic features, the at least two levels of style features, and the timbre features, a target speech with the second timbre is generated, including: The semantic features and the timbre features are superimposed to obtain the first fusion feature; The first fused feature and the style features of at least two levels are encoded and decoded to obtain the target Mel spectrum features; The target speech is generated based on the target Mel spectrum features.

3. The method according to claim 2, characterized in that, Encoding and decoding the first fused feature and the style features at least two levels are performed to obtain the target Mel spectrum features, including: The first fusion feature and the style feature of the local level are dimensionally concatenated to obtain the second fusion feature; The second fused feature is encoded to obtain the encoded feature; The encoding features, the global-level style features, and the frame-level style features are concatenated dimensionally to obtain the third fusion feature; The third fusion feature is decoded to obtain the target Mel spectrum feature.

4. The method according to claim 3, characterized in that, The encoding features, the global-level style features, and the frame-level style features are concatenated dimensionally to obtain a third fusion feature, including: The encoding feature and the timbre feature are superimposed to obtain the fourth fusion feature; The third fusion feature is obtained by dimensionally concatenating the fourth fusion feature, the global-level style feature, and the frame-level style feature.

5. The method according to any one of claims 1 to 4, characterized in that, The original speech is processed to obtain style features at least two levels corresponding to the original speech, including at least two of the following: The original speech is feature extracted by a pre-trained self-supervised learning model to obtain the self-supervised learning features corresponding to each speech frame in the original speech. The self-supervised learning features corresponding to each speech frame are encoded by a first reference encoder to obtain global style features. The original speech is feature extracted by a pre-trained speech recognition model to obtain the bottleneck features corresponding to each speech frame in the original speech. The bottleneck features corresponding to each speech frame are encoded by a second reference encoder to obtain the hidden features of each speech frame. The hidden features of each speech frame are downsampled and upsampled to obtain the local style features. Obtain the fundamental frequency features, articulation features, and energy features corresponding to each speech frame in the original speech, and determine the frame-level style features based on the fundamental frequency features, articulation features, and energy features corresponding to each speech frame.

6. The method according to any one of claims 1 to 5, characterized in that, The original speech is processed to obtain semantic features and at least two levels of style features, including: The original speech is processed by a trained speech conversion model to obtain the semantic features and at least two levels of style features corresponding to the original speech. Based on the identifier of the second timbre, determine the timbre characteristics corresponding to the second timbre, including: The speech conversion model determines the timbre characteristics corresponding to the second timbre based on the identifier of the second timbre; Based on the semantic features, the at least two levels of style features, and the timbre features, a target speech with the second timbre is generated, including: The speech conversion model generates a target speech with the second timbre based on the semantic features, the at least two levels of style features, and the timbre features. The speech conversion model is trained using multiple sets of training samples. Each set of training samples includes: original speech samples with a first sample timbre, an identifier for a second sample timbre, and target speech samples with the second sample timbre. The original speech samples and the target speech samples express the same semantics.

7. A training method for a speech conversion model, characterized in that, include: Multiple sets of first training samples are obtained. Each set of first training samples includes: a first sample original speech with a first sample timbre, an identifier with a second sample timbre, and a first sample target speech with a second sample timbre. The first sample original speech and the first sample target speech express the same semantics. The first sample original speech is processed by the speech conversion model to be trained to obtain the first semantic feature and at least two levels of first predicted style features corresponding to the first sample original speech; the at least two levels of first predicted style features include: global level style features, used to characterize the overall expression style of the first sample original speech; local level style features, used to characterize the expression style of different speech segments of the first sample original speech; and frame level style features, used to characterize the expression style of different speech frames in the first sample original speech. The speech conversion model determines the timbre features corresponding to the second sample timbre based on the identifier of the second sample timbre. The speech conversion model generates a first predicted target speech based on the first semantic feature, the first predicted style features at least two levels, and the timbre features corresponding to the second sample timbre. The speech conversion model is trained based on the identifiers of the first predicted target speech, the first sample original speech, the first sample target speech, and the second sample timbre; wherein the training objectives include: minimizing the style difference between the first predicted target speech and the first sample original speech, minimizing the semantic difference between the first predicted target speech and the first sample target speech, and minimizing the timbre difference between the first predicted target speech and the second sample timbre.

8. The method according to claim 7, characterized in that, The speech conversion model is trained based on the identifiers of the first predicted target speech, the first sample original speech, the first sample target speech, and the second sample timbre, including: Determine the style loss value based on the first predicted target speech and the first sample original speech; The semantic loss value is determined based on the first predicted target speech and the first sample target speech; Based on the identifiers of the first predicted target speech and the second sample timbre, determine the timbre loss value; The target loss value is determined based on the style loss value, the semantic loss value, and the timbre loss value; The speech conversion model is trained with the goal of minimizing the target loss value.

9. The method according to claim 8, characterized in that, Based on the first predicted target speech and the first sample original speech, determine the style loss value, including: The Mel spectrum features of the first predicted target speech are processed by a pre-trained style classification model to obtain the first intermediate features output by each network unit in the style classification model. The style classification model is used to process the Mel spectrum features of the original speech of the first sample to obtain the second intermediate features output by each network unit in the style classification model. The style loss value is determined based on the first intermediate feature and the second intermediate feature.

10. The method according to claim 8 or 9, characterized in that, Based on the first predicted target speech and the first sample target speech, a semantic loss value is determined, including: The semantic loss value is determined based on the Mel spectrum features of the first predicted target speech and the Mel spectrum features of the first sample target speech.

11. The method according to any one of claims 8 to 10, characterized in that, Based on the identifiers of the first predicted target speech and the second sample timbre, the timbre loss value is determined, including: The Mel spectrum features of the first predicted target speech are processed by a timbre classification model to obtain a first probability. The first probability is used to indicate the probability that the timbre identifier of the first predicted target speech matches the identifier of the second sample timbre. The timbre loss value is determined based on the first probability.

12. The method according to any one of claims 7 to 10, characterized in that, The first sample timbre and the second sample timbre are the same timbre; After the speech conversion model reaches the preset convergence condition, the following is also included: Multiple sets of second training samples are obtained, each set of second training samples includes: original second sample speech with third sample timbre and identifier of fourth sample timbre; the third sample timbre and the fourth sample timbre are different; The second sample original speech is processed by the speech conversion model to obtain the second semantic feature and at least two levels of second predicted style features corresponding to the second sample original speech. The timbre feature corresponding to the fourth sample timbre is determined according to the identifier of the fourth sample timbre. The second predicted target speech is generated according to the second semantic feature, the at least two levels of second predicted style features and the timbre feature corresponding to the fourth sample timbre. Based on the identifiers of the second predicted target speech, the second sample original speech, and the fourth sample timbre, the parameters of the speech conversion model are updated. The update objectives include: minimizing the style difference between the second predicted target speech and the second sample original speech, and minimizing the timbre difference between the second predicted target speech and the fourth sample timbre.

13. A voice conversion device, characterized in that, include: The acquisition module is used to acquire the original speech with the first timbre and the identifier of the second timbre to be converted; The feature extraction module is used to process the original speech to obtain the semantic features and at least two levels of style features corresponding to the original speech. The style features are used to indicate the expression style corresponding to the original speech; the style features at least two levels include: global style features, local style features, and frame-level style features; the global style features are used to characterize the overall expression style of the original speech; the local style features are used to characterize the expression style of different speech segments of the original speech; and the frame-level style features are used to characterize the expression style of different speech frames in the original speech. The determining module is used to determine the timbre characteristics corresponding to the second timbre based on the identifier of the second timbre; The generation module is used to generate target speech with the second timbre based on the semantic features, the style features at least two levels, and the timbre features.

14. A training device for a speech conversion model, characterized in that, include: The acquisition module is used to acquire multiple sets of first training samples. Each set of first training samples includes: a first sample original speech with a first sample timbre, an identifier with a second sample timbre, and a first sample target speech with a second sample timbre. The first sample original speech and the first sample target speech express the same semantics. The feature extraction module is used to process the first sample original speech using the speech conversion model to be trained, to obtain the first semantic features and at least two levels of first predicted style features corresponding to the first sample original speech; the at least two levels of first predicted style features include: global level style features, used to characterize the overall expression style of the first sample original speech; local level style features, used to characterize the expression style of different speech segments of the first sample original speech; and frame level style features, used to characterize the expression style of different speech frames in the first sample original speech. The determination module is used to determine the timbre features corresponding to the second sample timbre based on the identifier of the second sample timbre through the speech conversion model; The generation module is used to generate a first predicted target speech based on the first semantic feature, the first predicted style features at least two levels, and the timbre features corresponding to the second sample timbre through the speech conversion model; The training module is used to train the speech conversion model based on the identifiers of the first predicted target speech, the first sample original speech, the first sample target speech, and the second sample timbre; wherein the training objectives include: minimizing the style difference between the first predicted target speech and the first sample original speech, minimizing the semantic difference between the first predicted target speech and the first sample target speech, and minimizing the timbre difference between the first predicted target speech and the second sample timbre.

15. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes the computer execution instructions to implement the method as claimed in any one of claims 1 to 6, or the method as claimed in any one of claims 7 to 12.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 7, or the method as described in any one of claims 7 to 12.

17. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 6, or the method as described in any one of claims 7 to 12.

Images