Audio generation method, system and related device

Abstract

The invention discloses an audio generation method and system and a related device, and the method comprises the steps: obtaining a to-be-converted target text, and determining the language information and acoustic information corresponding to the target text; wherein the language information is related to text elements of the target text, and the acoustic information is related to a reference spectrum matched with the target text; determining a first target network matched with the language information from a plurality of first expert networks, acquiring rhythm prediction features matched with the language information by using the first target network, and determining a second target network matched with the acoustic information from a plurality of second expert networks, acquiring an acoustic prediction feature matched with the acoustic information by using the second target network; and based on the rhythm prediction feature and the acoustic prediction feature, generating a target audio after the target text is converted. In this way, the effect of converting the target text into the target audio can be improved.

Description

Technical Field

[0001] This application relates to the field of audio generation technology, and in particular to an audio generation method, system and related apparatus. Background Technology

[0002] In the early stages of text-to-audio conversion technology, systems primarily relied on a concatenation-based synthesis method. This involved pre-recording a large library of speech units (such as syllables, words, or phrases) and then selecting and concatenating the corresponding units from the input text. While this method could preserve some naturalness in the recorded speech, its output was stiff, lacked emotional variation, and had poor adaptability to texts beyond a limited vocabulary.

[0003] Therefore, improving the audio conversion effect has become an urgent problem to be solved. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide an audio generation method, system, and related apparatus that can improve the effect of converting target text into target audio.

[0005] To address the aforementioned technical problems, this application provides an audio generation method comprising: acquiring target text to be converted; determining linguistic and acoustic information corresponding to the target text; wherein the linguistic information is related to text elements of the target text, and the acoustic information is related to a reference spectrum matching the target text; determining a first target network matching the linguistic information from multiple first expert networks; obtaining prosodic prediction features matching the linguistic information using the first target network; determining a second target network matching the acoustic information from multiple second expert networks; obtaining acoustic prediction features matching the acoustic information using the second target network; and generating target audio after conversion of the target text based on the prosodic prediction features and the acoustic prediction features.

[0006] To address the aforementioned technical problems, another technical solution adopted in this application is: providing an audio generation system, comprising: an acquisition module, configured to acquire target text to be converted, and determine the linguistic information and acoustic information corresponding to the target text; wherein the linguistic information is related to the text elements of the target text, and the acoustic information is related to a reference spectrum matching the target text; a processing module, configured to determine a first target network matching the linguistic information from multiple first expert networks, obtain prosodic prediction features matching the linguistic information using the first target network, determine a second target network matching the acoustic information from multiple second expert networks, and obtain acoustic prediction features matching the acoustic information using the second target network; and a conversion module, configured to generate target audio after conversion of the target text based on the prosodic prediction features and the acoustic prediction features.

[0007] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide an electronic device, including a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the method mentioned in the above technical solution.

[0008] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium having program instructions stored thereon, wherein the program instructions, when executed by a processor, implement the method mentioned in the above technical solution.

[0009] The beneficial effects of this application are as follows: Unlike existing technologies, the audio generation method proposed in this application, after obtaining the target text to be converted, obtains two different dimensions of information based on the target text: linguistic information and acoustic information. For these different dimensions of information, a first target network matching the linguistic information is determined from multiple first expert networks, and the first target network is used to predict audio prosody based on the linguistic information, obtaining prosodic prediction features; and a second target network matching the acoustic information is determined from multiple second expert networks, and the second target network is used to predict acoustic features based on the acoustic information, obtaining acoustic prediction features. Based on the predicted prosodic prediction features and acoustic prediction features, the target text is converted into target audio, thereby improving the performance of the target audio in terms of pronunciation accuracy and prosodic richness. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating one embodiment of the audio generation method of this application; Figure 2 yes Figure 1 The flowchart of step S101 corresponds to another embodiment; Figure 3 This is a schematic diagram of one embodiment of the audio generation model of this application; Figure 4 yes Figure 1 The flowchart of step S102 corresponds to one embodiment; Figure 5 yes Figure 4 The flowchart of step S301 corresponds to another embodiment; Figure 6 yes Figure 1 The flowchart of step S102 corresponds to another embodiment; Figure 7 yes Figure 1 The flowchart of step S103 corresponds to another embodiment; Figure 8 This is a flowchart illustrating one implementation of the audio generation model training method of this application; Figure 9 This is a schematic diagram of one embodiment of the audio generation system of this application; Figure 10 This is a schematic diagram of the structure of one embodiment of the electronic device of this application; Figure 11 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments, and different embodiments can be adaptively combined. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0012] Please see Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the audio generation method of this application, which includes: S101: Obtain the target text to be converted, and determine the linguistic and acoustic information corresponding to the target text; wherein, the linguistic information is related to the text elements of the target text, and the acoustic information is related to the reference spectrum matched by the target text.

[0013] In one embodiment, target text to be converted into target audio is obtained. Based on the text elements in the target text, the language information corresponding to the target text is determined. Furthermore, based on the target text, a corresponding reference spectrum is obtained, and acoustic information matching the target text is determined based on the obtained reference spectrum.

[0014] In some implementation scenarios, after acquiring the target text, the language type corresponding to the target text is determined based on each text element in the target text. Additionally, the element features, word segmentation information, tone information, and pause information of each text element in the target text are acquired. At least one of the above language type, element features, word segmentation information, tone information, and pause information is used as language information.

[0015] In some implementation scenarios, after obtaining the target text, a reference spectrum matching the target text is constructed based on each text element in the target text, and acoustic information, including fundamental frequency information and energy information, is extracted from the reference spectrum.

[0016] Specifically, a pre-built pronunciation database is obtained, which stores pronunciations that match multiple candidate elements. Based on text elements in the target text, corresponding pronunciations are selected from the pronunciation database and sorted sequentially to obtain reference audio. The reference spectrum corresponding to the reference audio is obtained, and fundamental frequency and energy information are extracted from the reference spectrum as acoustic information.

[0017] In one embodiment, a preset fixed value is used as the acoustic information corresponding to the target text. This fixed value can be 0.

[0018] S102: Determine a first target network that matches the language information from multiple first expert networks, use the first target network to obtain prosodic prediction features that match the language information, determine a second target network that matches the acoustic information from multiple second expert networks, and use the second target network to obtain acoustic prediction features that match the acoustic information.

[0019] In one embodiment, multiple first expert networks are filtered based on language information to determine a first target network that matches the language information. The language information is then input into the first target network to obtain prosodic prediction features output by the first target network. Additionally, multiple second expert networks are filtered based on acoustic information to determine acoustic prediction features that match the acoustic information.

[0020] In some implementation scenarios, multiple first expert networks are used to process language information, and each first expert network is used to handle a fixed matching task, with the fixed tasks matched by different first expert networks differing. Similarly, multiple second expert networks are used to process acoustic information, and each second expert network is also used to handle a fixed matching task, with the fixed tasks matched by different second expert networks differing.

[0021] S103: Generate the target audio after the target text is converted based on prosodic prediction features and acoustic prediction features.

[0022] In one implementation, a target Mel spectrum is generated by decoding based on the obtained prosodic prediction features and acoustic prediction features. The target Mel spectrum is then converted into target audio.

[0023] The audio generation method proposed in this application, after obtaining the target text to be converted, obtains two different dimensions of information: linguistic information and acoustic information. For these different dimensions of information, a first target network matching the linguistic information is determined from multiple first expert networks, and the first target network is used to predict audio prosody based on the linguistic information, obtaining prosodic prediction features. Similarly, a second target network matching the acoustic information is determined from multiple second expert networks, and the second target network is used to predict acoustic features based on the acoustic information, obtaining acoustic prediction features. Based on the predicted prosodic and acoustic features, the target text is converted into target audio, thereby improving the performance of the target audio in terms of pronunciation accuracy and prosodic richness.

[0024] Please see Figure 2 , Figure 2 yes Figure 1 The flowchart of step S101 corresponds to another embodiment. Specifically, the language information acquisition step in step S101 includes: S201: Obtain the language type, word segmentation, and prosodic information of the target text; where word segmentation is obtained by splitting the text elements in the target text, and prosodic information includes the tone of each text element in the target text and the pause position of each text element.

[0025] In one implementation, the language type of the target text is determined based on the language type of each text element in the target text. Each text element in the target text is then split to obtain the corresponding word segment. Additionally, the tone of each text element and the pause positions between adjacent text elements are obtained.

[0026] In some implementation scenarios, if the text elements in the target text are English words, then the language type of the target text is determined to be English. Furthermore, to reduce the difficulty of subsequent processing, the text elements are split into corresponding word segments; for example, the text element "action" is split into the word segments "ac" and "tion".

[0027] S202: Obtain the timbre features that match the pronunciation object.

[0028] In one embodiment, in order to ensure that the timbre of the final generated target audio is consistent with that of the speech object, timbre features that match the speech object are obtained.

[0029] In some implementation scenarios, a pre-built timbre database is accessed. This database stores timbre features that match multiple candidate objects, and each candidate object has a unique timbre identifier. A user-specified pronunciation object is retrieved, and based on the timbre identifier of the pronunciation object, the corresponding timbre features are retrieved from the timbre database.

[0030] S203: Input the target text into the trained phoneme analysis model and obtain the phoneme features output by the phoneme analysis model based on the phonemes corresponding to each text element.

[0031] In one embodiment, a trained phoneme analysis model is obtained, and the target text is input into the phoneme analysis model. The feature extraction network in the phoneme analysis model is used to analyze the phonemes corresponding to each text element, thereby obtaining the phoneme features output by the phoneme analysis model. Specifically, under the language type corresponding to the target text, the aforementioned phoneme features are used to characterize the combination rules and correlations between the phonemes corresponding to all text elements in the target text.

[0032] In some implementation scenarios, in order to combine the combination rules between different phonemes in the subsequent process of generating target audio, the extracted factor features are carried out by using the trained factor analysis model. This helps to improve the application scope of the audio generation method proposed in this application, and can achieve high-quality audio conversion even for low-resource minority languages.

[0033] S204: Language type, word segmentation, prosodic information, timbre features, and phoneme features are considered as language information.

[0034] In one embodiment, the obtained language type, word segmentation, prosodic information, timbre features, and phoneme features are used as language information.

[0035] It should be noted that the order in which the content of the language information is acquired can also be different. For example, the content that constitutes the language information can be acquired simultaneously or sequentially in a corresponding order.

[0036] The above scheme improves the accuracy of acoustic prediction feature generation by acquiring various types of information from the target text as linguistic information and providing a basis for the generation of subsequent acoustic prediction features.

[0037] In one embodiment, the factor analysis model mentioned above is trained using multiple training samples. The specific training process of the phoneme analysis model includes: acquiring multiple training texts; acquiring a phoneme sequence corresponding to each training element in the training texts; masking at least a portion of the phonemes in the phoneme sequence; inputting the masked phoneme sequence into the phoneme analysis model; and using the phoneme analysis model to obtain training phoneme features that match the input phoneme sequence. Based on the training phoneme features, the phoneme analysis model outputs the predicted phoneme at the masked position. Based on the original phoneme and the predicted phoneme at the masked position, the training loss corresponding to the phoneme analysis model is calculated; and the parameters of the phoneme analysis model are adjusted using the training loss until the trained phoneme analysis model is obtained.

[0038] Specifically, multiple training texts are pre-acquired and preprocessed. For each preprocessed training text, the phoneme corresponding to each training element is determined, and the phonemes corresponding to each training element are arranged sequentially according to their positions in the training text to obtain a phoneme sequence. At least a portion of the phonemes in the phoneme sequence are masked, i.e., at least a portion of the phonemes are replaced with a mask identifier "[MASK]". The masked phoneme sequence is input into a phoneme analysis model to extract the combination rules and relationships between different phonemes using the feature extraction network in the phoneme analysis model, and outputs training phoneme features that match the masked phoneme sequence. The phoneme analysis model predicts the predicted phoneme at the mask position based on the training factor features. Based on the original phoneme and the predicted phoneme at the mask position, the training loss corresponding to the phoneme analysis model is calculated, and the parameters of the phoneme analysis model are adjusted using the training loss. Training stops when the preset convergence condition of the phoneme analysis model is met, and the trained phoneme analysis model is obtained; or, training stops when the preset number of training rounds is reached, and the trained phoneme analysis model is obtained. The specific model structure of the aforementioned phoneme analysis model can refer to existing neural network model structures, such as the BERT model structure.

[0039] In some implementation scenarios, the aforementioned methods for preprocessing training text include text cleaning to remove noise, unifying language type, and removing anomalous training elements. Unifying language type involves removing a small number of training elements from other language types, while anomalous training elements include those with spelling errors and / or formatting issues. Furthermore, the phonemes corresponding to each training element can be represented using existing phoneme annotation tools.

[0040] In some implementation scenarios, the process of masking at least a portion of the phonemes in a phoneme sequence includes: randomly selecting a portion of phonemes from the phoneme sequence as first candidate phonemes, and replacing the first candidate phonemes with the mask identifier "[MASK]". Additionally, randomly selecting a portion of phonemes from the phoneme sequence as second candidate phonemes, and randomly replacing each second candidate phoneme with another phoneme. Preferably, the number of first candidate phonemes accounts for 12% of the total number of phonemes in the phoneme sequence, and the number of second candidate phonemes accounts for 1.5% of the total number of phonemes in the phoneme sequence.

[0041] In some implementation scenarios, the specific process by which the aforementioned feature extraction network obtains training phoneme features that match the phoneme sequence includes: encoding each phoneme in the masked phoneme sequence to obtain a corresponding phoneme vector; and encoding the position of each phoneme in the masked phoneme sequence to obtain positional information representing each phoneme's position in the phoneme sequence. The phoneme vectors and positional information corresponding to each phoneme in the phoneme sequence are then used as training phoneme features.

[0042] The above scheme trains a phoneme analysis model, which is then used to extract phoneme features from the target text during subsequent audio generation, thereby improving the accuracy of target audio generation.

[0043] In one implementation, please refer to Figure 3 , Figure 3 This is a schematic diagram of one embodiment of the audio generation model of this application. The audio generation method in the above-described embodiment is implemented using a trained audio generation model. Figure 2 As shown, the audio generation model includes a first coding module and a second coding module that are coupled to each other. The first coding module includes multiple first expert networks, a language gating network and at least one first shared network; the second coding module includes multiple second expert networks, an acoustic gating network and at least one second shared network.

[0044] In some implementation scenarios, each first expert network and each second expert network are used to handle a fixed matching task. The language gating network is used to select a first target network from the first expert networks to process the language information corresponding to the current target text, and the acoustic gating network is used to select a second target network from the second expert networks to process the acoustic information corresponding to the current target text.

[0045] Furthermore, unlike the first expert network, in actual processing, each first expert network focuses on partial information of the language to perform a corresponding fixed task; however, the first shared network focuses on complete language information to perform a corresponding generalization task. Similarly, each second expert network focuses on partial information of the acoustics to perform a corresponding fixed task, while the second shared network focuses on complete acoustic information to perform a corresponding generalization task.

[0046] Please see Figure 4 , Figure 4 yes Figure 1 Step S102 corresponds to a flowchart of one embodiment. Specifically, the process of obtaining prosody prediction features in step S102 includes: S301: Based on language information, a first target network is determined from all first expert networks using a language gating network; wherein each first expert network is used to process a matching fixed task, and the fixed tasks matched by different first expert networks are different.

[0047] In one embodiment, language information obtained from the target text is input into a first encoding module. The language gating network in the first encoding network is used to analyze the input language information to calculate the language matching confidence level between each first expert network and the language information. Based on the language matching confidence level, a first target network that matches the language information is selected from all first expert networks.

[0048] In some implementation scenarios, the higher the language matching confidence level, the higher the matching degree between the corresponding first expert network and the language information. After calculating the language matching confidence level between each first expert network and the language information using a language gating network, the first expert network corresponding to the highest language matching confidence level is taken as the first target network.

[0049] Optionally, in other implementation scenarios, there can be multiple first target networks. Specifically, after calculating the language matching confidence between each first expert network and the language information, the first expert networks with the highest language matching confidence are all used as first target networks. The number of first target networks can be set according to actual needs.

[0050] S302: Input the language information into the first target network and obtain the first reference prosodic features output by the first target network.

[0051] In one embodiment, after determining the first target network using a language gating network, language information is input into the first target network to process the input language information and output the corresponding first reference prosodic features.

[0052] It should be noted that when there are multiple first target networks, language information is simultaneously input into each first target network so that each first target network processes the input language information and outputs the corresponding first reference prosodic features.

[0053] In some implementation scenarios, the first target network matching has a corresponding fixed task. After the language information is input into the first target network, it is used to extract information from the language information to obtain language extraction information related to the fixed matching task. The first target network then performs the fixed matching task based on the language extraction information, and finally outputs the first reference prosodic features.

[0054] In a specific application scenario, the first target network is used to perform a corresponding fixed task to predict the pause duration. After the language information is input into the first target network, the first target network is used to extract the pause position corresponding to each text element from the language information. The pause position and the text elements before and after the pause position are analyzed to predict the corresponding pause duration.

[0055] Alternatively, the first target network is used to perform a corresponding fixed task to predict the stress distribution of the subsequently generated audio. After the language information is input into the first target network, the first target network is used to extract the tone of the text element and the phoneme features corresponding to the text element from the language information. At least one of the tone and phoneme features is analyzed to predict the stress distribution.

[0056] Alternatively, the first target network can be used to perform specific tasks, such as word segmentation of text elements. After language information is input into the first target network, it extracts the corresponding word segments for each text element. For each word segment corresponding to a text element, the first target network establishes a matching correspondence. By processing the word segments corresponding to each text element using the first target network, the same word segmentation units do not need to be processed repeatedly in practical applications, thereby improving processing efficiency and reducing computational costs. For example, if multiple text elements in the target text include "action," which, after word segmentation, results in "ac" + "tion," semantic analysis of these two word segments is required during the target audio generation process. When a subsequent text element includes "motion," its corresponding word segmentation includes "mo" + "tion," so there is no need to repeatedly analyze the word segmentation "tion."

[0057] S303: Input the language information into each first shared network and obtain the second reference prosodic features output by each first shared network.

[0058] In one embodiment, the first encoding module further includes at least one first shared network, into which the language information corresponding to the target text is input to each first shared network, so as to use the first shared network to perform overall analysis of the complete language information, thereby outputting a second reference prosodic feature for audio conversion.

[0059] In a specific application scenario, to ensure that the subsequently generated audio has a consistent energy benchmark, the first sharing expert performs an energy normalization task based on complete language information and outputs corresponding second reference prosodic features. These second reference prosodic features have consistent overall energy and retain natural energy fluctuations, such as changes in stress and intonation.

[0060] It should be noted that in practical applications, the order in which the first reference prosodic feature and the second reference prosodic feature are obtained can also be other. For example, the first reference prosodic feature and the second reference prosodic feature can be obtained simultaneously; or, the second reference prosodic feature can be obtained first, and then the first reference prosodic feature can be obtained.

[0061] In one embodiment, in order to further improve the accuracy of the second reference prosodic feature, the implementation process of step S303 may further include: after obtaining the first reference prosodic feature output by the first target network through the above-described corresponding implementation method, inputting the first reference prosodic feature and language information together into each first shared network, so as to use the first shared network to perform overall analysis of the first reference prosodic feature and language information, thereby outputting the second reference prosodic feature.

[0062] S304: Based on the first reference prosodic features and the second reference prosodic features, obtain the prosodic prediction features that match the target text.

[0063] In one embodiment, prosodic prediction features matching the target text are obtained based on the obtained first reference prosodic features and second reference prosodic features.

[0064] In some implementation scenarios, the first reference prosodic feature and the second reference prosodic feature are used together as prosodic prediction features.

[0065] In one embodiment, the first reference prosodic features and the second reference prosodic features are fused to obtain prosodic prediction features. The fusion method includes, but is not limited to, feature concatenation.

[0066] In one embodiment, all the obtained first reference prosodic features and second reference prosodic features are weighted and summed to obtain prosodic prediction features.

[0067] Specifically, a first weight matching the first reference prosodic feature and a second weight matching each second reference prosodic feature are set. The first reference prosodic feature is multiplied by the first weight, and each second reference prosodic feature is multiplied by the second weight. All the products are summed to obtain the prosodic prediction feature.

[0068] The above scheme selects a first target network from multiple first expert networks based on language information, and adaptively performs a fixed task with a high degree of matching with the language information to obtain prosodic prediction features with a high degree of fit with the target text, thereby helping to improve the smoothness of the subsequently generated target audio.

[0069] Please see Figure 5 , Figure 5 yes Figure 4 The flowchart of step S301 corresponds to another embodiment. Specifically, the implementation process of step S301 includes: S401: Based on language information, a first target set matching the language information is determined from all first network sets using a first language gating network; wherein, each first network set matches a task domain related to the language information, and the task domains matched by different first network sets are different.

[0070] In one implementation, such as Figure 3 As shown, the first encoding module includes multiple first network sets, each containing multiple first expert networks and multiple language gating networks, namely, a first language gating network and a second language gating network. After acquiring the language information, the first language gating network is used to calculate the first matching degree between each first network set and the language information. The first network set corresponding to the highest first matching degree is taken as the first target set. By determining the first target set with the highest matching degree with the language information, it helps to subsequently select the first target network with the highest fit with the language information from all the first expert networks in the first target set.

[0071] In some implementation scenarios, each first network set in the first encoding module is matched with a task domain related to language information, and the task domains matched by different first network sets may differ. Furthermore, the first expert network in the first network set is used to handle fixed tasks within its corresponding task domain.

[0072] S402: Use a second language gating network to determine the first target network that matches the language information from all first expert networks corresponding to the first target set.

[0073] In one embodiment, after determining the first target set using a first language gating network, a second language gating network is invoked to calculate the language matching confidence score between each first expert network and the language information for all first expert networks in the first target set. Based on the language matching confidence score, a first target network that matches the language information is selected from all first expert networks. The specific process of selecting the first target network from multiple first expert networks using the language matching confidence score can be referred to the corresponding implementation described above, and will not be elaborated further here.

[0074] In a specific application scenario, in response to the dense distribution of stress in the target text, in order to make the stress of the final generated target audio prominent and the rhythm accurate, a first language gating network is used to select a first network set that matches the stress processing task domain as the first target set, so that the second language gating network can be used to select a first target network from the first target set to process the language information for a fixed task.

[0075] The above scheme includes a first network set in the first encoding module that matches different task domains, and each first network set includes multiple first expert networks. This allows for the flexible selection of the first target network in the corresponding task domain based on the language information during the generation of prosodic prediction features. This targeted processing of language information to generate prosodic prediction features helps improve the quality of the subsequently generated target audio.

[0076] Please see Figure 6 , Figure 6 yes Figure 1 The flowchart of step S102 corresponds to another embodiment. Specifically, the process of acquiring acoustic prediction features in step S102 includes: S501: Based on acoustic information, a second target set matching the acoustic information is determined from all second network sets using a first acoustic gating network.

[0077] In one embodiment, the second encoding network includes multiple second network sets, each second network set including multiple second expert networks and multiple acoustic gating networks, namely a first acoustic gating network and a second acoustic gating network. After acquiring acoustic information, the first acoustic gating network is used to calculate a second matching degree between each second network set and the acoustic information. The second network set corresponding to the highest second matching degree is taken as the second target set. By determining the second target set with the highest matching degree with the acoustic information, it is helpful to subsequently select the second target network with the highest fit with the acoustic information from all the second expert networks in the second target set.

[0078] In some implementation scenarios, each second network set in the second encoding module is matched with a task domain related to acoustic information, and the task domains matched by different second network sets differ. Furthermore, the second expert network in the second network set is used to handle fixed tasks within the corresponding task domain.

[0079] S502: In response to the fact that the second target set includes multiple second expert networks, a second target network matching the language information is determined from all the second expert networks using a second acoustic gating network.

[0080] In one embodiment, for all second expert networks in the second target set, a second acoustic gating network is invoked to calculate the acoustic matching confidence score between each second expert network and the acoustic information. Based on the acoustic matching confidence score, a second target network that matches the acoustic information is selected from the corresponding plurality of second expert networks. The specific process of selecting the second target network from the plurality of second expert networks using the acoustic matching confidence score can be referred to the corresponding embodiment described above, and will not be elaborated here.

[0081] S503: Input acoustic information into the second target network and obtain the first reference acoustic feature output by the second target network.

[0082] In one embodiment, acoustic information is input to a second target network to process the input acoustic information and output a corresponding first reference acoustic feature.

[0083] In some implementation scenarios, acoustic information is input into a second target network, which then extracts information from the acoustic information to obtain acoustic extraction information relevant to a matching fixed task. The second target network then performs the matching fixed task based on the acoustic extraction information, ultimately outputting the first acoustic prosodic feature.

[0084] In a specific application scenario, in response to a determined second target network performing a fixed task to enhance high-frequency energy, acoustic information is input into the second target network, and acoustic extraction information related to high-frequency energy is extracted from the acoustic information. Further, the second target network is used to perform a matching task for high-frequency energy enhancement based on the acoustic extraction information, ultimately outputting a first acoustic prosodic feature.

[0085] S504: Input the acoustic information into each second shared network and obtain the second reference acoustic features output by each second shared network.

[0086] In one embodiment, the second encoding network further includes at least one second shared network, into which acoustic information corresponding to the target text is input to each second shared network, so as to use the second shared network to perform overall analysis of the complete acoustic information, thereby outputting a second reference acoustic feature for audio conversion.

[0087] S505: Based on the first reference acoustic features and the second reference acoustic features, obtain acoustic prediction features that match the target text.

[0088] In one embodiment, prosodic prediction features matching the target text are obtained based on the obtained first and second reference prosodic features. The specific method for obtaining the prosodic prediction features can be referred to in the corresponding embodiments described above, and will not be elaborated further here.

[0089] The above scheme uses a second coding network to process acoustic information to obtain acoustic prediction features, thereby enabling the use of different coding networks to process the linguistic and acoustic information obtained from the target text separately, thus improving the accuracy of processing the linguistic and acoustic information separately.

[0090] In one embodiment, the audio generation model further includes a variable prediction module. After obtaining prosodic prediction features and acoustic prediction features through the corresponding embodiments described above, the prosodic prediction features and acoustic prediction features are input to the variable prediction module to obtain prediction parameter information output by the variable prediction module. The prediction parameter information includes prediction duration information, fundamental frequency prediction information, and energy prediction information.

[0091] Specifically, the variable prediction module includes a duration prediction network, a fundamental frequency prediction network, and an energy prediction network. The duration prediction network predicts the actual duration of each text element in the target text when converted to audio, based on the input prosodic and acoustic prediction features. The fundamental frequency prediction network predicts the fundamental frequency profile of the subsequently generated target audio based on the input prosodic and acoustic prediction features and outputs the fundamental frequency prediction information. The energy prediction network predicts the energy distribution of the subsequently generated target audio based on the input prosodic and acoustic prediction features, controls the volume intensity, and outputs the corresponding energy prediction information.

[0092] In a specific application scenario, taking the phoneme / d / corresponding to the target text as an example, the corresponding prediction duration information is 0.1 seconds, the fundamental frequency prediction information is 210Hz, and the energy prediction information is 0.8.

[0093] Please combine Figure 2 See Figure 7 , Figure 7 yes Figure 1The flowchart of step S103 corresponds to another embodiment. Specifically, the audio generation model further includes a first decoding module and a second decoding module coupled in sequence. The first decoding network includes multiple third expert networks and at least one third shared network, and the second decoding network includes multiple fourth expert networks and at least one fourth shared network. Based on this, the implementation process of step S103 includes: S601: Identify a third target network from all third expert networks in the first decoding network that matches the prosodic prediction features and acoustic prediction features.

[0094] In one embodiment, in response to the first decoding network including multiple third expert networks, a matching third target network is selected from all the third expert networks using a corresponding decoding gating network based on prosodic prediction features and acoustic prediction features.

[0095] In some implementation scenarios, the first decoding network includes multiple sets of third networks divided according to task domains, and each set of third networks includes multiple third expert networks, each of which performs a fixed task. Based on the prosodic prediction features and acoustic prediction features obtained through the corresponding implementation methods described above, a third target set matching the prosodic prediction features and acoustic prediction features is selected from all the sets of third networks using the first decoding gating network. Further, a third target network is selected from all the third expert networks in the third target set using a second decoding gating network, so that the third target network can be used to perform the corresponding fixed task based on the current prosodic prediction features and acoustic prediction features. The specific selection process of the third target network can be referred to the corresponding implementation methods described above, and will not be described in detail here.

[0096] S602: Based on prosodic prediction features and acoustic prediction features, the initial Mel spectrum is obtained using a third target network and a third shared network.

[0097] In one embodiment, the prosodic prediction features and acoustic prediction features corresponding to the target text are input into a third target network to extract information from the prosodic and acoustic prediction features, and then processed based on the extracted features. Additionally, the prosodic and acoustic prediction features corresponding to the target text are input into each third shared network to analyze the complete prosodic and acoustic prediction features. An initial Mel spectrum is obtained based on the outputs of the third target network and the third shared network, respectively.

[0098] In some implementation scenarios, the first decoding network is used to generate an initial Mel spectrum based on prosodic prediction features and acoustic prediction features. The selected third target network is used to perform specific fixed tasks, such as enhancing high-frequency harmonic details or reducing noise; the third expert network is used to process the entire spectrum, such as ensuring overall volume consistency or a smooth fundamental frequency transition.

[0099] In one embodiment, the prosodic prediction features, acoustic prediction features, and prediction parameter information output by the variable prediction module corresponding to the target text are input into the third target network and each third shared network, respectively. An initial Mel spectrum is obtained based on the outputs of the third target network and the third shared network.

[0100] S603: Determine the fourth target network that matches the initial Mel spectrum from all the fourth expert networks in the second decoding network.

[0101] In one embodiment, in response to the inclusion of multiple fourth expert networks in the second decoding network, a matching third target network is selected from all third expert networks based on the initial Mel spectrum using a corresponding decoding gating network.

[0102] In some implementation scenarios, the second decoding network includes multiple sets of fourth networks divided according to task domains, and each set of fourth networks includes multiple fourth expert networks, each of which performs a fixed task. Based on the initial Mel spectrum obtained through the corresponding implementation methods described above, a third decoding gating network is used to filter from all sets of fourth networks to obtain a fourth target set that matches the prosodic prediction features and acoustic prediction features. Further, a fourth target network is selected from all fourth expert networks in the fourth target set using the fourth decoding gating network, so that the initial Mel spectrum can be further refined using the fourth target network.

[0103] S604: Based on the initial Mel spectrum, the target Mel spectrum is obtained using the fourth target network and the fourth shared network.

[0104] In one embodiment, the obtained initial Mel spectrum is input into the fourth target network and the fourth shared network respectively, so as to improve the initial Mel spectrum using the fourth target network and the fourth shared network respectively, and obtain the target Mel spectrum.

[0105] In some implementations, the fourth expert network is used to refine some details in the initial Mel spectrum. For example, it can inject natural noise to add breathing noise to the initial Mel spectrum, making the subsequently generated target audio more natural, or improve the continuity between adjacent frames. The fourth expert network focuses more on the complete initial Mel spectrum to achieve smoothness between all phonemes, resulting in smoother subsequently generated target audio.

[0106] S605: Obtain the target audio based on the target Mel spectrum.

[0107] In one implementation, the final target Mel spectrum is converted into target audio.

[0108] Please see Figure 8 , Figure 8 This is a flowchart illustrating one implementation of the audio generation model training method of this application. Specifically, the training process of the audio generation model includes: S701: Acquire multiple training samples, each training sample including mutually matched training text and training audio.

[0109] In one embodiment, multiple training samples are acquired, each training sample including mutually matched training text and training audio. The training audio is obtained by using different pronunciations to represent the corresponding training text.

[0110] In some implementation scenarios, different pronunciation objects are used to read the training text to obtain the corresponding training audio, and the corresponding training text and training audio are used as a set of training audio.

[0111] In one embodiment, multiple training audio files are pre-acquired, and audio recognition is performed based on the training audio files to obtain the corresponding training text. Alternatively, multiple training text files are pre-acquired, and existing audio generation tools are used to generate training audio files corresponding to the training text files.

[0112] In one embodiment, in order to increase the number of training samples and thus improve the training effect, after constructing a partial training sample, data augmentation is performed on the training audio in the sample so that multiple training samples are obtained by expanding a set of training samples.

[0113] In some implementation scenarios, the current training sample includes a first training audio and a first training text. By performing EQ curve transformation on the first training audio in the current training sample to adjust the gain of different frequency components in the first training audio, an adjusted second training audio is generated. Based on the first training text and the adjusted second training audio, a completely new training sample is constructed.

[0114] In a specific application scenario, training audio for EQ curve transformation can involve enhancing low frequencies to make the sound fuller and attenuating high frequencies to reduce noise. This method does not directly change the duration or pitch of the audio, but only adjusts the frequency distribution in the audio to generate audio with different styles. Alternatively, EQ parameters and high-pass filters (HPF) can be used for audio transformation; for example, high-frequency control, such as attenuating 3-4kHz by 1dB, can avoid high-frequency glitches after stretching and preserve a natural feel.

[0115] S702: Obtain training language information for training text matching and training acoustic information for training audio matching from the training samples.

[0116] In one implementation, matching training language information is obtained based on the training text in the training samples. And, matching training acoustic information is obtained based on the training audio.

[0117] In some implementation scenarios, training language information is extracted from each text element in the training text; the specific acquisition process can be referred to the corresponding implementation method described above. Furthermore, fundamental frequency information and energy information are extracted from the spectrum corresponding to the training audio, and these information are used as the training acoustic information for training audio matching.

[0118] S703: Construct an audio generation model, using the first encoding module in the audio generation model to obtain prosodic prediction features based on training language information, and using the second encoding module in the audio generation model to obtain acoustic prediction features based on training acoustic information.

[0119] In one embodiment, an audio generation model is constructed, the specific structure of which can be referred to the corresponding embodiment described above. Training language information is input into the first encoding module of the audio generation model to obtain prosodic prediction features corresponding to the training samples. Training acoustic information is also input into the second encoding module of the audio generation model to obtain acoustic prediction features corresponding to the training samples. The specific implementation process can be referred to the corresponding embodiment described above.

[0120] S704: Decode the prosodic prediction features and acoustic prediction features corresponding to the training samples to obtain the predicted Mel spectrum that matches the training samples, and convert the predicted Mel spectrum into predicted audio.

[0121] In one embodiment, the prosodic prediction features and acoustic prediction features corresponding to the training samples are decoded using an audio generation model to obtain the predicted Mel spectrum output by the audio generation model, and the predicted Mel spectrum is then converted into predicted audio. The specific process for obtaining the predicted Mel spectrum can refer to the method for obtaining the target Mel spectrum in the corresponding embodiments described above, and will not be elaborated upon here.

[0122] S705: Based on the predicted audio and the training audio, calculate the audio conversion loss of the audio generation model, and use the audio conversion loss to adjust the parameters in the audio generation model until the trained audio generation model is obtained.

[0123] In one embodiment, in response to obtaining the predicted audio output by the audio generation model corresponding to the training samples, the audio conversion loss of the model is calculated based on the predicted audio and the training audio, and the parameters of the audio generation model are adjusted using the audio conversion loss. Training stops upon reaching a preset convergence condition, and the trained audio generation model is obtained; or, training stops upon reaching a preset threshold number of training iterations, and the trained audio generation model is obtained.

[0124] Please see Figure 9 , Figure 9 This is a schematic diagram of one embodiment of the audio generation system of this application. The audio generation system includes an acquisition module 10, a processing module 20, and a conversion module 30 that are coupled to each other. Specifically: The acquisition module 10 is used to acquire the target text to be converted and determine the language information and acoustic information corresponding to the target text; wherein, the language information is related to the text elements of the target text, and the acoustic information is related to the reference spectrum matched by the target text.

[0125] The processing module 20 is used to determine a first target network that matches the language information from multiple first expert networks, obtain prosodic prediction features that match the language information using the first target network, determine a second target network that matches the acoustic information from multiple second expert networks, and obtain acoustic prediction features that match the acoustic information using the second target network.

[0126] The conversion module 30 is used to generate target audio after conversion of target text based on prosodic prediction features and acoustic prediction features.

[0127] The audio generation method utilizes a trained audio generation model, which includes a first coding module and a second coding module that are coupled to each other. The first coding module includes multiple first expert networks, a language gating network, and at least one first shared network. The second coding network includes multiple second expert networks, an acoustic gating network, and at least one second shared network.

[0128] The processing module 20 determines a first target network that matches the language information from multiple first expert networks, and uses the first target network to obtain prosodic prediction features that match the language information. This includes: determining the first target network from all first expert networks based on the language information using a language gating network; wherein each first expert network is used to process a fixed matching task, and the fixed tasks matched by different first expert networks are different; inputting the language information into the first target network to obtain a first reference prosodic feature output by the first target network; and inputting the language information into each first shared network to obtain a second reference prosodic feature output by each first shared network; and obtaining prosodic prediction features that match the target text based on the first reference prosodic feature and the second reference prosodic feature.

[0129] The first encoding module includes multiple first network sets, each of which includes multiple first expert networks. The processing module 20 determines a first target network from all first expert networks based on language information using a language gating network. This includes: determining a first target set matching the language information from all first network sets using a first language gating network; wherein each first network set matches a task domain related to the language information, and the task domains matched by different first network sets differ; and determining a first target network matching the language information from all first expert networks corresponding to the first target set using a second language gating network.

[0130] The audio generation model further includes a first decoding module and a second decoding module coupled in sequence. The first decoding network includes multiple third expert networks and at least one third shared network, and the second decoding network includes multiple fourth expert networks and at least one fourth shared network. The conversion module 30 generates target audio after conversion of target text based on prosodic prediction features and acoustic prediction features, including: determining a third target network that matches the prosodic prediction features and acoustic prediction features from all third expert networks in the first decoding network; obtaining an initial Mel spectrum using the third target network and the third shared network based on the prosodic prediction features and acoustic prediction features; determining a fourth target network that matches the initial Mel spectrum from all fourth expert networks in the second decoding network; obtaining a target Mel spectrum using the fourth target network and the fourth shared network based on the initial Mel spectrum; and obtaining the target audio based on the target Mel spectrum.

[0131] The training process of the audio generation model includes: acquiring multiple training samples, each including mutually matched training text and training audio; acquiring training language information matching the training text and training acoustic information matching the training audio in the training samples; constructing an audio generation model, using the first encoding module in the audio generation model to obtain prosodic prediction features based on the training language information, and using the second encoding module in the audio generation model to obtain acoustic prediction features based on the training acoustic information; decoding the prosodic prediction features and acoustic prediction features corresponding to the training samples to obtain predicted Mel spectra matching the training samples, and converting the predicted Mel spectra into predicted audio; calculating the audio conversion loss of the audio generation model based on the predicted audio and training audio, and using the audio conversion loss to adjust the parameters in the audio generation model until the trained audio generation model is obtained.

[0132] The steps for acquiring language information include: acquiring the language type, word segmentation, and prosodic information corresponding to the target text; wherein, word segmentation is obtained by splitting the text elements in the target text, and prosodic information includes the tone of each text element and the pause position of each text element; acquiring timbre features matching the pronunciation object; and inputting the target text into the trained phoneme analysis model to acquire the phoneme features output by the phoneme analysis model based on the phonemes corresponding to each text element; and using the language type, word segmentation, prosodic information, timbre features, and phoneme features as language information.

[0133] The phoneme analysis model is trained using multiple training texts. The training process includes: acquiring multiple training texts; obtaining a phoneme sequence for each training text based on the phonemes corresponding to each training element in the training texts; masking at least a portion of the phonemes in the phoneme sequence; inputting the masked phoneme sequence into the phoneme analysis model; using the phoneme analysis model to obtain training phoneme features that match the input phoneme sequence; outputting the predicted phoneme at the masked position based on the training phoneme features; calculating the training loss corresponding to the phoneme analysis model based on the original phoneme and the predicted phoneme at the masked position; and adjusting the parameters of the phoneme analysis model using the training loss until the trained phoneme analysis model is obtained.

[0134] Please see Figure 10 , Figure 10 This is a schematic diagram of one embodiment of the electronic device of this application. The electronic device includes a memory 40 and a processor 50 coupled to each other. The memory 40 stores program instructions, and the processor 50 executes the program instructions to implement the emotion recognition method mentioned in any of the above embodiments. Specifically, the electronic device includes, but is not limited to, desktop computers, laptops, tablets, servers, etc., and is not limited thereto. In addition, the processor 50 may also be called a CPU (Center Processing Unit). The processor 50 may be an integrated circuit chip with signal processing capabilities. The processor 50 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor may be a microprocessor or any conventional processor. In addition, the processor 50 may be implemented by integrated circuit chips.

[0135] Please see Figure 11 , Figure 11 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 60 stores program instructions 70 that can be executed by a processor. When the program instructions 70 are executed by the processor, they implement the emotion recognition method mentioned in any of the above embodiments.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0137] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0138] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0139] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0140] The above are merely embodiments of this application and do not limit the scope of this patent application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. An audio generation method, characterized in that, include: Obtain the target text to be converted, and determine the language information and acoustic information corresponding to the target text; wherein, the language information is related to the text elements of the target text, and the acoustic information is related to the reference spectrum matched by the target text; A first target network matching the language information is determined from multiple first expert networks, and prosodic prediction features matching the language information are obtained using the first target network. A second target network matching the acoustic information is determined from multiple second expert networks, and acoustic prediction features matching the acoustic information are obtained using the second target network. Based on the prosodic prediction features and the acoustic prediction features, the target audio after the target text is converted is generated.

2. The audio generation method according to claim 1, characterized in that, The audio generation method is implemented using a trained audio generation model, which includes a first encoding module and a second encoding module that are coupled to each other. The first encoding module includes multiple first expert networks, a language gating network, and at least one first shared network. The second encoding network includes multiple second expert networks, an acoustic gating network, and at least one second shared network.

3. The audio generation method according to claim 2, characterized in that, The step of determining a first target network that matches the language information from multiple first expert networks, and using the first target network to obtain prosodic prediction features that match the language information, includes: Based on the language information, a first target network is determined from all the first expert networks using the language gating network; wherein each of the first expert networks is used to process a matching fixed task, and the fixed tasks matched by different first expert networks are different; The language information is input into the first target network to obtain the first reference prosodic feature output by the first target network; and... The language information is input into each of the first shared networks to obtain the second reference prosodic features output by each of the first shared networks; Based on the first reference prosodic feature and the second reference prosodic feature, the prosodic prediction feature matching the target text is obtained.

4. The audio generation method according to claim 3, characterized in that, The first encoding module includes multiple first network sets, each first network set including multiple first expert networks. The step of determining the first target network from all the first expert networks based on the language information using the language gating network includes: Based on the language information, a first target set matching the language information is determined from all first network sets using a first language gating network; wherein each first network set matches a task domain related to the language information, and the task domains matched by different first network sets are different; The first target network that matches the language information is determined from all first expert networks corresponding to the first target set using a second language gating network.

5. The audio generation method according to claim 2, characterized in that, The audio generation model further includes a first decoding module and a second decoding module coupled sequentially. The first decoding network includes multiple third expert networks and at least one third shared network, and the second decoding network includes multiple fourth expert networks and at least one fourth shared network. Generating the target audio after converting the target text based on the prosodic prediction features and the acoustic prediction features includes: A third target network matching the prosodic prediction feature and the acoustic prediction feature is determined from all the third expert networks in the first decoding network; Based on the prosodic prediction features and the acoustic prediction features, the initial Mel spectrum is obtained using the third target network and the third shared network; From all the fourth expert networks in the second decoding network, determine the fourth target network that matches the initial Mel spectrum; Based on the initial Mel spectrum, the target Mel spectrum is obtained using the fourth target network and the fourth shared network; Based on the target Mel spectrum, the target audio is obtained.

6. The audio generation method according to claim 2, characterized in that, The training process of the audio generation model includes: Obtain multiple training samples, each of which includes mutually matched training text and training audio; Obtain the training language information of the training text matching and the training acoustic information of the training audio matching in the training samples; The audio generation model is constructed, and the first encoding module in the audio generation model is used to obtain prosodic prediction features based on the training language information, and the second encoding module in the audio generation model is used to obtain acoustic prediction features based on the training acoustic information. Decode the prosodic prediction features and acoustic prediction features corresponding to the training samples to obtain the predicted Mel spectrum that matches the training samples, and convert the predicted Mel spectrum into predicted audio. Based on the predicted audio and the training audio, the audio conversion loss of the audio generation model is calculated, and the parameters in the audio generation model are adjusted using the audio conversion loss until the trained audio generation model is obtained.

7. The audio generation method according to claim 1, characterized in that, The steps for obtaining the language information include: Obtain the language type, word segmentation, and prosodic information corresponding to the target text; wherein, the word segmentation is obtained by splitting the text elements in the target text, and the prosodic information includes the tone of each text element and the pause position of each text element; and, Obtain the timbre features that match the pronunciation object: and, The target text is input into the trained phoneme analysis model to obtain the phoneme features output by the phoneme analysis model based on the phonemes corresponding to each text element; The language type, word segmentation, prosodic information, timbre features, and phoneme features are used as the language information.

8. The audio generation method according to claim 7, characterized in that, The phoneme analysis model is trained using multiple training texts, and the training process of the phoneme analysis model includes: Obtain multiple training texts, and based on the phonemes corresponding to each training element in the training texts, obtain the phoneme sequence corresponding to each training text; At least some phonemes in the phoneme sequence are masked, and the masked phoneme sequence is input into the phoneme analysis model. The phoneme analysis model is used to obtain training phoneme features that match the input phoneme sequence. Based on the trained phoneme features, the predicted phoneme at the mask position is output using the phoneme analysis model. Based on the original and predicted phonemes at the mask position, the training loss corresponding to the phoneme analysis model is calculated, and the parameters of the phoneme analysis model are adjusted using the training loss until the trained phoneme analysis model is obtained.

9. An audio generation system, characterized in that, include: An acquisition module is used to acquire the target text to be converted and determine the language information and acoustic information corresponding to the target text; wherein, the language information is related to the text elements of the target text, and the acoustic information is related to the reference spectrum matched by the target text; The processing module is configured to determine a first target network that matches the language information from multiple first expert networks, obtain prosodic prediction features that match the language information using the first target network, determine a second target network that matches the acoustic information from multiple second expert networks, and obtain acoustic prediction features that match the acoustic information using the second target network. The conversion module is used to generate the target audio after the target text is converted, based on the prosodic prediction features and the acoustic prediction features.

10. An electronic device, characterized in that, include: A memory and a processor are coupled to each other, the memory storing program instructions, and the processor executing the program instructions to implement the method as described in any one of claims 1-8.

11. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the method as described in any one of claims 1-8.

Images