Method, apparatus, device, and product for synthesizing audio
Patent Information
- Application Number
- CN202510176694.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-17
- Publication Date
- 2026-08-18
Smart Images
Figure CN122598604A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computers, and more specifically to methods, apparatus, devices and products for synthesizing audio. Background Technology
[0002] Current audiobook solutions primarily combine manual processing with small-model algorithms. Taking multi-character audiobook interpretation as an example, the human team is responsible for in-depth analysis of the novel's content, identifying the age, personality traits, emotional changes, and roles and functions of different characters. Based on this analysis, a unique voice image is designed for each character, including features such as timbre, tone, speaking speed, and accent. Small-model algorithms are mainly used for text analysis and speech synthesis. Their principle is to make judgments by extracting features and matching patterns from the input data. They have a small parameter scale and a relatively simple structure. Small models can be rule-based models, simple machine learning models (such as decision trees and Naive Bayes), or small neural networks (such as small recurrent neural networks). Summary of the Invention
[0003] In a first aspect of the embodiments of this disclosure, a method for synthesizing audio is provided. The method includes determining multiple characters in text and character information for the multiple characters. The method includes determining the character to which each of the multiple dialogues belongs based on the context of the multiple dialogues in the text. The method also includes determining multiple timbres of the multiple characters based on the character information and the dialogues of the multiple characters. Furthermore, the method includes synthesizing audio of the text based on the multiple timbres of the multiple characters and the multiple dialogues.
[0004] In a second aspect of the embodiments of this disclosure, an apparatus for synthesizing audio is provided. The apparatus includes a role determination module configured to determine multiple roles in text and role information for the multiple roles. The apparatus includes a dialogue attribution determination module configured to determine the role to which each dialogue in the multiple dialogues belongs based on the context of the multiple dialogues in the text. The apparatus also includes a timbre determination module configured to determine multiple timbres for the multiple roles based on the role information and the dialogues of the multiple roles. Furthermore, the apparatus includes an audio synthesis module configured to synthesize audio of the text based on the multiple timbres of the multiple roles and the multiple dialogues.
[0005] In a third aspect of embodiments of this disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement a method for synthesizing audio. The method involves determining multiple characters in text and character information for the multiple characters. The method includes determining the character to which each of the multiple dialogues belongs based on the context of the multiple dialogues in the text. The method also includes determining multiple timbres of the multiple characters based on the character information and the dialogues of the multiple characters. Furthermore, the method includes synthesizing audio of the text based on the multiple timbres of the multiple characters and the multiple dialogues.
[0006] In a fourth aspect of embodiments of this disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to implement a method for synthesizing audio. The method includes determining a plurality of characters in text and character information for the plurality of characters. The method includes determining the character to which each of the plurality of dialogues belongs based on the context of the plurality of dialogues in the text. The method also includes determining a plurality of timbres for the plurality of characters based on the character information and the dialogues of the plurality of characters. Furthermore, the method includes synthesizing audio of the text based on the plurality of timbres of the plurality of characters and the plurality of dialogues.
[0007] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description
[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:
[0009] Figure 1 A schematic diagram of an example environment in which several embodiments of the present disclosure may be implemented is shown;
[0010] Figure 2 A flowchart of a method for synthesizing audio according to some embodiments of the present disclosure is shown;
[0011] Figure 3 A schematic diagram illustrating an example process for synthesizing audio for a novel according to some embodiments of the present disclosure is shown;
[0012] Figure 4A schematic diagram illustrating an example of information extraction from a chapter of a novel according to some embodiments of the present disclosure is shown;
[0013] Figure 5 A schematic diagram illustrating an example of summarizing information about characters in a novel according to some embodiments of the present disclosure is shown;
[0014] Figure 6 A schematic diagram illustrating an example of dialogue / narration segmentation and sentiment annotation in a novel according to some embodiments of the present disclosure is shown;
[0015] Figure 7 A schematic diagram illustrating an example of assigning roles to dialogue in a novel according to some embodiments of the present disclosure is shown;
[0016] Figure 8 A schematic diagram illustrating an example of character design determination for a main character in a novel according to some embodiments of the present disclosure is shown;
[0017] Figure 9 A block diagram of an apparatus for synthesizing audio according to some embodiments of the present disclosure is shown; and
[0018] Figure 10 A block diagram of a device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation
[0019] It is understood that all user-related data involved in this technical solution should be obtained and used only after authorization from the user. This means that if it is necessary to use a user's personal information in this technical solution, the user's explicit consent and authorization are required before obtaining this data; otherwise, no related data collection and use will be carried out. It should also be understood that when implementing this technical solution, relevant laws and regulations should be strictly followed in the process of data collection, use, and storage, and necessary technical measures should be taken to protect user data security and ensure the secure use of data.
[0020] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.
[0021] For example, upon receiving a user's proactive request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.
[0022] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0023] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.
[0024] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0025] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects unless explicitly stated. Other explicit and implicit definitions may also be included below.
[0026] As mentioned above, although combining small models with human processing can achieve multi-role audiobook performances, this approach has several drawbacks, primarily in terms of content quality, production cost, and efficiency. Regarding content quality, for example, the differentiation of characters is not precise enough. During speech synthesis, the small model lacks finesse in distinguishing the timbre, intonation, and emotion of different characters, and it frequently misidentifies the character a dialogue belongs to (e.g., mistaking A's speech for B's), resulting in a mismatch between the timbre and the character, negatively impacting the user's audiobook experience. In terms of production cost and efficiency, for example, manually determining the attributes of characters (gender, age, faction, etc.) leads to high labor costs and low efficiency.
[0027] To this end, this disclosure provides a method for synthesizing audio. In the solution provided in this disclosure, by summarizing all roles and role information in the text, a large language model is used to determine the role to which a dialogue belongs based on the context of multiple dialogues in the text. The role information and the role's dialogue are then used to determine the role's timbre, thereby avoiding manual intervention in the audio synthesis process and improving audio synthesis efficiency and the accuracy of role attribution.
[0028] It should be understood that the technical solutions disclosed herein were developed with the permission of the relevant parties, where permitted by law and regulation. For example, in the field of audio synthesis for novels, this was done after obtaining the copyright of the novel to which the audio is to be synthesized.
[0029] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of this disclosure may be implemented is shown. For example... Figure 1 As shown, text 102 can be a novel containing narration and multiple dialogues between multiple characters; however, not every dialogue may have a clearly defined speaker. To synthesize the multi-character performance audio 118 of text 102, it needs to be provided to a character determination unit 104 and a dialogue attribution determination unit 106. The character determination unit 104 can determine all characters in text 102 and character information associated with these characters, such as name, age, affiliation, etc. The dialogue attribution determination unit 106 can determine the character to which each dialogue belongs based on the context of the multiple dialogues in text 102 (e.g., context content, information contained in the context). Although some embodiments in this application use a novel as an example of text, other forms of text are also possible, and this application does not limit this.
[0030] In environment 100, multiple characters and their corresponding character information 108, determined by the character determination unit 104, and multiple dialogues 110 of the multiple characters, determined by the dialogue attribution determination unit 106, are provided to the timbre determination unit 112. The timbre determination unit 112 determines multiple timbres 114 of the multiple characters in the text 102. The timbres can be, for example, a charming young man's voice, a gentle and sweet voice, or a domineering CEO's voice. The determined multiple timbres 114 of the multiple characters and the multiple dialogues 110 of the multiple characters are provided to the audio synthesis unit 116, which synthesizes the multi-character performance audio 118 of the text 102.
[0031] In environment 100, the functions of each of the role determination unit 104, dialogue attribution determination unit 106, and timbre determination unit 112 can be achieved using a large language model. A large language model (LLM) is a model built on deep learning technology. Through training on large-scale text data, it learns the structure, semantics, and grammatical rules of language, possessing powerful language understanding and generation capabilities. In novel audio synthesis, it can deeply understand the semantics, plot, and character traits of the novel text, helping to generate coherent, logically sound, and emotionally resonant audio content, making character dialogues and scene descriptions in the novel more vivid.
[0032] By identifying the characters and their corresponding information in the text, and determining the character affiliation based on the context of the dialogue, the voice timbre of the characters can be determined. This avoids the need for manual intervention in the audio synthesis process, improves the efficiency of audio synthesis and the accuracy of character affiliation determination, and enhances the user's listening experience.
[0033] Figure 2 A flowchart of a method 200 for synthesizing audio according to some embodiments of the present disclosure is shown. Method 200 can be performed in... Figure 1 The method 200 is performed in the environment 100 shown. It may also include additional operations not shown and / or the operations shown may be omitted. The order of the boxes shown in the figure may be changed. The scope of this disclosure is not limited in this respect.
[0034] In box 202, identify multiple roles in the text and their role information. For example, in Figure 1 In the environment 100 shown, the role determination unit 104 can obtain multiple roles and corresponding role information 108 from the text 102. In some embodiments, the role determination unit 104 can extract a series of information such as the role name, gender, age, character setting, and activities of each chapter or section in the text 102, and summarize this information according to the role name (if the same role has multiple titles, it can be summarized by the full name).
[0035] In box 204, based on the context of multiple dialogues in the text, determine the role to which each dialogue belongs. For example, in Figure 1 In the environment 100 shown, the dialogue attribution determination unit 106 can determine the role to which each dialogue belongs based on the context of the text 102 using LLM. In some embodiments, the dialogue attribution determination unit 106 can segment the text 102 into narration and dialogue, obtain the sentiment of each dialogue, and determine the role to which the dialogue belongs based on the narration before and after the dialogue and other dialogues.
[0036] In box 206, based on character information and dialogue among multiple characters, determine multiple timbres for multiple characters. For example, in Figure 1 In the environment 100 shown, the timbre determination unit 112 can determine the timbre of a character based on multiple characters and corresponding character information 108, as well as multiple dialogues 110 of the multiple characters, using an LLM. In some embodiments, the timbre determination unit 112 can determine the character's personality based on multiple characters and corresponding character information 108, as well as multiple dialogues 110 of the multiple characters, such as (female lead) quirky and clever, (female supporting character) ruthless and shrewish, (male lead) domineering CEO, (male supporting character) sunny and lively, etc., and select a timbre that matches the determined personality from a variety of timbres.
[0037] In box 208, audio for the text is synthesized based on multiple voices and dialogues from multiple characters. For example, in Figure 1 In the environment 100 shown, the audio synthesis unit 116 synthesizes a multi-character performance audio 118 of text 102 from multiple dialogues 110 of multiple characters and multiple timbres 114 of multiple characters. In some embodiments, after acquiring multiple dialogues 110 of multiple characters and multiple timbres 114 of multiple characters, the audio synthesis unit 116 matches the character's timbre to the character's dialogue, and can also adjust parameters such as volume, pitch, and speech rate of the timbre according to the emotions and key words in the sentences, so that when listening, the listener can clearly distinguish each character through different timbres and vividly feel the communication and emotional conflicts between the characters in the novel.
[0038] This method of synthesizing audio from text based on characters and character information, as well as the context of dialogue, avoids the need for manual intervention in the audio synthesis process, improves audio synthesis efficiency and the accuracy of character attribution determination, and enhances the user's audiobook listening experience.
[0039] Figure 3 A schematic diagram of an example process 300 for synthesizing audio for a novel according to some embodiments of the present disclosure is shown. Figure 3 As shown, the example process 300 can be divided into three stages: the first stage is the novel dialogue attribution stage, including boxes 301 to 304; the second stage is the character voice determination stage, including boxes 305 to 309; and the final stage is the novel audio synthesis stage, completed in box 310. The example process 300 may also include additional operations not shown and / or the operations shown may be omitted. The order of the boxes shown in the figure can be changed, and the scope of this disclosure is not limited in this respect.
[0040] In box 301, information extraction is performed on each chapter of the novel. In some embodiments, a depth-first search (DFS) method is used to extract the characters and their corresponding information in each chapter. The specific operations of DFS in character and information extraction can include starting node selection, a deep exploration phase, a backtracking process, and continuing to explore other branches. In starting node selection, traversal typically begins from the beginning of the text, such as using the first paragraph or the first sentence as the starting node. In the deep exploration phase, starting from the starting node, the text is explored sequentially. When a word that might be a person's name is encountered (this can be determined through a pre-established name database or by naming rules, such as names being nouns and having a capital first letter), it is marked as a potential character. Then, content related to this character is explored in depth, such as collecting descriptions of the character's characteristics, such as age, gender, aliases, descriptions of the character's appearance, and descriptions of the character's psychological and behavioral activities. During the backtracking process, when no more information about the character can be obtained on a path, the traversal returns to the previous node. After backtracking, a new unexplored branch is selected to continue the traversal. For example, after rewinding from one chapter, you can start exploring the content of the next chapter to find new characters or other useful information.
[0041] To visually demonstrate the results of chapter information extraction, through... Figure 4 This diagram illustrates an example 400 of information extraction from a chapter of a novel. In example 400, depth-first search extracts several characters from a specific chapter, such as character A and character B. Under each character, their alias, gender, age, age basis, number of lines spoken, appearance, character setting, and behavior are listed. This information is crucial for subsequently determining the character's affiliation in dialogue and their vocal tone. For character A, depth-first search reveals her gender to be female, her age to be young (the age basis is not explicitly mentioned in the text), and based on her identity and experience, her number of lines spoken is 74. Her appearance is upright, and her character setting is resolute and brave, transforming from timid to assertive. Her behavior is related to event E with XX. For character B, depth-first search reveals his gender to be male, his age to be middle-aged (the age basis is not explicitly mentioned in the text), and based on his identity and children, his number of lines spoken is 25. His appearance is villainous, and his character setting is ruthless and heartless. His behavior is related to event D with XX.
[0042] return Figure 3In box 302, the character information for the entire book is summarized. In some embodiments, the characters and character information obtained through box 301 have a high degree of overlap. For example, each chapter may have a character A, and character A may have multiple aliases in the novel. Therefore, it is necessary to summarize all the character information that actually comes from the same character A into the character information column of character A to improve the efficiency of subsequent processes. Various merging strategies can be adopted to summarize information, including but not limited to precise matching merging based on character name (name matching merging, alias matching merging), merging strategies based on character features and attributes (feature similarity merging, character relationship clue merging), merging strategies based on plot clues and event associations (event participation merging, plot coherence merging), and merging strategies based on text position and context associations (adjacent text merging, context reference merging).
[0043] To visually demonstrate the results of summarizing character information across all chapters, we will use... Figure 5 This diagram illustrates an example 500 of summarizing information about characters in a novel. In example 500, a merging strategy is used to summarize character information about character A from each chapter into a character A information column. For example... Figure 5 As shown, character A has three aliases: Alias 1, Alias 2, and Alias 3. She is a protagonist, female, and her age ranges from youth to middle age. She speaks 4764 lines and belongs to the following factions: good guys, bad guys, good guys. Her character traits are: "(Female lead) quirky and clever," "(Female lead) resolute," "(Female lead) gentle and cute," etc. Her actions are summarized as: "Action A, Action B, Action C," etc. Her personality is described as: "resolute and brave, transforming from timid to assertive," "outspoken," "straightforward and lively," "brave and decisive," etc. This compiled character information will help to more accurately determine the character's age and faction, and will also help to determine the character's character traits and thus their vocal tone.
[0044] return Figure 3 In box 303, the novel is segmented into dialogue / narration, and the emotion of each line of dialogue is obtained. In some embodiments, the methods for segmenting dialogue / narration include, but are not limited to, using punctuation and format (generally, content enclosed in quotation marks can be identified as dialogue, or dialogue and narration can be distinguished based on special formats, such as dialogue sections on a separate line), analyzing sentence structure and grammar (finding verbs that indicate speaking actions, such as "say," "ask," "answer," "mutter," etc., to determine the subject and object of the sentence), and semantic understanding (when there are no obvious dialogue markers, semantic analysis is used to determine the emotion). In some embodiments, the methods for assigning emotion to dialogue include, but are not limited to, analysis based on vocabulary and interjections, combining context and character traits, and using facial expressions and action descriptions to assist in the determination.
[0045] To visually demonstrate the results of segmenting a novel into dialogue / narration and imbuing the dialogue with emotion, through... Figure 6 This diagram illustrates an example 600 of segmenting a novel into dialogue / narration and assigning emotional labels. In example 600, in the presented novel content, “Chapter 1 Title,” “Setting Description 1,” “Character Expression Description,” “Setting Description 2,” and “Setting Description 3” are labeled as narration, sentences 601 to 607 are labeled as dialogue, the emotional column of the narration is assigned the value “neutral,” the emotional column of sentences 601 to 602 is assigned the value “angry,” sentences 603 to 605 are assigned the value “neutral,” and sentences 606 to 607 are assigned the value “terrified.”
[0046] return Figure 3 In box 304, character attribution is performed on dialogues in the novel. In some embodiments, a supervised fine-tuning (SFT) model can be used for dialogue attribution determination. The SFT model is a model that undergoes supervised fine-tuning based on a pre-trained language model. It understands the syntax and semantics of dialogue sentences, which helps to better analyze the features of dialogue sentences, thereby improving the accuracy of dialogue attribution determination. For example, it can use its understanding of lexical semantics to determine that sentences with certain words are more likely to belong to a particular character. Furthermore, its over-supervised fine-tuning allows the SFT model to be specifically optimized for dialogue attribution determination tasks.
[0047] In some embodiments, the SFT model can determine the role to which a dialogue belongs based on several sentences of contextual narration preceding and following the dialogue, along with the dialogue itself. For example, for each dialogue to be judged, its first two sentences of contextual narration and its last two sentences of contextual narration, along with the dialogue itself, are input into the SFT model to obtain contextual attention, thereby improving the accuracy of role attribution determination. Figure 6 For example, dialogue 1 is the dialogue to be judged, and the content that needs to be input into the SFT model includes narration 1, narration 2, dialogue 1, narration 3, and narration 4.
[0048] In some embodiments, the SFT model can also simultaneously determine the role to which each dialogue in several sentences belongs. Since judging only one dialogue at a time would miss contextual information, several consecutive dialogues can be judged simultaneously in a single judgment process to achieve discourse-level understanding through the SFT model, further improving the role determination capability. For example, 30 consecutive dialogues can be simultaneously input into the SFT model, and the role to which each of these 30 dialogues belongs can be determined at the same time.
[0049] In some embodiments, the SFT model can also make judgments based on character behavior information. The dialogue attribution task inevitably reduces the SFT model's ability to understand the plot development of a novel; therefore, providing the model with character behavior information helps aid its understanding. For example, the behavioral summary of the character information obtained after executing box 302 can be input into the SFT model to make more accurate judgments based on these behaviors. A character's behavior could include their stance on which events they participated in.
[0050] To visually demonstrate the results of the role attribution determination, through... Figure 7 This diagram illustrates an example 700 of character attribution for dialogue in a novel. In example 700, the character attribution result includes a speaker column, a text content column, and a sentiment column. In the speaker column, the SFT model identifies dialogues 1, 3, and 6 from example 600 as spoken by character A and assigns them the attribute of character A; identifies dialogue 2 as spoken by character B and assigns it the attribute of character B; identifies dialogues 4 and 7 as spoken by character C and assigns them the attribute of character C; and identifies dialogue 5 as spoken by character D and assigns it the attribute of character D.
[0051] return Figure 3 In box 305, the main characters in the novel are identified. Box 305 marks the beginning of the character voice determination stage. Since novels often contain many characters, but not all are main characters—some may only speak a few lines—the appropriateness of the voice for these minor characters has little impact on the listener's experience. Therefore, to improve audio synthesis efficiency, it's unnecessary to perform a voice determination process for minor characters. Instead, it's necessary to identify the main characters. In some embodiments, characters can be sorted in descending order based on the number of lines spoken in the compiled character information. The top n characters (where n is an integer and can be flexibly set according to the number of characters in the novel) are identified as main characters, and a voice selection process is performed for these main characters in subsequent steps. The voices of minor characters can be assigned by the audio synthesis software during audio synthesis. This method improves audio synthesis efficiency and ensures that the voices of the main characters are appropriately matched, enhancing the user's listening experience.
[0052] In box 306, for the identified main characters, age-related auxiliary information is extracted from the novel. In some embodiments, age-related auxiliary information can be extracted using LLM based on the novel's genre. For example, in fantasy / cultivation genres, the focus is on extracting physical descriptions, while in ancient genres, the focus is on extracting generational information. Age-related auxiliary information may include, but is not limited to, time span markers (division of growth stages, historical period relevance), character relationship representation (description of intergenerational relationships, peer group division), character personality development (typical personality traits at different age stages), and plot hints (abilities and experience, appearance, clothing, voice characteristics, social status, etc.).
[0053] In box 307, the age / faction of the main characters is determined. In some embodiments, the age / faction of the main characters is determined based on age auxiliary information extracted by the LLM and the aggregated character information. For example, a voting model can be built within the LLM, using multiple neurons or sub-models to determine the age and faction of the characters separately, with each neuron or sub-model acting as a "voter." These "voters" make their judgments based on their understanding of text features and pre-trained weights. Furthermore, the LLM can assign weights to different text features to influence the voting results. For age determination, strong cues such as the character explicitly stating their age can be given higher weights; for faction determination, descriptions of the character's key moral choice scenarios can be given higher weights. The voting results are then statistically analyzed. For example, for age determination, the number of "voters" who believe the character is a teenager, young adult, middle-aged, or elderly is counted, and the probability of voting for each age group is calculated. For faction determination, the probability of voting for factions such as justice, villain, and neutral is calculated.
[0054] In box 308, the character profile of the main character is determined. In some embodiments, the character profile is determined based on the main character's age, faction, other information in the aggregated character information, and the character's dialogue. For example, various character profiles for the same character can be statistically analyzed in LLM, and a score can be calculated for each profile. To visually demonstrate the execution results of boxes 307 and 308, the following steps are taken: Figure 8This example 800 illustrates a characterization assessment of a main character in a novel. In example 800, the main character A's age is {Vote: "Young adult", LLM judgment: "Young adult", vote percentage: 0.992564, age basis: "Chapter 102: Event C occurs in the novel. Chapter 103 does not explicitly mention it, but based on the activities and state, the character is judged to be young adult"...}, her faction is {Vote: "Honest", LLM judgment: "Honest", vote percentage: 0.995327}, and her characterization is {"(Female lead) Quirky and clever", judgment basis: {Label: "(Female lead) Quirky and clever", analysis: often has various unexpected actions and ideas, full of clever ideas, score: 9}, {Label: "(Female lead) Resolute personality", analysis: opinionated, unwilling to admit defeat, actively responds to various situations, score: 8}...}.
[0055] In box 309, select the voice timbre for the main character. In some embodiments, a suitable voice timbre can be selected from the voice timbre library based on the main character's personality and dialogue. For example, for the "(female lead) quirky and clever" personality, a clear and sweet girlish voice, a lively and playful child's voice, or a lively and varied falsetto can be matched. The voice timbre can be flexibly selected according to the dialogue scene and the emotions accompanying the dialogue. For example, a clear and sweet girlish voice can be selected for dialogue in everyday conversation scenes, a lively and playful child's voice can be selected for lighthearted and humorous scenes, and a lively and varied falsetto can be selected for scenes involving role-playing or deceiving opponents.
[0056] In box 310, multi-character audio from the novel is synthesized. In some embodiments, the pre-processed novel text and selected timbre parameters are imported into audio synthesis software. Then, following the dialogue order in the novel, the corresponding timbre is applied to the dialogue of each character for synthesis. During the synthesis process, the pauses between dialogues are adjusted to conform to the normal rhythm of conversation. The speech rate, volume, and tone are adjusted according to the emotion of the dialogue; for example, when a character is angry, the volume is increased, the speech rate is accelerated, and the timbre is made tense; when a character is sad, the volume is decreased, the speech rate is slowed, and the timbre sounds heavier. For the narration, a suitable timbre or style is selected for synthesis. The narration timbre can be relatively neutral and steady to clearly convey scene and plot information. The speech rate of the narration can also be adjusted according to the tension of the plot, with a slightly faster speech rate in tense scenes and a slightly slower speech rate in calm scenes. Appropriate background music and sound effects are selected according to the plot and scene of the novel to enhance the expressiveness of the audio. For example, in tense fight scenes, add intense battle sound effects and suspenseful background music; in romantic scenes, add soft music and warm environmental sound effects, such as birdsong and flowing water. Finally, adjust the overall volume and sound quality of the synthesized audio to ensure a balance between the dialogue, narration, background music, and sound effects of each character.
[0057] The audio synthesis method shown in some embodiments of this disclosure improves the attribution accuracy from 95% to 97.5% compared to the small model + manual attribute judgment scheme, and reduces the faction judgment error rate from 8% to 3%. In the manual listening and review process, the timbre matching degree, the overall listening experience of the chapter, and the overall listening experience of the book are all greatly improved.
[0058] Figure 9 A block diagram of an apparatus 900 for synthesizing audio according to some embodiments of the present disclosure is shown. Figure 9 As shown, device 900 includes a role determination module 902, configured to determine multiple roles in text and role information for the multiple roles. Device 900 also includes a dialogue attribution determination module 904, configured to determine the role to which each dialogue belongs based on the context of the multiple dialogues in the text. Device 900 also includes a timbre determination module 906, configured to determine multiple timbres for the multiple roles based on the role information and the dialogues of the multiple roles. Furthermore, device 900 includes an audio synthesis module 908, configured to synthesize audio from the text based on the multiple timbres of the multiple roles and the multiple dialogues.
[0059] Figure 10 A block diagram of a device 1000 capable of implementing various embodiments of the present disclosure is shown. For example... Figure 10 As shown, device 1000 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 1001, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 1002 or loaded from storage unit 1008 into random access memory (RAM) 1003. Various programs and data required for the operation of device 1000 can also be stored in RAM 1003. The CPU / GPU 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004. Although not shown in... Figure 10 As shown, device 1000 may also include a coprocessor.
[0060] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0061] The various methods or processes described above can be executed by CPU / GPU 1001. For example, in some embodiments, the methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by CPU / GPU 1001, one or more steps or actions in the methods or processes described above can be performed.
[0062] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.
[0063] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.
[0064] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.
[0065] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages and conventional procedural programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.
[0066] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0067] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0068] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0069] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.
[0070] The following are some example implementations of this disclosure.
[0071] Example 1. A method for synthesizing audio, comprising:
[0072] Identify multiple characters in the text and their character information;
[0073] Based on the context of multiple dialogues in the text, determine the role to which each of the multiple dialogues belongs;
[0074] Based on the character information and the dialogue of the multiple characters, determine multiple timbres for the multiple characters; and
[0075] The audio of the text is synthesized based on the multiple timbres of the multiple characters and the multiple dialogues.
[0076] Example 2. According to the method described in Example 1, determining the multiple characters in the text and the character information of the multiple characters includes:
[0077] By performing a depth-first traversal, the characters and character information for each chapter in the text are extracted; and
[0078] Based on the roles and role information of each chapter, the multiple roles in the text and the role information of the multiple roles are determined by merging the role information of each chapter by role.
[0079] Example 3. The method according to any one of Examples 1 to 2, wherein determining the role to which each of the plurality of dialogues belongs based on the context of the plurality of dialogues in the text includes:
[0080] Based on a predetermined number of sentences of contextual narration preceding and following each of the plurality of dialogues, and each dialogue itself, the role to which each of the plurality of dialogues belongs is determined by a language model.
[0081] Example 4. The method according to any one of Examples 1 to 3, wherein determining the role to which each of the plurality of dialogues belongs based on the context of the plurality of dialogues in the text includes:
[0082] Based on the multiple dialogues in the aforementioned text, a language model is used to simultaneously determine the role to which each dialogue belongs in a predetermined number of sentences of the multiple dialogues.
[0083] Example 5. The method according to any one of Examples 1 to 4, wherein determining the role to which each of the plurality of dialogues belongs based on the context of the plurality of dialogues in the text includes:
[0084] For each chapter in the text, summarize the behaviors of the multiple characters; and
[0085] Based on the behavior of the multiple roles, the role to which each of the multiple dialogues belongs is determined by a language model.
[0086] Example 6. The method according to any one of Examples 1 to 5, wherein determining multiple timbres of the multiple characters based on the character information and the dialogue of the multiple characters includes:
[0087] Based on the character information and the dialogue of the multiple characters, multiple personas of the multiple characters are determined through a language model; and
[0088] Based on the multiple character settings and the dialogues of the multiple characters, the multiple voice timbres of the multiple characters are determined.
[0089] Example 7. The method according to any one of Examples 1 to 6, wherein determining multiple personas of the multiple characters based on the character information and the dialogue of the multiple characters includes:
[0090] Based on the dialogues of the multiple characters, determine the number of dialogues for each character;
[0091] Based on the number of dialogues, determine the order of the multiple roles; and
[0092] Based on the order of the multiple characters, the character settings of the multiple characters are determined.
[0093] Example 8. The method according to any one of Examples 1 to 7, further comprising determining the character settings of the plurality of characters based on their order, includes:
[0094] Based on the ranking of the multiple roles, the main roles are determined;
[0095] For the main character, age auxiliary information is extracted based on the subject matter of the text using the language model;
[0096] Based on the age assistance information, the age of the main character is determined;
[0097] Based on the character information, determine the faction of the main character; and
[0098] The character design of the main character is determined based on the age, faction, other character information in the character information, and the dialogue.
[0099] Example 9. The method according to any one of Examples 1 to 8, wherein the text is a novel, and the character information includes at least one of name, alias, age, gender, character setting, appearance, behavior, and faction.
[0100] Example 10. An apparatus for synthesizing audio, comprising:
[0101] The role determination module is configured to determine multiple roles in the text and the role information of the multiple roles;
[0102] The dialogue attribution determination module is configured to determine the role to which each of the multiple dialogues belongs based on the context of the multiple dialogues in the text.
[0103] A timbre determination module is configured to determine multiple timbres for the multiple characters based on the character information and the dialogue of the multiple characters; and
[0104] An audio synthesis module is configured to synthesize audio of the text based on the multiple timbres of the multiple characters and the multiple dialogues.
[0105] Example 11. The apparatus according to Example 10, wherein the role determination module includes:
[0106] The information extraction module is configured to extract the characters and character information for each chapter in the text through the depth-first traversal; and
[0107] The information merging module is configured to determine the multiple roles in the text and the role information of the multiple roles by merging the role information of each chapter based on the roles and role information of each chapter.
[0108] Example 12. The apparatus according to any one of Examples 10-11, wherein the dialogue attribution determination module comprises:
[0109] The first attribution determination module is configured to determine the role to which each dialogue belongs in the plurality of dialogues based on a predetermined number of contextual narrations preceding and following each dialogue and each dialogue, using a language model.
[0110] Example 13. The apparatus according to any one of Examples 10-12, wherein the dialogue attribution determination module further comprises:
[0111] The second attribution determination module is configured to determine the role to which each dialogue belongs in a predetermined number of sentences of the multiple dialogues in the text is based on the multiple dialogues in the text, using a language model.
[0112] Example 14. The apparatus according to any one of Examples 10-13, wherein the dialogue attribution determination module further comprises:
[0113] The behavior summary module is configured to summarize the behaviors of the multiple roles for each chapter in the text; and
[0114] The third attribution determination module is configured to determine the role to which each of the multiple dialogues belongs based on the behavior of the multiple roles using a language model.
[0115] Example 15. The apparatus according to any one of Examples 10-14, wherein the timbre determination module further comprises:
[0116] The character designation module is configured to determine multiple character designs for the multiple characters based on the character information and their dialogue, using a language model; and
[0117] The timbre determination submodule is configured to determine the multiple timbres of the multiple characters based on the multiple character settings and the dialogue of the multiple characters.
[0118] Example 16. The apparatus according to any one of Examples 10-15, wherein the person setting determination module comprises:
[0119] The dialogue quantity determination module is configured to determine the dialogue quantity of each role based on the dialogues of the multiple roles.
[0120] A role ranking module is configured to determine the ranking of the multiple roles based on the number of dialogues; and
[0121] The first character designation module is configured to determine the character design of the multiple characters based on their ranking.
[0122] Example 17. The apparatus according to any one of Examples 10-16, wherein the person determination submodule further comprises:
[0123] The main role determination module is configured to determine the main role based on the sorting of the multiple roles;
[0124] The age information extraction module is configured to extract age auxiliary information based on the subject matter of the text using the language model for the main character.
[0125] An age determination module is configured to determine the age of the main character based on the age assistance information;
[0126] A faction determination module is configured to determine the faction of the main character based on the character information; and
[0127] The second character designation module is configured to determine the character designation of the main character based on the age, the faction, other character information in the character information, and the dialogue.
[0128] Example 18. An electronic device comprising:
[0129] Processor; and
[0130] A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform actions, the actions including:
[0131] Identify multiple characters in the text and their character information;
[0132] Based on the context of multiple dialogues in the text, determine the role to which each of the multiple dialogues belongs;
[0133] Based on the character information and the dialogue of the multiple characters, determine multiple timbres for the multiple characters; and
[0134] The audio of the text is synthesized based on the multiple timbres of the multiple characters and the multiple dialogues.
[0135] Example 19. The electronic device according to Example 18, wherein determining multiple characters in the text and character information of the multiple characters includes:
[0136] By performing a depth-first traversal, the characters and character information for each chapter in the text are extracted; and
[0137] Based on the roles and role information of each chapter, the multiple roles in the text and the role information of the multiple roles are determined by merging the role information of each chapter by role.
[0138] Example 20. An electronic device according to any one of Examples 18-19, wherein determining the role to which each of the plurality of dialogues belongs based on the context of the plurality of dialogues in the text includes:
[0139] Based on a predetermined number of sentences of contextual narration preceding and following each of the plurality of dialogues, and each dialogue itself, the role to which each of the plurality of dialogues belongs is determined by a language model.
[0140] Example 21. An electronic device according to any one of Examples 18-20, wherein determining the role to which each of the plurality of dialogues belongs based on the context of the plurality of dialogues in the text further includes:
[0141] Based on the multiple dialogues in the aforementioned text, a language model is used to simultaneously determine the role to which each dialogue belongs in a predetermined number of sentences of the multiple dialogues.
[0142] Example 22. An electronic device according to any one of Examples 18-21, wherein determining the role to which each of the plurality of dialogues belongs based on the context of the plurality of dialogues in the text includes:
[0143] For each chapter in the text, summarize the behaviors of the multiple characters; and
[0144] Based on the behavior of the multiple roles, the role to which each of the multiple dialogues belongs is determined by a language model.
[0145] Example 23. An electronic device according to any one of Examples 18-22, wherein determining multiple timbres of the multiple characters based on the character information and the dialogue of the multiple characters includes:
[0146] Based on the character information and the dialogue of the multiple characters, multiple personas of the multiple characters are determined through a language model; and
[0147] Based on the multiple character settings and the dialogues of the multiple characters, the multiple voice timbres of the multiple characters are determined.
[0148] Example 24. An electronic device according to any one of Examples 18-23, wherein determining multiple personas of the multiple characters based on the character information and the dialogue of the multiple characters includes:
[0149] Based on the dialogues of the multiple characters, determine the number of dialogues for each character;
[0150] Based on the number of dialogues, determine the order of the multiple roles; and
[0151] Based on the order of the multiple characters, the character settings of the multiple characters are determined.
[0152] Example 25. An electronic device according to any one of Examples 18-24, wherein determining the character settings of the plurality of characters based on the order of the plurality of characters further includes:
[0153] Based on the ranking of the multiple roles, the main roles are determined;
[0154] For the main character, age auxiliary information is extracted based on the subject matter of the text using the language model;
[0155] Based on the age assistance information, the age of the main character is determined;
[0156] Based on the character information, determine the faction of the main character; and
[0157] The character design of the main character is determined based on the age, faction, other character information in the character information, and the dialogue.
[0158] Example 26. A computer-readable storage medium having stored thereon computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of Examples 1 to 9.
[0159] Example 27. A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of Examples 1 to 9.
[0160] Although this disclosure has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.
Claims
1. A method for synthesizing audio, comprising: Identify multiple characters in the text and their character information; Based on the context of multiple dialogues in the text, determine the role to which each of the multiple dialogues belongs; Based on the character information and the dialogue of the multiple characters, determine multiple timbres of the multiple characters; as well as The audio of the text is synthesized based on the multiple timbres of the multiple characters and the multiple dialogues.
2. The method according to claim 1, wherein determining the plurality of characters in the text and the character information of the plurality of characters includes: By using a depth-first traversal, the characters and character information of each chapter in the text are extracted; as well as Based on the roles and role information of each chapter, the multiple roles in the text and the role information of the multiple roles are determined by merging the role information of each chapter by role.
3. The method of claim 1, wherein determining the role to which each of the plurality of dialogues belongs based on the context of the plurality of dialogues in the text comprises: Based on a predetermined number of sentences of contextual narration preceding and following each of the plurality of dialogues, and each dialogue itself, the role to which each of the plurality of dialogues belongs is determined by a language model.
4. The method of claim 1, wherein determining the role to which each of the plurality of dialogues belongs based on the context of the plurality of dialogues in the text comprises: Based on the multiple dialogues in the aforementioned text, a language model is used to simultaneously determine the role to which each dialogue belongs in a predetermined number of sentences of the multiple dialogues.
5. The method of claim 1, wherein determining the role to which each of the plurality of dialogues belongs based on the context of the plurality of dialogues in the text comprises: For each chapter in the text, summarize the behaviors of the multiple characters; as well as Based on the behavior of the multiple roles, the role to which each of the multiple dialogues belongs is determined by a language model.
6. The method according to claim 1, wherein determining the multiple timbres of the multiple characters based on the character information and the dialogue of the multiple characters includes: Based on the character information and the dialogue of the multiple characters, multiple personas of the multiple characters are determined through a language model; as well as Based on the multiple character settings and the dialogues of the multiple characters, the multiple voice timbres of the multiple characters are determined.
7. The method according to claim 6, wherein determining multiple personas of the multiple characters based on the character information and the dialogue of the multiple characters includes: Based on the dialogues of the multiple characters, determine the number of dialogues for each character; Based on the number of dialogues, determine the order of the multiple roles; as well as Based on the order of the multiple characters, the character settings of the multiple characters are determined.
8. The method according to claim 7, wherein determining the character settings of the plurality of characters based on their order further comprises: Based on the ranking of the multiple roles, the main roles are determined; For the main character, age auxiliary information is extracted based on the subject matter of the text using the language model; Based on the age assistance information, the age of the main character is determined; Based on the character information, the faction of the main character is determined; as well as The character design of the main character is determined based on the age, faction, other character information in the character information, and the dialogue.
9. The method according to claim 1, wherein the text is a novel, and the character information includes at least one of name, alias, age, gender, character setting, appearance, behavior, and faction.
10. An apparatus for synthesizing audio, comprising: The role determination module is configured to determine multiple roles in the text and the role information of the multiple roles; The dialogue attribution determination module is configured to determine the role to which each of the multiple dialogues belongs based on the context of the multiple dialogues in the text. A timbre determination module is configured to determine multiple timbres for the multiple characters based on the character information and the dialogue of the multiple characters; as well as An audio synthesis module is configured to synthesize audio of the text based on the multiple timbres of the multiple characters and the multiple dialogues.
11. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1 to 9.
12. A computer program product tangibly stored on a non-transitory computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 9.