Audio generation method, device, and readable storage medium for an audiobook

Abstract

Embodiments of the present application disclose an audio generation method and device for audio books and a computer readable storage medium, which are used to generate audio books while improving the accuracy of audio generation of the audio books. The method comprises obtaining a target text of an audio book, inputting the target text into a character segmentation model, identifying a corresponding adjacent relationship between each two adjacent sentences in the target text by the character segmentation model, determining a speaking character prompt sentence corresponding to each dialogue sentence in the target text according to at least one adjacent relationship corresponding to the target text, extracting a speaking character in the speaking character prompt sentence corresponding to each dialogue sentence, merging the speaking character, obtaining a target character and text content corresponding to the target character output by the character segmentation model, and performing audio processing and effect post-processing on the text content corresponding to each target character to obtain audio.

Description

Technical Field

[0001] This application relates to the field of audio generation for audiobooks, and more specifically, to audio generation methods, audio generation devices, and computer-readable storage media for audiobooks. Background Technology

[0002] Thanks to the development of the internet, accessing and disseminating classic literary works, as well as the production, creation, and consumption of online novels, are now much more convenient and efficient than ever before. This rich literary content has also provided a continuous stream of textual material for the long-form audio field. People can manually transform classic works, novels, essays, and other literary pieces into audiobooks, allowing users to switch from reading to listening and freeing their eyes.

[0003] The existing audio generation method for audiobooks involves first cleaning the target text of the audiobook. The cleaning process includes distinguishing the various characters in the target text and the text content corresponding to each character. After cleaning, the text content corresponding to each character can be recorded to obtain the audio for each character. Finally, the audio for each character is processed by audio editing to obtain the final audio.

[0004] However, existing audiobook audio generation methods mainly rely on a fully manual process. When cleaning the target text to distinguish the various characters and their corresponding text content, the process is only roughly judged and manually operated. When segmenting complex characters in an audiobook, errors in human judgment or manual operation are prone to occur, resulting in low accuracy in character segmentation. Therefore, the accuracy of audiobook audio generation is low. Summary of the Invention

[0005] This application provides an audiobook audio generation method, an audiobook audio generation device, and a computer-readable storage medium, which are used to generate audio for audiobooks while improving the accuracy of audio generation.

[0006] In a first aspect, embodiments of this application provide an audio generation method for audiobooks, including:

[0007] Obtain the target text of the audiobook;

[0008] The target text is input into a pre-trained role segmentation model, which identifies the adjacency relationship between every two adjacent sentences in the target text. Based on at least one adjacency relationship in the target text, the model determines the speaking role prompt for each dialogue sentence in the target text. The speaking role is extracted from the speaking role prompt for each dialogue sentence, and the speaking roles are merged to obtain the target role and the text content corresponding to the target role. The target role and the text content corresponding to the target role are output by the pre-trained role segmentation model.

[0009] For each target character, the text content corresponding to the target character is processed into audio to obtain the audio of the target character.

[0010] The audio of each target character is processed with sound effects and then post-processed to obtain the post-processed audio.

[0011] Optionally, the pre-trained role segmentation model includes a pre-trained role recognition model, a prompt recognition model, a speaking role extraction model, and a role merging model;

[0012] The pre-trained role segmentation model identifies the adjacency relationship between every two adjacent sentences in the target text, determines the speaking role cue for each dialogue sentence in the target text based on at least one adjacency relationship, extracts the speaking role from the speaking role cue for each dialogue sentence, and merges the speaking roles to obtain the target role and the corresponding text content, including:

[0013] The pre-trained role recognition model identifies at least one role in the target text, and for each role, the role literal value of the role is replaced with the target role symbol to obtain the replaced target text;

[0014] The pre-trained prompt recognition model identifies the adjacency relationship between every two adjacent sentences in the replaced target text, and infers the speaking role prompt corresponding to each dialogue sentence in the target text based on at least one adjacency relationship corresponding to the target text. The adjacency relationship between every two adjacent sentences includes prompt-dialogue relationship, dialogue-prompt relationship, dialogue-response relationship, dialogue-continuation relationship, dialogue-multiple responses relationship, prompt-interpretation relationship, and / or no relationship.

[0015] The pre-trained speaking role extraction model extracts speaking roles from the speaking role prompts;

[0016] The pre-trained character merging model is used to merge the speaking characters to obtain the target character and the text content corresponding to the target character.

[0017] Optionally, before the pre-trained prompt recognition model identifies the adjacency relationship between every two adjacent sentences in the replaced target text, and infers between every two adjacent sentences based on at least one adjacency relationship corresponding to the target text to obtain the speaking role prompt corresponding to each dialogue sentence in the target text, the method further includes:

[0018] Obtain a target text sample; wherein, the target text sample is marked with the adjacency relationship between every two sentences; the target text sample is the text sample obtained by replacing the literal value of each character with the target character symbol;

[0019] The target text sample is input into the prompt sentence recognition model to obtain the predicted adjacency relationship between each adjacent sentence in the target text sample output by the prompt sentence recognition model;

[0020] The loss between the predicted adjacency relationship and the labeled adjacency relationship is calculated based on the regression loss function. When the loss meets the convergence condition, the trained prompt recognition model is obtained.

[0021] Optionally, the target text includes at least one chapter;

[0022] Before inputting the target text into the pre-trained role segmentation model, the method further includes:

[0023] The target text is segmented into chapters to obtain at least one chapter of the target text;

[0024] The step of inputting the target text into a pre-trained role segmentation model includes:

[0025] For each chapter of the target text, the chapter is input into the pre-trained character segmentation model.

[0026] Optionally, the step of segmenting the target text into chapters to obtain at least one chapter of the target text includes:

[0027] The target text is divided into chapters using a preset regular expression to obtain at least one chapter of the target text.

[0028] Optionally, for each target character, the step of performing audio processing on the text content corresponding to the target character to obtain the audio processed for the target character includes:

[0029] For each target role, determine the ID of the target role;

[0030] The ID of the target character and the text content corresponding to the target character are input into a pre-trained audio model. The pre-trained audio model performs audio processing on the text content corresponding to the target character, and the audio corresponding to the target character is output by the pre-trained audio model.

[0031] Optionally, before inputting the ID of the target character and the text content corresponding to the target character into the pre-trained audio model, the method further includes:

[0032] Obtain at least one character sample, as well as the ID corresponding to each character sample and the text content sample corresponding to each character sample; the ID corresponding to each character sample is labeled with the audio features of real speech;

[0033] For each character sample, the ID corresponding to the character sample and the text content sample corresponding to the character sample are input into the audio model to obtain the predicted audio features output by the audio model.

[0034] The loss between the predicted audio features and the labeled audio features is calculated based on the regression loss function. When the loss meets the convergence condition, the trained audio model is obtained.

[0035] Optionally, the step of performing post-processing on the audio corresponding to each target character to obtain the post-processed audio includes:

[0036] The audio of each target character's voice-processed audio is aligned to the audio track to obtain the aligned audio; the aligned audio is the audio after the post-processing of the effect.

[0037] Optionally, the audio processed by the voice-over process for each target character includes N audio sentences; where N is an integer greater than or equal to 2.

[0038] The step of performing post-processing on the audio corresponding to each target character to obtain the post-processed audio includes:

[0039] For each audio sentence, determine the beginning semantics, keywords, and ending semantics of the audio sentence;

[0040] The first pause interval between the nth audio sentence and the (n-1)th audio sentence is determined based on the beginning semantics corresponding to the nth audio sentence; and the second pause interval between the nth audio sentence and the (n+1)th audio sentence is determined based on the ending semantics corresponding to the nth audio sentence; where 2≤n≤N;

[0041] The scene sound effect corresponding to the nth audio sentence is determined based on the keywords corresponding to the nth audio sentence;

[0042] The first pause interval is used as the pause interval between the nth audio sentence and the (n-1)th audio sentence, and the second pause interval is used as the pause interval between the nth audio sentence and the (n+1)th audio sentence. The scene sound effect is added to the nth audio sentence to obtain N audio sentences with pause intervals and scene sound effect processing. The pause intervals and N audio sentences with scene sound effect processing are used as the audio after post-processing of the effect.

[0043] Secondly, embodiments of this application provide an audio generation device for audiobooks, comprising:

[0044] The acquisition unit is used to acquire the target text of the audiobook;

[0045] The input unit is used to input the target text into a pre-trained role segmentation model, which identifies the adjacency relationship between every two adjacent sentences in the target text, determines the speaking role prompt sentence corresponding to each dialogue sentence in the target text based on at least one adjacency relationship corresponding to the target text, extracts the speaking role from the speaking role prompt sentence corresponding to each dialogue sentence, and performs merging processing on the speaking roles to obtain the target role and the text content corresponding to the target role, thus obtaining the target role and the text content corresponding to the target role output by the pre-trained role segmentation model.

[0046] The audio processing unit is used to process the text content corresponding to each target character into audio for each target character, so as to obtain the audio processed by the target character.

[0047] The post-processing unit is used to perform post-processing on the audio of each target character after it has been processed into sound, so as to obtain the post-processed audio.

[0048] Thirdly, embodiments of this application provide an audio generation device for audiobooks, comprising:

[0049] Central processing unit, memory, input / output interfaces, wired or wireless network interfaces, and power supply;

[0050] The memory is either a short-term storage memory or a persistent storage memory;

[0051] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the aforementioned audiobook audio generation method.

[0052] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the aforementioned audiobook audio generation method.

[0053] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to execute the aforementioned audiobook audio generation method.

[0054] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: the pre-trained role segmentation model can identify the adjacency relationship between every two adjacent sentences in the target text, determine the speaking role prompt sentence corresponding to each dialogue sentence in the target text based on at least one adjacency relationship corresponding to the target text, extract the speaking role from the speaking role prompt sentence corresponding to each dialogue sentence, and perform merging processing on the speaking roles to obtain the target role and the text content corresponding to the target role. The accuracy of determining the speaking role prompt sentence corresponding to each dialogue sentence in the target text can be improved based on at least one adjacency relationship corresponding to the target text, thereby improving the accuracy of extracting the speaking role. Furthermore, the merging processing on the speaking roles further improves the accuracy of determining the target role. Therefore, the accuracy of audio generation for audiobooks is relatively high. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the architecture of an audiobook audio generation system disclosed in an embodiment of this application;

[0056] Figure 2 This is a flowchart illustrating an audiobook audio generation method disclosed in an embodiment of this application;

[0057] Figure 3 This is a flowchart illustrating a method for segmenting target text and its corresponding text content using a role segmentation model, as disclosed in an embodiment of this application.

[0058] Figure 4 This is a flowchart illustrating a method for identifying the adjacency relationship between every two adjacent sentences of the replaced target text using a prompt sentence recognition model, as disclosed in an embodiment of this application.

[0059] Figure 5 This is a flowchart illustrating a method for training an audio-enabled model disclosed in an embodiment of this application.

[0060] Figure 6 This is a flowchart illustrating another audiobook audio generation method disclosed in an embodiment of this application;

[0061] Figure 7 This is a schematic diagram of the structure of an audiobook audio generation device disclosed in an embodiment of this application;

[0062] Figure 8 This is a schematic diagram of the structure of another audiobook audio generation device disclosed in an embodiment of this application. Detailed Implementation

[0063] This application provides an audiobook audio generation method, an audiobook audio generation device, and a computer-readable storage medium, which are used to generate audio for audiobooks while improving the accuracy of audio generation.

[0064] Please see Figure 1 The architecture of the audiobook audio generation system in this application embodiment includes:

[0065] The audiobook generation device 101 and client 102 are configured. When generating audio for an audiobook, the audiobook generation device 101 can connect to the client 102. The client 102 can send an audio generation request for the target text of the audiobook to the audiobook generation device 101. The audiobook generation device 101 can respond to the request, generate the audio corresponding to the target text, and send the audio to the client 102.

[0066] based on Figure 1 Please refer to the audio generation system shown. Figure 2 , Figure 2 This is a flowchart illustrating an audiobook audio generation method disclosed in an embodiment of this application. The method includes:

[0067] 201. Obtain the target text of the audiobook.

[0068] In this embodiment, when generating audio for an audiobook, the target text of the audiobook can be obtained.

[0069] Specifically, audiobooks include, but are not limited to, multicast audiobooks and single-cast audiobooks.

[0070] 202. Input the target text into the pre-trained role segmentation model. The pre-trained role segmentation model identifies the adjacency relationship between every two adjacent sentences in the target text. Based on at least one adjacency relationship in the target text, determine the speaking role prompt sentence corresponding to each dialogue sentence in the target text. Extract the speaking role from the speaking role prompt sentence corresponding to each dialogue sentence, and merge the speaking roles to obtain the target role and the text content corresponding to the target role. The target role and the text content corresponding to the target role are output by the pre-trained role segmentation model.

[0071] After obtaining the target text of the audiobook, the target text can be input into a pre-trained role segmentation model. The pre-trained role segmentation model identifies the adjacency relationship between every two adjacent sentences in the target text. Based on at least one adjacency relationship corresponding to the target text, the speaking role prompt sentence corresponding to each dialogue sentence in the target text is determined. The speaking role is extracted from the speaking role prompt sentence corresponding to each dialogue sentence, and the speaking roles are merged to obtain the target role and the text content corresponding to the target role. The target role and the text content corresponding to the target role are output by the pre-trained role segmentation model.

[0072] Understandably, the method of identifying the adjacency relationship between every two adjacent sentences in the target text using a pre-trained role segmentation model, determining the speaking role cue sentence corresponding to each dialogue sentence in the target text based on at least one adjacency relationship, extracting the speaking role from the speaking role cue sentence corresponding to each dialogue sentence, and merging the speaking roles to obtain the target role and its corresponding text content can be as follows: First, a pre-trained role recognition model identifies at least one role in the target text, and for each role, replaces the role literal value with the target role symbol to obtain the replaced target text; then, a pre-trained cue sentence recognition model identifies the adjacency relationship between every two adjacent sentences in the replaced target text, and based on... At least one adjacency relation corresponding to the target text is inferred between every two adjacent sentences to obtain the speaking role cue sentence corresponding to each dialogue sentence in the target text. The adjacency relations between every two adjacent sentences include cue-dialogue relations, dialogue-cue relations, dialogue-response relations, dialogue-continuation relations, dialogue-multiple responses relations, cue-interpretation relations, and / or no relation. Then, a pre-trained speaking role extraction model extracts the speaking role from the speaking role cue sentences. Finally, a pre-trained role merging model merges the speaking roles to obtain the target role and its corresponding text content. The pre-trained role segmentation model includes a pre-trained role recognition model, a cue sentence recognition model, a speaking role extraction model, and a role merging model. Alternatively, other reasonable methods can be used, such as using a pre-trained cue sentence recognition model to identify the adjacency relations between every two adjacent sentences in the replaced target text, and inferring the speaking role cue sentence corresponding to each dialogue sentence in the target text based on at least one adjacency relation corresponding to the target text. Specific methods are not limited here.

[0073] 203. For each target character, the text content corresponding to the target character is processed into audio, resulting in the audio of the target character.

[0074] After inputting the target text into a pre-trained character segmentation model and obtaining the target character and its corresponding text content output by the pre-trained character segmentation model, the text content corresponding to each target character can be processed into audio to obtain the audio of the target character.

[0075] 204. Perform post-processing on the audio of each target character to obtain the post-processed audio.

[0076] For each target character, the corresponding text content is processed into audio. After obtaining the audio of the target character, the audio of each target character can be post-processed to obtain the post-processed audio.

[0077] In this embodiment, a pre-trained role segmentation model can identify the adjacency relationship between every two adjacent sentences in the target text. Based on at least one adjacency relationship, a speaking role cue sentence corresponding to each dialogue sentence in the target text is determined. The speaking role is extracted from the speaking role cue sentence corresponding to each dialogue sentence, and the speaking roles are merged to obtain the target role and its corresponding text content. The accuracy of determining the speaking role cue sentence for each dialogue sentence in the target text is improved by using at least one adjacency relationship, thereby improving the accuracy of the extracted speaking role. Furthermore, the merging of speaking roles further enhances the accuracy of determining the target role. Therefore, the audio generation accuracy of the audiobook is high.

[0078] In this embodiment, a pre-trained role segmentation model identifies the adjacency relationship between every two adjacent sentences in the target text. Based on at least one adjacency relationship corresponding to the target text, a speaking role cue is determined for each dialogue sentence in the target text. The speaking role is extracted from the speaking role cue for each dialogue sentence, and the speaking roles are merged to obtain the target role and its corresponding text content. There are various methods based on... Figure 2 The audio generation methods shown are described below, and one of them is described in detail below.

[0079] In this embodiment, when generating audio for an audiobook, the target text of the audiobook can be obtained.

[0080] Specifically, audiobooks include, but are not limited to, multi-play audiobooks and single-play audiobooks. Multi-play AI audiobooks refer to audiobooks created directly or with the assistance of artificial intelligence technology, featuring different voices. Audiobooks refer to information in written form, such as novels, stories, and news, converted into audible audio, such as audio novels; different voices refer to voices with different timbres portraying different characters, such as the voices of a young man, an elderly woman, a child, and a teenage girl. The emergence of multi-play AI audiobooks has enriched the categories of audiobooks that were previously entirely produced manually, and also offers significant advantages in terms of production efficiency and cost. Single-play audiobooks refer to a single person reading an entire book aloud, which may imitate the voices of different characters, but the voice remains that of the same person throughout.

[0081] After obtaining the target text of the audiobook, the target text can be input into a pre-trained role segmentation model. The pre-trained role segmentation model identifies the adjacency relationship between every two adjacent sentences in the target text. Based on at least one adjacency relationship corresponding to the target text, the speaking role prompt sentence corresponding to each dialogue sentence in the target text is determined. The speaking role is extracted from the speaking role prompt sentence corresponding to each dialogue sentence, and the speaking roles are merged to obtain the target role and the text content corresponding to the target role. The target role and the text content corresponding to the target role are output by the pre-trained role segmentation model.

[0082] The process involves a pre-trained role segmentation model identifying the adjacency relationships between every two adjacent sentences in the target text. Based on at least one adjacency relationship, a speaking role cue is determined for each dialogue sentence in the target text. The speaking role is then extracted from each speaking role cue and merged to obtain the target role and its corresponding text content. This can be achieved by first using a pre-trained role recognition model to identify at least one role in the target text, and for each role, replacing the role literal with the target role symbol to obtain the replaced target text. Then, a pre-trained cue recognition model identifies the adjacency relationships between every two adjacent sentences in the replaced target text and, based on the target... At least one adjacency relation corresponding to the text is inferred between every two adjacent sentences to obtain the speaking role cue sentence corresponding to each dialogue sentence in the target text; wherein, the adjacency relation between every two adjacent sentences includes cue-dialogue relation, dialogue-cue relation, dialogue-response relation, dialogue-continuation relation, dialogue-multiple responses relation, cue-interpretation relation, and / or no relation; then, the pre-trained speaking role extraction model extracts the speaking role from the speaking role cue sentence; finally, the pre-trained role merging model merges the speaking roles to obtain the target role and the text content corresponding to the target role, wherein the pre-trained role segmentation model includes a pre-trained role recognition model, a cue sentence recognition model, a speaking role extraction model, and a role merging model.

[0083] It is important to understand that role recognition models include, but are not limited to, Bert-CRF-based named entity models, which can be used for role extraction. These models are responsible for replacing role literals in text with role wildcards, thereby improving the system's robustness in recognizing different texts (such as novels, sitcoms, essays, fairy tales, etc.). Secondly, the adjacency relationships between any two adjacent sentences include, but are not limited to, "hint and dialogue relationship," "dialogue and hint relationship," "dialogue reply relationship," "dialogue continuation relationship," "dialogue group echo relationship," "hint explanation relationship," and "no relationship." Specifically, the hint and dialogue relationship indicates that "the previous sentence is a hint, and the next sentence is dialogue"; the dialogue and hint relationship indicates that "the next sentence is a hint, and the previous sentence is dialogue"; the dialogue reply relationship indicates that "both sentences are dialogue, and the next sentence is a reply to the previous sentence"; the dialogue continuation relationship indicates that "both sentences are dialogue, and the next sentence and the previous sentence are spoken by the same person"; the dialogue group echo relationship indicates that "both sentences are dialogue, and the next sentence and the previous sentence are echoes from multiple people"; the hint explanation relationship indicates that "neither sentence is dialogue, and the next sentence is an explanation of the previous sentence"; and "no relationship" indicates that "the two sentences are unrelated." For example, if the adjacency relationship between two adjacent sentences in the target text is "hint and dialogue relationship," then the sentence preceding the dialogue sentence in the target text can be determined as the speaker's hint sentence. Furthermore, the speech role extraction model can analyze speech role prompts to extract the speech roles from them. Finally, the role merging model can be responsible for merging the role wildcards of the speech roles extracted by the speech role extraction model into specific roles. Role merging methods include, but are not limited to, merging roles through dictionary lookups.

[0084] Specifically, a pre-trained role segmentation model identifies the adjacency relationships between every two adjacent sentences in the target text. Based on at least one adjacency relationship in the target text, it determines the speaking role cue for each dialogue sentence. The speaking role is then extracted from the speaking role cue for each dialogue sentence, and these speaking roles are merged to obtain the target role and its corresponding text content. For the method described above, please refer to [link to method description]. Figure 3 , Figure 3 This is a flowchart illustrating a method for segmenting target text and its corresponding text content using a role segmentation model, as disclosed in an embodiment of this application. Figure 3Given a target text such as: "Cai Wushi, who was slapped by Mu Feng, shouted: 'You mere level eight martial artist, don't run away!'", the pre-trained character recognition model can first identify the characters Mu Feng and Cai Wushi in the target text. The character literal value "Mu Feng" for Mu Feng can be replaced with the target character symbol "[ch-0]", and the character literal value "Cai Wushi" for Cai Wushi can be replaced with the target character symbol "[ch-1]", resulting in the replaced target text: "[ch-1], who was slapped by [ch-0], shouted: 'You mere level eight martial artist, don't run away!'" Then, the pre-trained prompt recognition model can identify the adjacency relationship between the two adjacent sentences in the replaced target text, and determine the speaking character prompt based on this relationship. The adjacency relationship between sentences is "prompt and dialogue relationship," and the speaking character prompt is "[ch-1], who was slapped by [ch-0], shouted." Next, a pre-trained speech character extraction model can extract the speech character from the speech character prompt, with the speech character being [ch-1]. Finally, a pre-trained character merging model merges the speech characters to obtain the target character and its corresponding text content. The merging process can be done through reasonable methods such as dictionary lookup. For example, [ch-1] can be merged into Cai Li Mo, whose role identifier (ID) is SPEAKER-29, and the corresponding text content for Cai Li Mo can be determined.

[0085] In this process, a pre-trained role segmentation model identifies the adjacency relationship between every two adjacent sentences in the target text. Based on at least one adjacency relationship in the target text, the speaking role prompt sentence corresponding to each dialogue sentence in the target text is determined. The speaking role is extracted from the speaking role prompt sentence corresponding to each dialogue sentence, and the speaking roles are merged to obtain the target role and the text content corresponding to the target role. Before this, a prompt sentence recognition model can also be trained to obtain the trained prompt sentence recognition model.

[0086] The method for training the prompt recognition model and obtaining the trained prompt recognition model can be as follows: First, obtain target text samples. The target text samples are labeled with the adjacency relationship between every two sentences. The target text samples are obtained by replacing the literal value of each character with the target character symbol. Then, input the target text samples into the prompt recognition model to obtain the predicted adjacency relationship between each adjacent sentence in the target text sample output by the prompt recognition model. Finally, calculate the loss between the predicted adjacency relationship and the labeled adjacency relationship according to the regression loss function. When the loss meets the convergence condition, the trained prompt recognition model is obtained.

[0087] Specifically, the text sample can be first input into a character-level text encoder to obtain the target text sample output by the character-level text encoder. The target text sample is labeled with the adjacency relationship between every two sentences. The target text sample includes a vector of each character. Then, the vector corresponding to the special symbol [C] at the beginning of each sentence is used as a sentence-level vector representation. Next, the target text sample after concatenating the adjacent sentence vectors is input into the prompt sentence recognition model (fully connected neural network) to obtain the predicted adjacency relationship between each adjacent sentence in the target text sample output by the prompt sentence recognition model. Finally, the loss between the predicted adjacency relationship and the labeled adjacency relationship is calculated according to the regression loss function. When the loss meets the convergence condition, the trained prompt sentence recognition model is obtained.

[0088] For example, please refer to Figure 4 , Figure 4 This is a flowchart illustrating a method for identifying the adjacency relationship between every two adjacent sentences in the replaced target text using a prompt recognition model, as disclosed in an embodiment of this application. Figure 4 It can be seen that the target text (

C

C

C

C

C

[0089] Before inputting the target text into the pre-trained role segmentation model, the target text can be segmented into chapters to obtain at least one chapter of the target text. The method of inputting the target text into the pre-trained role segmentation model can be to input the chapter into the pre-trained role segmentation model for each chapter of the target text.

[0090] One method for segmenting the target text into chapters to obtain at least one chapter of the target text is to segment the target text into chapters using a preset regular expression.

[0091] Specifically, chapter segmentation involves dividing the entire target text (such as a novel or other text with at least one chapter) according to chapter numbers. There are several ways to segment by chapter number. For example, it can be achieved by setting a format type, or by using a preset regular expression to segment the target text into chapters. For instance, a preset regular expression like "Chapter X," "Chapter X," or "Episode X" would allow you to iterate through each paragraph of the target text (novel) line by line, segmenting the paragraphs between chapter names that conform to the predetermined format and saving them as independent chapter content. For example, a preset regular expression like "Chapter 2," "Chapter 2," or "Episode 2" would segment the paragraphs corresponding to Chapter 2, Chapter 2, or Episode 2 of the target text. Some texts use "Chapter" to represent different chapters, some use "Chapter," some use "Episode," and some use "Episode," or other reasonable words to represent different chapters; the specifics are not limited here. It is understandable that the preset regular expression, besides "Chapter X", "Episode X", or "Episode X", can also be any other reasonable method that can segment the entire target text into chapters; specific methods are not limited here. It is also understandable that when the target text includes at least one chapter, chapter segmentation can be performed first, and then the audio of each segmented chapter can be processed separately for audiobook generation, improving the feasibility of generating audiobook audio by chapter.

[0092] After inputting the target text into a pre-trained character segmentation model and obtaining the target character and its corresponding text content output by the pre-trained character segmentation model, the text content corresponding to each target character can be processed into audio to obtain the audio of the target character.

[0093] One method for processing the text content corresponding to each target character into audio is to determine the target character's ID for each target character, then input the target character's ID and the text content corresponding to the target character into a pre-trained audio processing model. The pre-trained audio processing model processes the text content corresponding to the target character into audio, and the pre-trained audio processing model outputs the audio content corresponding to the target character.

[0094] Before inputting the target character's ID and the corresponding text content into the pre-trained audio model, the audio model can be trained to obtain the trained audio model.

[0095] The method for training the voice model and obtaining the trained voice model can be as follows: First, obtain at least one character sample, as well as the ID and text content sample corresponding to each character sample. The ID corresponding to each character sample is labeled with the audio features of the real speech. Then, for each character sample, input the ID and text content sample corresponding to the character sample into the voice model to obtain the predicted audio features output by the voice model. Finally, calculate the loss between the predicted audio features and the labeled audio features according to the regression loss function. When the loss meets the convergence condition, the trained voice model is obtained.

[0096] Specifically, the technologies used in voice-enhanced models can be speech synthesis techniques. For example, they can employ a cascaded synthesis approach based on an acoustic front-end, a non-autoregressive acoustic model (such as FastSpeech, as shown in the dashed box in the figure below), and a vocoder (such as HifiGan), or an end-to-end approach (such as VITS). Please refer to [link / reference] for details. Figure 5 , Figure 5 This is a flowchart illustrating a method for training a voiced model disclosed in an embodiment of this application. Figure 5 As can be seen, the dashed box represents the non-autoregressive acoustic model. The voice-enhanced model (non-autoregressive acoustic model) includes an audio coding network, an encoder network, a variance adapter network, and a decoder network. For each role ID, a phoneme sequence can be obtained first based on the role ID and the corresponding text. Then, the phoneme sequence is input into the encoder network to obtain the encoded text sequence output by the encoder network. At the same time, real speech can be input into the audio coding network, which converts the audio features of the real speech into a feature sequence. Then, the variance adapter network jointly modulates the encoded text sequence with the audio features output by the audio coding network. Then, the decoder network obtains the predicted audio features. Finally, after obtaining the real audio features of the real speech, the loss between the predicted audio features and the real audio features is evaluated using a loss function such as MSE loss. When the loss meets the convergence condition, the trained voice-enhanced model is obtained.

[0097] It's worth noting that when adding a timbre ID, the corresponding real speech and text can be added during the training of the voice model. This allows the voice model (acoustic model) to acquire the audio features of the new timbre ID during modeling, thereby increasing the richness of selectable timbres. During use, simply specify the desired timbre ID, and the voice model (acoustic model) can predict the corresponding audio features, which are then converted into an audible sound waveform using a vocoder.

[0098] It is understandable that voice-over can include narrating any given sentence using AI (artificial intelligence) and / or human methods. Human methods include recording separately by different broadcasters at different times and locations. After the target file is segmented according to roles by a pre-trained role segmentation model, in addition to obtaining the text content of each target role and its corresponding text, the text content of the narration can also be obtained, and the narration text content can be narrated. For example, the current narration can be narrated using AI role voice A, the next dialogue can be narrated using AI role voice B, and the next dialogue can be narrated manually. Which sentences are narrated using which method can be determined according to actual needs, and is not limited here. For sentences narrated manually, it is only necessary to determine the broadcaster's name and the corresponding voice-over text, then distribute the text to the corresponding broadcaster, who can then record their voice. The recordings from each broadcaster are then uploaded and collected online, enabling a multi-person collaborative online work mode that is efficient and simple.

[0099] For each target character, the corresponding text content is processed into audio. After obtaining the audio of the target character, the audio of each target character can be post-processed to obtain the post-processed audio.

[0100] One method for obtaining the post-processed audio for each target character is to perform audio track alignment on the audio processed for each target character. The aligned audio is the post-processed audio. Specifically, audio track alignment refers to integrating the sounds of different sentences and paragraphs into a complete and naturally transitioning audio.

[0101] Another method for post-processing the audio for each target character to obtain the post-processed audio is as follows: the audio for each target character consists of N audio sentences, where N is an integer greater than or equal to 2. For each audio sentence, the initial semantics, keywords, and ending semantics are determined first. Then, based on the initial semantics of the nth audio sentence, the first pause interval between the nth and (n-1)th audio sentences is determined; and based on the ending semantics of the nth audio sentence, the pause interval between the nth and (n+1)th audio sentences is determined. The second pause interval between sentences; where 2≤n≤N, then the scene sound effect corresponding to the nth audio sentence is determined according to the keyword corresponding to the nth audio sentence. Finally, the first pause interval is used as the pause interval between the nth audio sentence and the (n-1)th audio sentence, and the second pause interval is used as the pause interval between the nth audio sentence and the (n+1)th audio sentence. The scene sound effect is added to the nth audio sentence, resulting in N audio sentences with pause intervals and scene sound effect added. The N audio sentences with pause intervals and scene sound effect added are used as the audio after post-processing.

[0102] Specifically, the sentence can be categorized based on its three-part semantic structure, resulting in the sentence's initial semantics, keywords, and ending semantics. The initial semantics determine the pause between the current and previous sentences, while the ending semantics determine the pause between the current and next sentences. Longer pauses are typically added during dialogue transitions between different characters or when keywords indicating a shift in thought (such as "but," "however," "just," etc.). The sentence's keywords serve as a reference for adding scene sound effects. Scene sound effects refer to various sound events, such as the sound of high heels, clapping, laughter, car horns, thunder, etc. For example, if the sentence states "a car is approaching from a distance," a car driving sound could be added.

[0103] It's understandable that adding scene sound effects is one way to add music sound effects. Besides adding scene sound effects, adding music sound effects can also be done by adding background music. Specifically, background music can be different types of light music categorized by theme, such as cheerful, sad, open field, quiet mountain forest, or bustling city sounds. Correspondingly, another way to post-process the audio of each target character is to add background music to that audio.

[0104] It is also understandable that, through post-processing such as audio track matching and adding music and sound effects (adding scene sound effects and / or background music), the final audiobook audio can achieve a more harmonious and natural listening experience.

[0105] To facilitate understanding of the embodiments of this application, the audio generation method for audiobooks is described below using multicast audiobooks and unicast audiobooks as examples. Please refer to... Figure 6 , Figure 6 This is a flowchart illustrating another audiobook audio generation method disclosed in an embodiment of this application. Figure 6 It is understood that for multicast audiobooks (such as novels), the novel text can be divided into chapters, and then for each chapter's text content, characters can be segmented to obtain each character and their text content. The corresponding AI voice for each character's text content can then be obtained. Post-processing, including audio track matching, background music (BGM), and sound effects (scene sound effects), is then applied to finally produce the multicast AI voice. For unicast audiobooks, the novel text can be divided into chapters to obtain the text content of each chapter, and then the corresponding unicast AI voice for each chapter's text content can be obtained.

[0106] It is worth mentioning that the audiobook audio generation method of this application embodiment can be processed not only in the computer background, but also in the cloud. The cloud has higher processing efficiency and faster running speed. Therefore, the embodiment of this application has technical and economic advantages.

[0107] It is understandable that, in addition to the method described above—which uses a pre-trained role segmentation model to identify the adjacency relationship between every two adjacent sentences in the target text, determines the speaking role prompt sentence corresponding to each dialogue sentence in the target text based on at least one adjacency relationship, extracts the speaking role from the speaking role prompt sentence corresponding to each dialogue sentence, and merges the speaking roles to obtain the target role and the text content corresponding to the target role; in addition to the method described above—which trains the prompt sentence recognition model to obtain the trained prompt sentence recognition model; in addition to the method described above—which segments the target text into chapters to obtain at least one chapter of the target text; in addition to the method described above—which processes the text content corresponding to each target role into audio for each target role to obtain the audio for the target role; in addition to the method described above—which trains the audio model to obtain the trained audio model; in addition to the method described above—which performs post-processing on the audio for each target role to obtain the audio for the target role; other reasonable methods may also be used, and specific methods are not limited here.

[0108] In this embodiment, a pre-trained role segmentation model can identify the adjacency relationship between every two adjacent sentences in the target text. Based on at least one adjacency relationship, the speaking role prompts for each dialogue sentence in the target text are determined. Speaking roles are extracted from the speaking role prompts for each dialogue sentence, and these speaking roles are then merged to obtain the target role and its corresponding text content. By reasoning between every two adjacent sentences based on at least one adjacency relationship, the accuracy of determining the speaking role prompts for each dialogue sentence in the target text is improved, thereby improving the accuracy of the extracted speaking roles. Furthermore, the merging of speaking roles further improves the accuracy of determining the target role, thus improving the accuracy of audiobook audio generation. Secondly, the pre-trained role segmentation model can automatically segment the target text according to roles, requiring less time and effort to obtain each role and its corresponding text content, thus improving the efficiency of audio generation. Furthermore, audio processing can include voicing any specified sentence using AI (artificial intelligence) and / or manual methods. Compared to traditional manual recording methods, AI (artificial intelligence) can be used, where a pre-trained audio processing model voices the text content corresponding to the target character, improving processing efficiency. Additionally, the audio processing model (acoustic model) can acquire the audio features of the new timbre ID during modeling, increasing the richness of selectable timbres. Moreover, the combination of AI and manual methods, or either AI or manual methods, can be chosen based on actual needs, increasing the selectivity of the audio processing method. Finally, the audio generation of the audiobook in this embodiment can be processed in a computer backend or in the cloud. Cloud processing is more efficient and faster; therefore, this embodiment has technical and economic advantages.

[0109] The audio generation method for audiobooks in the embodiments of this application has been described above. The audio generation device for audiobooks in the embodiments of this application is described below. Please refer to [link / reference]. Figure 7 One embodiment of the audiobook audio generation device in this application includes:

[0110] Unit 701 is used to obtain the target text of the audiobook;

[0111] Input unit 702 is used to input the target text into a pre-trained role segmentation model, which identifies the adjacency relationship between every two adjacent sentences in the target text, determines the speaking role prompt sentence corresponding to each dialogue sentence in the target text based on at least one adjacency relationship corresponding to the target text, extracts the speaking role from the speaking role prompt sentence corresponding to each dialogue sentence, and performs merging processing on the speaking roles to obtain the target role and the text content corresponding to the target role, thus obtaining the target role and the text content corresponding to the target role output by the pre-trained role segmentation model.

[0112] The audio processing unit 703 is used to process the text content corresponding to each target character into audio for each target character, so as to obtain the audio processed audio corresponding to the target character.

[0113] The post-processing unit 704 is used to perform post-processing on the audio of each target character after it has been processed into sound, so as to obtain the audio after post-processing.

[0114] In this embodiment, the pre-trained role segmentation model can identify the adjacency relationship between every two adjacent sentences in the target text. Based on at least one adjacency relationship corresponding to the target text, a speaking role prompt sentence is determined for each dialogue sentence in the target text. The speaking role is extracted from the speaking role prompt sentences corresponding to each dialogue sentence, and the speaking roles are merged to obtain the target role and its corresponding text content. Determining the speaking role prompt sentence for each dialogue sentence in the target text based on at least one adjacency relationship improves the accuracy of identifying the speaking role prompt sentence, thereby improving the accuracy of the extracted speaking role. Furthermore, the merging of speaking roles further improves the accuracy of identifying the target role, thus improving the accuracy of audiobook audio generation.

[0115] Please refer to the following: Figure 8 Another embodiment of the audio generating device 800 for audiobooks in this application includes:

[0116] Central processing unit 801, memory 805, input / output interface 804, wired or wireless network interface 803, and power supply 802;

[0117] Memory 805 is either a short-term storage memory or a persistent storage memory;

[0118] The central processing unit 801 is configured to communicate with the memory 805 and execute instructions stored in the memory 805 to perform the aforementioned operations. Figure 2 The method in the illustrated embodiment.

[0119] This application also provides a computer-readable storage medium, which includes instructions that, when executed on a computer, cause the computer to perform the aforementioned actions. Figure 2 The method in the illustrated embodiment.

[0120] This application also provides a computer program product containing instructions, which, when run on a computer, causes the computer to perform the aforementioned... Figure 2 The method in the illustrated embodiment.

[0121] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0122] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0123] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0124] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0125] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0126] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for generating audio for an audiobook, characterized in that, include: Obtain the target text of the audiobook; The target text is input into a pre-trained role segmentation model, which includes a pre-trained role recognition model, a cue sentence recognition model, a speaking role extraction model, and a role merging model. The pre-trained role segmentation model identifies the adjacency relationships between every two adjacent sentences in the target text. Based on at least one adjacency relationship corresponding to the target text, inference is performed between every two adjacent sentences to determine the speaking role cue sentence corresponding to each dialogue sentence in the target text. The speaking role is extracted from the speaking role cue sentences corresponding to each dialogue sentence, and the speaking roles are merged to obtain the target role and its corresponding text content. The target role and its corresponding text content are output by the pre-trained role segmentation model, including: identifying at least one role in the target text by the pre-trained role recognition model. For each role, the role literal value is replaced with the target role symbol to obtain the replaced target text. The pre-trained prompt recognition model identifies the adjacency relationship between every two adjacent sentences in the replaced target text, and infers between every two adjacent sentences based on at least one adjacency relationship corresponding to the target text to obtain the speaking role prompt sentence corresponding to each dialogue sentence in the target text. The adjacency relationship between every two adjacent sentences includes prompt and dialogue relationship, dialogue and prompt relationship, dialogue reply relationship, dialogue continuation relationship, dialogue group response relationship, prompt explanation relationship, and / or no relationship. The pre-trained speaking role extraction model extracts the speaking role from the speaking role prompt sentences. The pre-trained role merging model merges the speaking roles to obtain the target role and the text content corresponding to the target role. For each target character, the text content corresponding to the target character is processed into audio to obtain the audio of the target character. The audio of each target character is processed with sound effects and then post-processed to obtain the post-processed audio.

2. The method according to claim 1, characterized in that, Before the pre-trained prompt recognition model identifies the adjacency relationship between every two adjacent sentences in the replaced target text, and infers between every two adjacent sentences based on at least one adjacency relationship corresponding to the target text to obtain the speaking role prompt corresponding to each dialogue sentence in the target text, the method further includes: Obtain a target text sample; wherein, the target text sample is marked with the adjacency relationship between every two sentences; the target text sample is the text sample obtained by replacing the literal value of each character with the target character symbol; The target text sample is input into the prompt sentence recognition model to obtain the predicted adjacency relationship between each adjacent sentence in the target text sample output by the prompt sentence recognition model; The loss between the predicted adjacency relationship and the labeled adjacency relationship is calculated based on the regression loss function. When the loss meets the convergence condition, the trained prompt recognition model is obtained.

3. The method according to claim 1, characterized in that, The target text includes at least one chapter; Before inputting the target text into the pre-trained role segmentation model, the method further includes: The target text is segmented into chapters to obtain at least one chapter of the target text; The step of inputting the target text into a pre-trained role segmentation model includes: For each chapter of the target text, the chapter is input into the pre-trained character segmentation model.

4. The method according to claim 3, characterized in that, The step of segmenting the target text into chapters to obtain at least one chapter of the target text includes: The target text is divided into chapters using a preset regular expression to obtain at least one chapter of the target text.

5. The method according to claim 1, characterized in that, For each target character, the text content corresponding to that target character is processed into audio to obtain the processed audio for that target character, including: For each target role, determine the ID of the target role; The ID of the target character and the text content corresponding to the target character are input into a pre-trained audio model. The pre-trained audio model performs audio processing on the text content corresponding to the target character, and the audio corresponding to the target character is output by the pre-trained audio model.

6. The method according to claim 5, characterized in that, Before inputting the ID of the target character and the text content corresponding to the target character into the pre-trained audio model, the method further includes: Obtain at least one character sample, as well as the ID corresponding to each character sample and the text content sample corresponding to each character sample; the ID corresponding to each character sample is labeled with the audio features of real speech; For each character sample, the ID corresponding to the character sample and the text content sample corresponding to the character sample are input into the audio model to obtain the predicted audio features output by the audio model. The loss between the predicted audio features and the labeled audio features is calculated based on the regression loss function. When the loss meets the convergence condition, the trained audio model is obtained.

7. The method according to claim 1, characterized in that, The step of performing post-processing on the audio corresponding to each target character to obtain the post-processed audio includes: The audio of each target character's voice-processed audio is aligned to the audio track to obtain the aligned audio; the aligned audio is the audio after the post-processing of the effect.

8. The method according to claim 1, characterized in that, The audio processed by the voice for each target character includes N audio sentences; where N is an integer greater than or equal to 2. The step of performing post-processing on the audio corresponding to each target character to obtain the post-processed audio includes: For each audio sentence, determine the beginning semantics, keywords, and ending semantics of the audio sentence; The first pause interval between the nth audio sentence and the (n-1)th audio sentence is determined based on the beginning semantics corresponding to the nth audio sentence; and the second pause interval between the nth audio sentence and the (n+1)th audio sentence is determined based on the ending semantics corresponding to the nth audio sentence; where 2≤n≤N; The scene sound effect corresponding to the nth audio sentence is determined based on the keywords corresponding to the nth audio sentence; The first pause interval is used as the pause interval between the nth audio sentence and the (n-1)th audio sentence, and the second pause interval is used as the pause interval between the nth audio sentence and the (n+1)th audio sentence. The scene sound effect is added to the nth audio sentence to obtain N audio sentences with pause intervals and scene sound effect processing. The pause intervals and N audio sentences with scene sound effect processing are used as the audio after post-processing of the effect.

9. An audio generation device for audiobooks, characterized in that, include: Central processing unit and memory; The memory is either a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 8.

Images