Method and device for generating wonderful content of audio book, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a wonderful content generation method and device of an audio book, electronic equipment and a computer readable storage medium. The method for generating the wonderful content of the audio book comprises the following steps: acquiring an original audio of the audio book, and converting the original audio into an original text; determining a wonderful content generation strategy of the audio book, and generating prompt information associated with the wonderful content generation strategy in combination with the original text; wherein the wonderful content generation strategy is a strategy determined in a complete plot strategy, a multi-role dialogue strategy or a content simplification strategy; inputting the prompt information into a large language model, and generating reply content corresponding to the prompt information by the large language model; and generating the wonderful content of the audio book based on the reply content. The efficiency of generating the wonderful content of the audio book can be improved, and meanwhile the flexibility of extracting the wonderful content is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of audio books. More specifically, embodiments of the present invention relate to a method, device, electronic device, and computer-readable storage medium for generating wonderful content of an audio book. Background Art

[0002] As a new reading method, audiobooks provide users with convenient auditory enjoyment, allowing users to enjoy the fun of reading in a variety of occasions. For audiobooks, users can only roughly understand its content through the name, introduction and other features of the audiobook. In order to enable users to directly experience the wonderful and attractive content in the audiobook, it is necessary to extract the wonderful content from the complete audiobook and push it to the user, so as to attract the user to watch the extracted wonderful content and generate interest in the complete audiobook corresponding to the wonderful content.

[0003] Currently, a common way to extract the wonderful content of audiobooks is to rely on manual listening to the audio content of the complete audiobook, and select the wonderful content in the audio content of the audiobook based on personal subjective judgment, and then push the wonderful content to the user. However, the current method of extracting the wonderful content of audiobooks is relatively inefficient and has low flexibility in extracting the wonderful content. Summary of the invention

[0004] In view of the above technical problems, the present invention provides a method, device, electronic device and computer-readable storage medium for generating wonderful content of an audio book. The technical solution is as follows:

[0005] According to a first aspect of the present invention, there is provided a method for generating wonderful content of an audio book, the method comprising:

[0006] Obtaining the original audio of the audio book, and converting the original audio into original text;

[0007] Determine a wonderful content generation strategy of the audio book, and generate prompt information associated with the wonderful content generation strategy in combination with the original text; wherein the wonderful content generation strategy is a strategy determined in a complete plot strategy, a multi-character dialogue strategy, or a content simplification strategy;

[0008] Inputting the prompt information into a large language model, and generating a reply content corresponding to the prompt information by the large language model;

[0009] Based on the reply content, generate the wonderful content of the audio book.

[0010] According to a second aspect of the present invention, there is provided a device for generating wonderful contents of an audio book, the device comprising:

[0011] An acquisition unit, used for acquiring the original audio of the audio book and converting the original audio into original text;

[0012] A determination unit, configured to determine a wonderful content generation strategy of the audio book, and generate prompt information associated with the wonderful content generation strategy in combination with the original text; wherein the wonderful content generation strategy is a strategy determined in a complete plot strategy, a multi-character dialogue strategy, or a content simplification strategy;

[0013] A generating unit, configured to input the prompt information into a large language model, and generate a reply content corresponding to the prompt information by the large language model;

[0014] The generating unit is further configured to generate the wonderful contents of the audio book based on the reply contents.

[0015] According to a third aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0016] processor;

[0017] a memory for storing processor-executable instructions;

[0018] The processor is configured to implement the method as described in the first aspect.

[0019] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps in the method as described in the first aspect.

[0020] The technical solution provided by the present invention converts the original audio of the audio book into original text, so that the original text can be processed by a large language model to efficiently extract the content associated with the wonderful content generation strategy from the original text, and generate the wonderful content of the audio book based on the extracted content. The degree of automation is high, no human participation is required, and the subjective influence of individuals on the judgment of wonderful content is avoided, thereby improving the efficiency of generating wonderful content of the audio book. At the same time, multiple strategies for generating wonderful content can be provided to adapt to the characteristics of different categories of audio books and different usage scenarios, thereby improving the flexibility of extracting wonderful content.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0023] Figure 1 It is a schematic diagram of an application scenario of the wonderful content of an audio book of related technology;

[0024] Figure 2 It is a schematic diagram of a production scenario of the wonderful content of an audio book of related technology;

[0025] Figure 3 It is a flowchart of a method for generating wonderful contents of an audio book according to an embodiment of the present invention;

[0026] Figure 4 It is a schematic structural diagram of a device for generating wonderful contents of an audio book according to an embodiment of the present invention;

[0027] Figure 5 It is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be described in detail below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field should belong to the scope of protection of the present invention.

[0029] An audiobook is a publication that expresses content through sound, usually recorded by one or more people based on a manuscript with the help of different voice expressions and recording formats. Common audiobook formats include tapes, CDs, and digital files (such as MP3). Audiobooks first appeared in the 1980s, mainly published by audiobook publishers or blind book societies. The content can be readings, radio dramas, or special reports. With the rapid development of the Internet, audiobooks have been launched on major audio platforms, making it faster and more convenient for users to listen to audiobooks. The popularization of audiobooks not only facilitates daily life, but also frees up hands, allowing people to enjoy listening to books while doing other things.

[0030] As a new way of reading, audiobooks provide users with convenient auditory enjoyment, allowing users to enjoy the pleasure of reading in a variety of occasions. Figure 1As shown, for audiobooks, users can only roughly understand the content through the name, introduction and other features of the audiobooks. In order to enable users to directly feel the wonderful and attractive parts of the audiobooks, it is necessary to extract the wonderful content from the complete audio of the audiobooks and push it to users, so as to attract users to watch the extracted wonderful content and become interested in the complete audiobooks corresponding to the wonderful content, so that they will listen to the complete audio of the audiobooks.

[0031] See also Figure 2 As shown, currently, a common method for extracting the wonderful content of audiobooks is to rely on manual listening to the complete audio of the audiobook, and manually intercept the wonderful content in the complete audio of the audiobook based on personal subjective judgment, and then push the wonderful content to the user. Specifically, the current method for extracting the wonderful content of audiobooks includes the following steps: (1) Manual listening: Professionals or volunteers carefully listen to all or part of the content of the audiobook. (2) Content analysis: The listener needs to analyze all or part of the content of the audiobook to identify segments with important information or emotional value. (3) Segment interception: After identifying the wonderful segment, the listener manually operates the device to intercept and mark it. (4) Review and proofreading: The intercepted segment may need to be reviewed and proofread to ensure its accuracy and quality. However, the current method for extracting the wonderful content of audiobooks relies more on manual operation, the individual's judgment of the wonderful content is highly subjective, the efficiency is relatively low, and the strategy followed for extracting the wonderful content is relatively fixed and less flexible.

[0032] It is worth noting that the above-mentioned introductions to the methods of extracting the wonderful contents of audio books are all exemplary. In practical applications, other extraction methods are not excluded and are not specifically limited thereto.

[0033] In view of the above problems, an embodiment of the present invention provides a method for generating wonderful content of an audio book, which can greatly reduce the workload of developers in writing codes, reduce code redundancy, and reduce the difficulty of code maintenance. Figure 3 As shown, the method comprises the following steps:

[0034] S301: Acquire the original audio of the audio book, and convert the original audio into original text.

[0035] S302: Determine a wonderful content generation strategy for the audio book, and generate prompt information associated with the wonderful content generation strategy in combination with the original text.

[0036] The wonderful content generation strategy is a strategy determined in a complete plot strategy, a multi-character dialogue strategy or a content simplification strategy.

[0037] S303: Input the prompt information into a large language model, and use the large language model to generate a reply content corresponding to the prompt information.

[0038] S304: Generate wonderful content of the audio book based on the reply content.

[0039] The technical solution provided by the embodiment of the present invention converts the original audio of the audio book into the original text, so that the original text can be processed by a large language model to efficiently extract the content associated with the wonderful content generation strategy from the original text, and generate the wonderful content of the audio book based on the extracted content. It has a high degree of automation and does not require human participation, thus avoiding the subjective influence of individuals on the judgment of wonderful content and improving the efficiency of generating wonderful content of the audio book. At the same time, it can provide a variety of strategies for generating wonderful content to adapt to the characteristics of different categories of audio books and different usage scenarios, and improve the flexibility of extracting wonderful content.

[0040] Considering that specific nouns, such as names of people and places, may appear in the original audio of an audiobook, it is more likely that recognition errors will occur in the process of identifying these specific nouns appearing in the original audio and converting them into text, resulting in a high error rate in the converted text.

[0041] To solve this problem, as an example, before converting the original audio of an audiobook into original text, a keyword list can be constructed, which contains several specific nouns. The specific nouns are used to replace the recognition results of the vocabulary associated with the specific nouns in the original audio of the audiobook; based on the keyword list, the original audio of the audiobook can be converted into the above-mentioned original text. The recognition results of the vocabulary associated with the specific noun in the original audio are replaced by the specific nouns in the pre-constructed keyword list. Regardless of whether the recognition result is correct or not, the correct specific nouns are used for replacement as much as possible to ensure the accuracy of the original text finally converted as much as possible.

[0042] As another example, the vocabulary associated with the specific noun in the original audio may refer to the vocabulary whose pronunciation is associated with the specific noun. As another example, the vocabulary whose pronunciation is associated with the specific noun may be the vocabulary that is exactly the same as the pronunciation of the specific noun, or may be the vocabulary that is similar to the pronunciation of the specific noun. For example: assuming that a specific noun in the keyword list is "Li Bai", then the word pronounced as "li bai" is a word associated with the specific noun "Li Bai", so in the original audio of the audio book, when the recognition result of the word pronounced as "li bai" appears, the recognition result is replaced with "Li Bai" in the keyword list. Regardless of the recognition result, the replacement is performed to avoid the recognition error of the name "Li Bai" as much as possible; in addition, assuming that a specific noun in the keyword list is "Great Wall", then the word pronounced as "chang cheng" is a word associated with the specific noun "Great Wall", so in the original audio of the audio book, when the recognition result of the word pronounced as "changcheng" appears, the recognition result is replaced with "Great Wall" in the keyword list, so as to avoid the recognition error of the place name "Great Wall" as much as possible.

[0043] In order to further improve the accuracy of the original text converted from the original audio of the audio book, as an example, the initial text converted from the original audio of the audio book can be first input into the large language model, and the large language model performs error correction on the initial text to obtain the final original text output after the error correction processing by the large language model.

[0044] As an example, the original audio of the audio book can be converted into the original text based on the Automatic Speech Recognition (ASR) technology, or the original audio of the audio book can be converted into the original text through other conversion methods, which are not specifically limited.

[0045] The wonderful content generation strategy of the audio book can be determined in a variety of ways. As an example, if a user selection indication is detected, the wonderful content generation strategy can be determined based on the user selection indication; if the user selection indication is not detected, the wonderful content generation strategy can be determined based on the characteristics of the original audio of the audio book. The characteristics of the original audio of the audio book may include audio characteristics of the original audio and category characteristics of the original audio. When determining the wonderful content generation strategy of the audio book, the user's indication is given priority. If the user provides an indication, the wonderful content generation strategy is automatically determined based on the user's indication to ensure that the user's needs are met first. If the user does not provide an indication, the wonderful content generation strategy is determined based on the audio characteristics and category characteristics of the original audio of the audio book.

[0046] As an example, the user referred to in the above user selection indication may be a person responsible for operating the audiobook, or may be other types of persons, and there is no specific limitation on this.

[0047] There may be a variety of specific implementations of user selection instructions. As an example, the user selection instructions may include selection instructions for the complete plot strategy, the multi-character dialogue strategy, and the content simplification strategy. Through the user selection instructions, the user can directly select the complete plot strategy, the multi-character dialogue strategy, and the content simplification strategy according to preference. Based on the selection, the wonderful content generation strategy can be directly determined to better meet user needs. As another example, the user selection instructions may include selection instructions for the usage scenarios of the wonderful content of the audiobook. The wonderful content generation strategy can be automatically determined based on the usage scenarios of the wonderful content of the audiobook selected by the user. It is worth noting that the above introduction to the specific implementation of the user selection instructions is an exemplary display. In actual applications, other specific implementations are not excluded and are not limited to this.

[0048] There may be a variety of specific implementations of the usage scenarios of the wonderful content of the above-mentioned audiobooks. As an example, the usage scenario of the wonderful content of an audiobook may be a scenario in which a potential audiobook consumer browses the opening of an audiobook. In this scenario, the multi-character dialogue strategy may be determined as the wonderful content generation strategy; as another example, the usage scenario of the wonderful content of an audiobook may be a scenario in which a potential audiobook consumer browses a trailer of an audiobook. In this scenario, the multi-character dialogue strategy may be determined as the wonderful content generation strategy; as another example, the usage scenario of the wonderful content of an audiobook may be a scenario in which the wonderful content of an audiobook is promoted and distributed as advertising content to potential audiobook consumers. In this scenario, the complete plot strategy may be determined as the wonderful content generation strategy; as another example, the usage scenario of the wonderful content of an audiobook may be a scenario in which a potential audiobook consumer browses the overall content introduction of an audiobook (such as a novel). In this scenario, the content simplification strategy may be determined as the wonderful content generation strategy.

[0049] The audio features of the original audio of the above-mentioned audio book can be implemented in many specific ways. As an example, the audio features of the original audio of the audio book can at least include the number of timbres contained in the original audio and the duration of the original audio. Among them, the above-mentioned multi-character dialogue strategy cannot be determined as the above-mentioned wonderful content generation strategy when the number of timbres contained in the original audio is less than the timbres number threshold; the above-mentioned content simplification strategy cannot be determined as the above-mentioned wonderful content generation strategy when the duration of the original audio is less than the duration threshold; when the above-mentioned multi-character dialogue strategy and the above-mentioned content simplification strategy cannot be determined as the above-mentioned wonderful content generation strategy, the above-mentioned complete plot strategy can be determined as the wonderful content generation strategy. It is possible to flexibly determine the wonderful content generation strategy based on the audio features of the original audio of the audiobook. For example, when the original audio contains a small number of timbres, it means that the original audio contains a small number of characters, which also means that the number of dialogues may be small, so it is not suitable to adopt a multi-character dialogue strategy as a wonderful content generation strategy; when the original audio contains a large number of timbres, it means that the original audio contains a large number of characters, which also means that the number of dialogues may be large. At this time, a multi-character dialogue strategy is adopted to extract more high-quality dialogues from the original audio. When the original audio duration is short, it means that the original audio contains less content and lacks sufficient content depth, so it is not suitable to adopt a content simplification strategy as a wonderful content generation strategy; when the original audio duration is long, it means that the original audio contains more content and has sufficient content depth. At this time, a content simplification strategy is adopted to extract high-quality simplification content from the original audio. When neither the above-mentioned multi-character dialogue strategy nor the above-mentioned content simplification strategy can be determined as the above-mentioned wonderful content generation strategy, the complete plot strategy can be used as a fallback strategy and be determined as the wonderful content generation strategy. Relying on the complete plot strategy, one or more complete storylines can be selected from the many complete storylines of the same audiobook as wonderful content to attract listeners.

[0050] The category characteristics of the original audio of the above-mentioned audio book can be determined in a variety of ways. As an example, the category characteristics of the original audio of the audio book can be determined by the category tags pre-selected for the original audio by the author of the audio book when uploading the original audio; as another example, the category characteristics of the original audio of the audio book can be determined by the category tags pre-selected for the original audio by the manufacturer of the audio book when producing the original audio. Therefore, there is no specific limitation on the method of determining the category characteristics of the original audio of the audio book.

[0051] The category characteristics of the original audio of the audiobook can be implemented in many ways, and when determining the wonderful content generation strategy, different wonderful content generation strategies may correspond to different weights under different category characteristics. As an example, the category characteristics may be suspense and supernatural, and when the category characteristics are suspense and supernatural, the weight of the complete plot strategy is the largest when determining the wonderful content generation strategy. When the wonderful content generation strategy is the complete plot strategy, the generated wonderful content can better highlight the tense and mysterious atmosphere of the suspense and supernatural audiobook content. As another example, the category characteristics may be romance stories, and when the category characteristics are romance stories, the weight of the multi-character dialogue strategy is the largest when determining the wonderful content generation strategy. When the wonderful content generation strategy is the multi-character dialogue strategy, the generated wonderful content can better highlight the emotional richness and dynamic changes of the romance story audiobook content.

[0052] There are many ways to generate prompt information associated with the wonderful content generation strategy in combination with the original text. As an example, when the wonderful content generation strategy is a complete plot strategy, the prompt information associated with the complete plot strategy may include selection instructions for several complete target plots in the original text, and the several complete target plots may respectively belong to different plot properties. As another example, the reply content corresponding to the prompt information generated by the large language model based on the prompt information may include the selection results of several complete target plots in the original text.

[0053] It is understandable that the original text itself can contain multiple complete plots, and the number of all complete plots contained in the original text is greater than or equal to the number of several complete target plots in the original text selected from the above reply content.

[0054] As another example, when the wonderful content generation strategy is a multi-character dialogue strategy, the prompt information associated with the multi-character dialogue strategy may include selection instructions for dialogue segments of several characters in the original text, and fusion instructions for the selected dialogue segments, and the reply content may include the fusion results of the selected dialogue segments. The multi-character dialogue strategy aims to extract and highlight the emotional core of the audiobook - the character dialogue. At the same time, after using the large language model to identify the dialogue segments of each character, these dialogue segments can also be artistically spliced ​​and reorganized according to the tension and excitement of the plot (a wonderful mixed-cut result of multiple dialogue segments can be obtained), further highlighting the expression effect and appeal of the audiobook.

[0055] As another example, for each of the selected dialogue segments, dry sound extraction technology may be used to remove background noise and / or incoherent background music in each of the dialogue segments to ensure the purity and auditory comfort of the dialogue segments.

[0056] It is understandable that the original text itself can contain dialogue segments of multiple characters, and the number of dialogue segments of all characters contained in the original text is greater than or equal to the number of dialogue segments selected from the above reply content.

[0057] As an example, the original text converted from the original audio of an audio book may include a speaking timestamp, and the prompt information input to the large language model may include an instruction to select the speaking timestamp of the text in the original text, and the reply content may include the speaking timestamp intercepted from the original text. The prompt information combined with the original text is input to the large language model, and the speaking timestamp of the wonderful content of the audio book generated by the large language model can greatly reduce the processing and storage burden.

[0058] The following takes the wonderful content generation strategy as the above-mentioned complete plot strategy as an example to exemplarily introduce a specific implementation of the prompt information (prompt) associated with the complete plot strategy input into the large language model:

[0059] / / Prompt start

[0060] Objective: You are an expert in novel tweets, provide an audiobook program script, and capture and output multiple complete and exciting story segments;

[0061] step:

[0062] (1) Clips that meet the following characteristics can be selected as highlights (content weight is not in any particular order):

[0063] a. Conflict Explosion: The conflict between the main characters reaches its peak, and the emotion and tension reach their highest point.

[0064] b. Emotional climax: The character's emotional experience reaches an extreme, such as deep sadness, joy, or anger.

[0065] c. Climax Battle: In an action or adventure story, the confrontation between the protagonist and the antagonist.

[0066] d. Revealing the truth: The secrets or mysteries hidden in the story are revealed, bringing surprise or shock to the readers.

[0067] e. Significant Sacrifice: The character makes a significant sacrifice for a higher goal or for others.

[0068] f. Deep dialogue: The dialogue between characters is profound and philosophical, which triggers readers to think deeply.

[0069] g. Touching scenes: emotional exchanges between characters, such as family affection, friendship or love, touch the readers' emotions.

[0070] h. Humorous elements: In a tense or heavy story, humorous passages can provide relief and make readers laugh.

[0071] (2) Length of the clips: Each clip cannot be too short and must be at least 120 seconds. Please ensure that the story content of the clip is complete and the beginning of the clip cannot be a turning point.

[0072] (3) Requirement on the number of clips to be selected: The number of output highlight clips shall not exceed {pd_number}.

[0073] Input format:

[0074] [Program number]-[timestamp]|[script content] ...

[0076] ----------------------------------

[0077] Output format:

[0078] [Program number]-[Start timestamp]-[End timestamp] ...

[0080] ----------------------------------

[0081] Sample output:

[0082] 4898990-00:01:00.012-00:01:55.222

[0083] 4898991-00:03:00.230-00:03:55.509 ...

[0085] / / At the end of prompt

[0086] The wonderful content of the audio book can be generated in a variety of ways based on the above reply content. As an example, when the wonderful content generation strategy is a strategy determined in the complete plot strategy or the multi-character dialogue strategy, the audio clip corresponding to the reply content can be cut out from the original audio of the audio book based on the reply content of the large language model, and the wonderful content of the audio book can be generated based on the cut audio clip. For example, when the reply content of the large language model includes a speech timestamp cut out from the original text, the audio clip corresponding to the speech timestamp can be cut out from the original audio based on the cut speech timestamp.

[0087] As another example, another method of generating wonderful content of an audiobook based on the reply content of a large language model may include: when the wonderful content generation strategy is a content simplification strategy, the reply content of the large language model may be converted into new audio, and the wonderful content of the audiobook is generated based on the new audio. As another example, when the wonderful content generation strategy is a content simplification strategy, the prompt information input into the large language model may include instructions for simplifying the above-mentioned original text, the reply content of the large language model may include new text obtained after the simplification of the original text, and the reply content is converted into new audio, that is, the new text contained in the reply content is converted into new audio. The content simplification strategy may be a strategy that does not directly use a fragment of the original audio as wonderful content, which can be designed specifically for long audiobook content, and aims to rewrite and refine the core and essence of the story of the audiobook through selected text (the new text contained in the reply content generated by the large language model relying on the prompt information), while stimulating the curiosity and desire to explore of the audience, improving the efficiency of content communication, and enhancing the expression of emotions and the attractiveness of the story.

[0088] There are many ways to convert the new text contained in the reply content into new audio. As an example, the new text contained in the reply content can be converted into new audio based on the Text To Speech (TTS) technology.

[0089] Considering that the above-mentioned new text may have multiple characteristics, and multiple characteristics may be suitable for different timbre generation methods, as an example, if the proportion of dialogues contained in the new text is greater than or equal to the proportion threshold, the characters corresponding to each dialogue in the new audio converted from the new text can be configured as different timbres (multi-timbre dialogue generation method); if the proportion of dialogues contained in the new text is less than the proportion threshold, the new audio converted from the new text is only configured as one timbre (storytelling-style single voice generation method), or, if the number of characters contained in the new text is greater than or equal to the character number threshold, each character in the new audio is configured as a different timbre; if the number of characters contained in the new text is less than the character number threshold, the new audio is only configured as one timbre. The new audio finally obtained according to the content simplification strategy can be flexible, and different timbre generation methods can be selected according to the different characteristics of the new text, such as intelligently selecting dialogue-style multi-timbre generation (multi-timbre dialogue generation method) or storytelling-style single voice generation method to adapt to rich character plots or narrative styles dominated by narration. This personalized dubbing method allows each character or narrator to be presented with a unique voice, further enriching the audience's listening experience.

[0090] As another example, in terms of timbre selection, the multi-timbre dialogue generation method can select appropriate timbre for each character based on the personality, gender, age and other factors of the characters appearing in the new text; while the storyteller-style single voice generation method can select a more appropriate timbre based on the type of audiobook and the specific content of the new text.

[0091] As an example, generating prompt information associated with the wonderful content generation strategy in combination with the original text, and then inputting the prompt information into the large language model may mean inputting the original text and the prompt information into the large language model synchronously, or may mean embedding the original text into the prompt information and then inputting it into the large language model, and there is no specific limitation on this.

[0092] As an example, the reply content of the large language model described in any of the above embodiments can be input into the text graph model, and the text graph model generates an image corresponding to the reply content, and then generates the wonderful content of the audio book based on the reply content and the image corresponding to the reply content. After the reply content is generated, the text graph model is used to configure the corresponding image for the reply content, and the reply content is combined with the image to generate wonderful content, which can bring a more vivid and visual content experience to the user. As another example, the above-mentioned text graph model can be an MJ (Midjourney) model or other text graph models, and there is no specific limitation on this.

[0093] As another example, when the generated wonderful content of an audio book is used in a scene where potential audio book consumers browse the opening of an audio book, or a scene where potential audio book consumers browse the ending of an audio book, advertising words and / or guiding words can be added to the wonderful content of the audio book, thereby further enhancing the attractiveness of the wonderful content and increasing the conversion rate and commercial value of the wonderful content.

[0094] As an example, after generating the wonderful content of an audio book, you can also match sound effects for the generated wonderful content of the audio book in the pre-built sound effect library, and configure the matched sound effects in the wonderful content of the audio book. The wonderful content is enriched by configuring sound effects, which enhances the emotional expression and environmental atmosphere of the audio. As another example, the wonderful content of the audio book can be mixed with sound effects (for the same audio clip) and transition spliced ​​(for different audio clips) through ffmpeg technology, ensuring the smoothness and professionalism of the audio output, providing users with a seamless and immersive listening experience.

[0095] As an example, the wonderful content of the generated audio book can be pushed to the user to attract the user to listen to the wonderful content of the audio book and thus generate interest in reading the entire audio book.

[0096] Corresponding to the above method embodiment, the present invention also provides a device for generating wonderful content of an audio book, see Figure 4 As shown, the device may include:

[0097] The acquisition unit 401 is used to acquire the original audio of the audio book and convert the original audio into original text;

[0098] The determination unit 402 is used to determine the wonderful content generation strategy of the audio book, and generate prompt information associated with the wonderful content generation strategy in combination with the original text; wherein the wonderful content generation strategy is a strategy determined in the complete plot strategy, the multi-character dialogue strategy or the content simplification strategy;

[0099] A generating unit 403, configured to input the prompt information into a large language model, and generate a reply content corresponding to the prompt information by the large language model;

[0100] The generating unit 403 is further configured to generate the wonderful contents of the audio book based on the reply contents.

[0101] As an example, the determination unit 402 is specifically used to determine the wonderful content generation strategy based on the user selection indication if a user selection indication is detected; if the user selection indication is not detected, determine the wonderful content generation strategy based on the characteristics of the original audio; the characteristics include audio characteristics of the original audio and category characteristics of the original audio.

[0102] As an example, the user selection instruction includes selection instructions for the complete plot strategy, the multi-character dialogue strategy, and the content condensed strategy, or the user selection instruction includes selection instructions for usage scenarios of the wonderful content of the audio book.

[0103] As an example, the audio features include at least the number of timbres contained in the original audio and the duration of the original audio; wherein, the multi-character dialogue strategy cannot be determined as the wonderful content generation strategy when the number of timbres is less than a timbres number threshold; the content simplification strategy cannot be determined as the wonderful content generation strategy when the duration is less than a duration threshold; when both the multi-character dialogue strategy and the content simplification strategy cannot be determined as the wonderful content generation strategy, the complete plot strategy is determined as the wonderful content generation strategy.

[0104] As an example, the original text includes a speaking timestamp; the reply content includes a speaking timestamp extracted from the original text.

[0105] As an example, the acquisition unit 401 is also used to construct a keyword list, which contains several specific nouns, and the specific nouns are used to replace the recognition results of the vocabulary associated with the specific nouns in the original audio; the acquisition unit 401 is specifically used to convert the original audio into the original text based on the keyword list.

[0106] The generating unit 403 is further configured to match sound effects for the generated wonderful contents of the audio book in a pre-built sound effect library, and configure the matched sound effects in the wonderful contents of the audio book.

[0107] As an example, the wonderful content generation strategy is a strategy determined in the complete plot strategy or the multi-character dialogue strategy; the generation unit 403 is specifically used to extract an audio clip from the original audio based on the reply content, and generate the wonderful content of the audio book based on the extracted audio clip.

[0108] As an example, when the wonderful content generation strategy is the complete plot strategy, the prompt information includes selection instructions for several complete target plots in the original text, and the several complete target plots respectively belong to different plot properties.

[0109] As an example, when the wonderful content generation strategy is the multi-character dialogue strategy, the prompt information includes selection instructions for dialogue segments of several characters in the original text, and fusion instructions for the selected dialogue segments, and the reply content includes the fusion results of the selected dialogue segments.

[0110] As an example, the wonderful content generation strategy is the content simplification strategy; the generation unit 403 is specifically used to convert the reply content into new audio, and generate the wonderful content of the audio book based on the new audio.

[0111] As an example, the prompt information includes instructions for simplifying the original text, and the reply content includes new text obtained after simplifying the original text; the generation unit 403 is specifically used to convert the new text into the new audio.

[0112] As an example, if the proportion of dialogues contained in the new text is greater than or equal to a proportion threshold, the characters corresponding to the various dialogues in the new audio are respectively configured with different timbres; if the proportion of dialogues contained in the new text is less than the proportion threshold, the new audio is configured with only one timbre, or, if the number of characters contained in the new text is greater than or equal to a character number threshold, then the various characters in the new audio are respectively configured with different timbres; if the number of characters contained in the new text is less than the character number threshold, then the new audio is configured with only one timbre.

[0113] As an example, the generation unit 403 is specifically used to input the reply content into a text graph model, and the text graph model generates an image corresponding to the reply content; based on the reply content and the image corresponding to the reply content, generate the wonderful content of the audio book.

[0114] As an example, the generating unit 403 is further configured to push the generated wonderful contents of the audio book to the user.

[0115] The present invention also provides an electronic device, such as Figure 5 As shown, the electronic device includes:

[0116] Processor 501;

[0117] Memory 502 for storing processor executable instructions;

[0118] The processor 501 is configured to implement the method for generating wonderful content of an audio book as described in any of the above embodiments.

[0119] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for generating wonderful content of an audio book described in any of the above embodiments is implemented.

[0120] The above description is only a specific implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for generating wonderful content of an audio book, characterized in that: include: Obtaining the original audio of the audio book, and converting the original audio into original text; Determine a wonderful content generation strategy of the audio book, and generate prompt information associated with the wonderful content generation strategy in combination with the original text; wherein the wonderful content generation strategy is a strategy determined in a complete plot strategy, a multi-character dialogue strategy, or a content simplification strategy; Inputting the prompt information into a large language model, and generating a reply content corresponding to the prompt information by the large language model; Based on the reply content, generate the wonderful content of the audio book.

2. The method according to claim 1, characterized in that The step of determining a strategy for generating wonderful content of the audio book includes: If a user selection indication is detected, determining the wonderful content generation strategy based on the user selection indication; If the user selection indication is not detected, the wonderful content generation strategy is determined based on the characteristics of the original audio; the characteristics include audio characteristics of the original audio and category characteristics of the original audio.

3. The method according to claim 2, characterized in that The user selection instruction includes selection instructions for the complete plot strategy, the multi-role dialogue strategy, and the content condensed strategy, or The user selection instruction includes a selection instruction for a usage scenario of the wonderful content of the audio book.

4. The method according to claim 2, characterized in that: The audio features at least include the number of timbres contained in the original audio and the duration of the original audio; Among them, the multi-character dialogue strategy cannot be determined as the wonderful content generation strategy when the number of timbres is less than the timbre number threshold; the content simplification strategy cannot be determined as the wonderful content generation strategy when the duration is less than the duration threshold; when both the multi-character dialogue strategy and the content simplification strategy cannot be determined as the wonderful content generation strategy, the complete plot strategy is determined as the wonderful content generation strategy.

5. The method according to claim 1, characterized in that The original text includes a speaking timestamp; the reply content includes a speaking timestamp captured from the original text.

6. The method according to claim 1, characterized in that Before converting the original audio into original text, the method further includes: Constructing a keyword list, wherein the keyword list includes a plurality of specific nouns, and the specific nouns are used to replace the recognition results of the vocabulary associated with the specific nouns in the original audio; The converting the original audio into original text comprises: Based on the keyword list, the original audio is converted into the original text.

7. The method according to claim 1, characterized in that After generating the wonderful content of the audio book, the method further includes: In the pre-built sound effect library, sound effects are matched for the generated wonderful content of the audio book, and the matched sound effects are configured in the wonderful content of the audio book.

8. A device for generating wonderful contents of an audio book, characterized in that: include: An acquisition unit, used for acquiring the original audio of the audio book and converting the original audio into original text; A determination unit, configured to determine a wonderful content generation strategy of the audio book, and generate prompt information associated with the wonderful content generation strategy in combination with the original text; wherein the wonderful content generation strategy is a strategy determined in a complete plot strategy, a multi-character dialogue strategy, or a content simplification strategy; A generating unit, configured to input the prompt information into a large language model, and generate a reply content corresponding to the prompt information by the large language model; The generating unit is further configured to generate the wonderful contents of the audio book based on the reply contents.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images