Audio generation method, apparatus and device, and storage medium

Abstract

The present invention provides an audio generation method, apparatus and device, and a storage medium. The audio generation method comprises: acquiring a target text; determining roles involved in the target text to obtain a plurality of roles; determining speaker information matching each role, and determining the role to which each sentence in the target text belongs; and on the basis of the target text, the speaker information matching each role, and the role to which each sentence in the target text belongs, synthesizing an audio corresponding to the target text. After the audio is synthesized, a sound effect and background music can be added to the synthesized audio. According to the audio generation method provided by the present invention, an audio with rich sound can be generated on the basis of a target text, so that the experience of listeners is good.

Description

Audio generation method, device, equipment and storage medium Technical Field

[0001] This application claims priority to a domestic application filed with the Patent Office of China on August 14, 2023, with application number 202311030882.1 and invention name “A method, device, equipment and storage medium for audio generation”, the entire contents of which are incorporated by reference into this application. Background Art

[0002] Text, as the most fundamental carrier of information, is also the most common method for expressing public sentiment. With the development of text-to-speech (TTS) technology, text can be smoothly and naturally converted into corresponding audio. In other words, TTS technology can synthesize audio corresponding to text. Audiobooks are a successful application scenario for TTS technology.

[0003] However, traditional standard TTS systems can only use one standard male or female voice for pronunciation, which is monotonous and provides poor listener experience.

[0004] Summary of the Invention

[0005] In view of this, the present invention provides an audio generation method, apparatus, device, and storage medium to solve the problem that the audio synthesized by the existing TTS system is monotonous and the listener experience is poor. The technical solution is as follows:

[0006] In a first aspect, a method for generating audio is provided, comprising:

[0007] Get the target text;

[0008] Determine the roles involved in the target text to obtain a number of roles;

[0009] Determining speaker information matching each role, and determining the role to which each sentence in the target text belongs;

[0010] According to the target text, the speaker information matching each role, and the role to which each sentence in the target text belongs, the audio corresponding to the target text is synthesized as the target audio.

[0011] Optionally, obtaining the target text includes:

[0012] Get the text generated by the user input;

[0013] Extracting text generation elements from the text generation requirements;

[0014] The preset large language model is called to generate elements according to the text to generate the target text.

[0015] Optionally, determining the roles involved in the target text includes:

[0016] When a broadcast operation for the target text is received, determining the role involved in the target text;

[0017] The audio generation method further includes: playing the target audio.

[0018] Optionally, the audio generation method further includes:

[0019] Determining the location to add the sound effect and the corresponding sound effect type based on the target text;

[0020] Filtering sound effects from a sound effect library according to the sound effect type;

[0021] According to the sound effect adding position, the filtered sound effect is added to the synthesized audio to obtain audio with the sound effect as the target audio.

[0022] Optionally, the audio generation method further includes:

[0023] determining the text type of the target text;

[0024] Selecting background music for the target text from a background music library according to the text type of the target text;

[0025] The filtered background music is fused with the synthesized audio to obtain audio with background music as the target audio.

[0026] Optionally, before selecting background music for the target text from a background music library according to the text type of the target text, the method further includes:

[0027] Performing plot division on the target text to obtain a plurality of plot texts;

[0028] Determine the plot type for each plot text;

[0029] The step of selecting background music for the target text from a background music library according to the text type of the target text includes:

[0030] According to the text type of the target text and the plot type of each plot text in the target text, background music is selected from a background music library for each plot text in the target text.

[0031] Optionally, determining the role to which each sentence in the target text belongs includes:

[0032] Determining the type of each sentence in the target text, wherein the type of each sentence is one of narration and dialogue;

[0033] The role to which each sentence of the dialogue type belongs is determined from the plurality of roles.

[0034] Optionally, the audio generation method further includes:

[0035] Determining the sentiment type of a sentence in the target text;

[0036] The step of synthesizing audio corresponding to the target text according to the target text, speaker information matching each role, and the role to which each sentence in the target text belongs includes:

[0037] The audio corresponding to the target text is synthesized according to the target text, the speaker information matching each role, the role to which each sentence in the target text belongs, and the emotion type of the sentence in the target text.

[0038] Optionally, determining speaker information matching each role includes:

[0039] Determine the role attributes of each character;

[0040] According to the role attributes of each role, speaker information matching each role is screened from the speaker information database, or, according to the role attributes of each role and in combination with the type of the target text, speaker information matching each role is screened from the speaker information database.

[0041] Optionally, the determining of roles involved in the target text to obtain several roles includes:

[0042] Extracting roles from the target text to obtain a plurality of roles;

[0043] Multiple roles corresponding to the same object among the multiple roles are merged to obtain a final role.

[0044] In a second aspect, an audio generation device is provided, comprising: a text acquisition module, a first role determination module, a speaker matching module, a second role determination module, and an audio synthesis module;

[0045] The text acquisition module is used to acquire the target text;

[0046] The first role determination module is used to determine the roles involved in the target text to obtain a plurality of roles;

[0047] The speaker matching module is used to determine the speaker information that matches each role;

[0048] The second role determination module is used to determine the role to which each sentence in the target text belongs;

[0049] The audio synthesis module is used to synthesize the audio corresponding to the target text as the target audio based on the target text, the speaker information matching each role, and the role to which each sentence in the target text belongs.

[0050] In a third aspect, an audio generating device is provided, comprising: a memory and a processor;

[0051] The memory is used to store programs;

[0052] The processor is used to execute the program to implement each step of the audio generation method described in any one of the above items.

[0053] In a fourth aspect, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned audio generation methods are implemented.

[0054] The audio generation method provided by the present invention, after obtaining a target text, first determines the characters involved in the target text to obtain a plurality of characters, then determines the speaker information matching each character, and determines the character to which each sentence in the target text belongs. Finally, based on the target text, the speaker information matching each character, and the character to which each sentence in the target text belongs, the audio corresponding to the target text is synthesized. The audio generation method provided by the present invention can determine matching speaker information for each character involved in the target text, and can determine the character to which each sentence in the target text belongs, thereby generating multi-speaker audio. That is, the audio generation method provided by the present invention can generate audio with rich sound based on the target text, providing a better listener experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0056] FIG1 is a schematic flow chart of an audio generation method provided by an embodiment of the present invention;

[0057] FIG2 is an example of a text generation interface provided by an embodiment of the present invention;

[0058] FIG3 is a schematic flow chart of another audio generation method provided by an embodiment of the present invention;

[0059] FIG4 is a schematic flow chart of another audio generation method provided by an embodiment of the present invention;

[0060] FIG5 is a schematic structural diagram of an audio generating device provided by an embodiment of the present invention;

[0061] FIG6 is a schematic structural diagram of an audio generating device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] Before introducing the solution of the present invention, the English terms involved in this article are first explained:

[0063] Prompt: Instructions. When communicating with an AI (such as a large artificial intelligence model), you need to send instructions to the AI. This can be a text description, such as "Please recommend me a pop song" when you are talking to the AI, or it can be a parameter description in a certain format, such as asking the AI ​​to draw a picture according to a certain format, which requires describing the relevant drawing parameters.

[0064] Large-scale AI models, also known as large-scale deep learning models, are AI models based on deep learning technology. They consist of hundreds of millions of parameters and can achieve complex tasks such as natural speech processing, image recognition, and speech recognition by learning and training on large amounts of data. Large-scale AI models can include large models and large language models. Both large models and large language models refer to machine learning models with very large parameters, but their application scenarios and focus differ slightly. The definitions of the two are provided below.

[0065] Large-Scale Models (LSMs) typically refer to large neural network models, such as deep neural networks and convolutional neural networks, which can have millions or even billions of parameters. These models are often used to process large datasets in fields such as image recognition and natural language processing. The advantage of large models is that they can learn more complex features and patterns, thereby improving model accuracy and generalization.

[0066] Large language models (LLMs) are language models built using technologies such as probabilistic graphical models or recurrent neural networks, and they also have very large parameters. These models are typically used for natural language processing tasks such as machine translation, text generation, and question-answering systems. The advantage of large language models is that they can understand and generate natural language, enabling more intelligent interactions. Common large language models include GPT4 and other large language models developed by other companies.

[0067] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0068] Next, the audio generation method provided by the present invention is introduced through the following embodiments.

[0069] Please refer to FIG1 , which shows a schematic flow chart of an audio generation method provided by an embodiment of the present invention. The method may include:

[0070] Step S101: Acquire target text.

[0071] The target text in this embodiment may be, but is not limited to, any of the following texts: stories, jokes, popular science, news reports, and the like.

[0072] In one possible implementation, the target text may be directly obtained from a specified path. In another possible implementation, the target text may be generated.

[0073] Optionally, the process of generating the target text may include:

[0074] Step a1: Obtain the text generation requirement input by the user.

[0075] The text generation requirement input by the user in the text generation interface can be obtained. Please refer to Figure 2, which shows an example of the text generation interface. "Please help me write a children's story with the theme of little rabbits" in Figure 2 is the text generation requirement input by the user.

[0076] Optionally, in order to improve the text generation effect, the text generation interface may display input guidance information, and the input guidance information is used to guide the user to input key information to improve the quality of the generated text.

[0077] In addition, it should be noted that the user can input the text generation requirements through text input or through voice input. If the text generation requirements obtained in step a1 are input by the user through text input, then after obtaining the user input, the subsequent step a2 is directly executed. If the text generation requirements obtained in step a1 are input by the user through voice input, the voice is converted into text, and then the subsequent step a2 is executed.

[0078] Step a2: extract text generation elements from text generation requirements.

[0079] For example, the text generation requirement is “Please help me write a children's story with the theme of little rabbits”, then the theme “little rabbits” and the key element “children's story” can be extracted from the text generation requirement.

[0080] Optionally, a pre-trained feature extraction model can be used to extract text generation features from text generation requirements, wherein the feature extraction model uses training text (text generation requirements) annotated with features, so that the features extracted by the feature extraction model from the training text tend to be consistent with the features annotated in the training text as the training goal.

[0081] In addition to extracting text generation elements from text generation requirements based on the element extraction model, other methods can also be used to extract text generation elements from text generation requirements. For example, element extraction rules can be set in advance, and then text generation elements can be extracted from text generation requirements based on the set element extraction rules.

[0082] Step a3: Call the preset large language model, generate elements based on the text, and generate the target text.

[0083] Specifically, calling a preset large language model and generating a target text based on text generation elements may include: obtaining a preconfigured prompt format template, the prompt format template includes a user input information slot, and the first prompt format template is used to instruct the large language model to generate text based on the information in the user input information slot; filling the text generation elements extracted from the text generation requirements input by the user into the user input information slot to obtain an edited instruction prompt; inputting the edited instruction prompt into the large language model to obtain the text output by the large language model.

[0084] For example, the text generation requirement is "Please help me write a children's story about little rabbits." The text generation elements extracted from this text generation requirement are "little rabbit" and "children's story." "Little rabbit" and "children's story" are entered into the user input information slot in the prompt format template to obtain the edited instruction prompt. The edited instruction prompt is input into the large language model, which generates and outputs the story text based on "little rabbit" and "children's story." The framed content in Figure 2, "Once upon a time, there was a little rabbit living in the forest. He was very clever... In the end, the little rabbit understood a truth: as long as you have courage and wisdom, you can overcome any difficulty." is the story text generated by the large language model.

[0085] Step S102: Determine the roles involved in the target text and obtain a number of roles.

[0086] Optionally, the process of determining the roles involved in the target text and obtaining a number of roles may include:

[0087] Step b1: extract roles from the target text to obtain a number of roles.

[0088] Optionally, role entities can be identified from the target text based on a pre-trained role entity recognition model, where the identified role entities are the roles extracted from the target text, wherein the role entity recognition model is trained using training text in which role entities are marked.

[0089] Optionally, the role entity recognition model can be but is not limited to a seq2seq model (the training text is annotated with an attribute label for each word, and the attribute label of a word is used to indicate whether the word belongs to a role entity). The role entity recognition model can predict whether each word in the input text belongs to a role entity, and then determine the role entity based on the prediction result of each word.

[0090] Step b2: Merge multiple roles corresponding to the same object among the roles to obtain a final role.

[0091] Multiple roles corresponding to the same object may be determined from a number of roles, and then the multiple roles corresponding to the same object may be merged.

[0092] Exemplarily, the roles obtained through step b1 include "Zhang San", "Xiao Zhang", "Teacher Zhang", "Xiao Li", and "Teacher Wang". Among them, "Zhang San", "Xiao Zhang", and "Teacher Zhang" correspond to the same person, so "Zhang San", "Xiao Zhang", and "Teacher Zhang" can be merged into "Zhang San", and finally "Zhang San", "Xiao Li", and "Teacher Wang" are obtained.

[0093] Step S103: Determine the speaker information matching each role, and determine the role to which each sentence in the target text belongs.

[0094] There are multiple ways to determine the speaker information that matches each character. In one possible implementation, the role attributes of each character can be first determined, and then the speaker information that matches each character can be filtered from the speaker information database based on the role attributes of each character. In another possible implementation, the role attributes of each character can be first determined, and then the speaker information that matches each character can be filtered from the speaker information database based on the role attributes of each character and the type of the target text.

[0095] The character attributes may include, but are not limited to, gender, age, and personality traits. For example, the character attributes of character A are "male, childish, gentle," the character attributes of character B are "female, childish, lively," and the character attributes of character C are "female, childish, cute." The method for determining the type of the target text will be described in the following examples.

[0096] The process of determining the role to which each sentence in the target text belongs may include: determining the type of each sentence in the target text, wherein each sentence is of a narration or a dialogue type; and determining the role to which each sentence of the dialogue type belongs from among a plurality of roles. When determining the role to which a sentence belongs, the role to which the sentence belongs may be determined based on the content of the sentence. To more accurately determine the role to which the sentence belongs, the role to which the sentence belongs may be determined based on the content of the sentence and the contextual information of the sentence. It should be noted that the role to which a sentence of the narration type belongs is the narration.

[0097] Optionally, in addition to determining the role to which each sentence in the target text belongs, the sentiment type of each sentence in the target text may also be determined. Optionally, the sentiment type of the sentences in the target text may be determined based on a pre-trained sentiment classification model, wherein the sentiment classification model uses training text annotated with sentiment types, and is trained so that the sentiment type predicted by the sentiment classification model for the training text is consistent with the sentiment type annotated in the training text.

[0098] Here are some examples of the role each sentence in the target text belongs to and the sentiment type of each sentence:

[0099] Narrator: The little bear was walking on the grass when he suddenly heard a noise.

[0100] Little Bear (confused): What’s that sound?

[0101] Rabbit (Neutral): It's me, Little Bear! I've been waiting for you here.

[0102] Little Bear (confused): Oh, it's the rabbit! How come you are here?

[0103] Rabbit (Neutral): I heard there is a mysterious place in this forest. I want to go exploring with you.

[0104] Little Bear (happy): Okay, let’s go quickly!

[0105] Narrator: They came to a meadow and saw a squirrel jumping on a tree.

[0106] Little Bear (excited): That squirrel is so amazing! How did it jump up such a high tree?

[0107] Squirrel (happy): Haha, this is a piece of cake for me. You guys should try it too!

[0108] It should be noted that the narration is a special role. For the text of the narration, the emotional type may not be determined, or the emotional type of the narration text may be determined to be a specified emotional type (for example, neutral).

[0109] Step S104: synthesize the audio corresponding to the target text according to the target text, the speaker information matching each role, and the role to which each sentence in the target text belongs.

[0110] Based on the speech synthesis model, the audio corresponding to the target text can be synthesized according to the target text, the speaker information matching each role, and the role to which each sentence in the target text belongs.

[0111] The above content mentioned that in addition to determining the role to which each sentence in the target text belongs, the emotional type of the sentence in the target text can also be determined. If the emotional type of the sentence in the target text is determined, the audio corresponding to the target text can be synthesized based on the target text, the speaker information matching each role, the role to which each sentence in the target text belongs, and the emotional type of the sentence in the target text.

[0112] In one possible implementation, after obtaining the target text, steps S102 to S104 may be directly executed. In another possible implementation, after obtaining the target text, it is detected whether a user's broadcast operation for the target text is received. When the user's broadcast operation for the target text is received, steps S102 to S104 are executed. As shown in FIG2 , FIG2 shows an interface with a broadcast icon. When the user clicks the broadcast icon with a finger or an input device, the broadcast operation is received, and then steps S102 to S104 are executed. After generating audio corresponding to the target text, the generated audio can be played.

[0113] The audio generation method provided by the embodiment of the present invention, after obtaining the target text, first determines the roles involved in the target text to obtain several roles, then determines the speaker information matching each role, and determines the role to which each sentence in the target text belongs, and finally synthesizes the audio corresponding to the target text based on the target text, the speaker information matching each role, and the role to which each sentence in the target text belongs. The audio generation method provided by the embodiment of the present invention can determine the matching speaker information for each role involved in the target text, and can determine the role to which each sentence in the target text belongs, and can then generate multi-speaker audio, that is, the audio generation method provided by the embodiment of the present invention can generate audio with rich sound based on the target text, and the listener experience is better.

[0114] In order to improve the listener experience, an embodiment of the present invention provides another audio generation method, which may include:

[0115] Step S301: Acquire target text.

[0116] Step S302: Determine the roles involved in the target text and obtain a number of roles.

[0117] Step S303: Determine the speaker information matching each role, and determine the role to which each sentence in the target text belongs.

[0118] Step S304: synthesize the audio corresponding to the target text according to the target text, the speaker information matching each role, and the role to which each sentence in the target text belongs.

[0119] The specific implementation process and related description of steps S301 to S304 can be found in the specific implementation process and related description of steps S101 to S104 in the above embodiment, and will not be repeated in this embodiment.

[0120] Step S305: Determine the sound effect adding position and the corresponding sound effect type according to the target text.

[0121] There are several ways to determine the location and type of sound effects to add based on the target text:

[0122] In one possible implementation, the target text may be input into a pre-trained sound effect information prediction model to obtain the sound effect addition location and corresponding sound effect type predicted by the sound effect information prediction model (the sound effect type corresponding to the sound effect addition location refers to the type of sound effect added at the sound effect addition location). The sound effect information prediction model is trained using training text annotated with the sound effect addition location and corresponding sound effect type. When training the sound effect information prediction model, a training goal is to ensure that the sound effect addition location and sound effect type predicted by the sound effect information prediction model for the training text are consistent with the sound effect addition location and sound effect type annotated in the training text.

[0123] In another possible implementation, a sound effect trigger word list can be pre-constructed, which includes multiple sound effect trigger words and sound effect types corresponding to the multiple sound effect trigger words. A word that is identical to a sound effect trigger word in the sound effect trigger word list can be identified from the target text as a first target word, and the position of the first target word is determined as the sound effect adding position, and the sound effect type corresponding to the sound effect trigger word that is identical to the first target word is determined as the sound effect type corresponding to the sound effect adding position. Considering that words with the same semantics may have different descriptions, a word with the same or similar semantics as the sound effect trigger word in the sound effect trigger word list can be identified from the target text as a second target word, and the position of the second target word is determined as the sound effect adding position, and the sound effect type corresponding to the sound effect trigger word with the same or similar semantics as the second target word is determined as the sound effect type corresponding to the sound effect adding position.

[0124] Step S306: Filtering sound effects from the sound effect library according to the sound effect type.

[0125] The sound effect library includes sound effects that match various sound effect types. After obtaining the sound effect type in step S305, sound effects that match the sound effect type can be filtered out from the sound effect library. For example, if the sound effect type obtained in step S305 is thunder, sound effects that match thunder are obtained from the sound effect library.

[0126] It should be noted that this embodiment does not limit steps S305 to S306 to be executed after step S304 . Steps S305 to S306 only need to be executed after step S301 and before step S307 .

[0127] Step S307: Add the filtered sound effects to the synthesized audio according to the sound effect adding position to obtain audio with the sound effects.

[0128] Specifically, the sound effect adding position determined by step S305 is the sound effect adding position at the text level. When adding sound effects to the synthesized audio, the sound effect adding position at the audio level is first determined based on the sound effect adding position at the text level, and then the filtered sound effects are added to the sound effect adding position at the audio level, thereby obtaining audio with sound effects corresponding to the target text.

[0129] Sound effects can help create mental images in the listener's mind and increase their attention. For example, story audio may contain expressions that describe or indicate certain sounds. Adding sound effects to these expressions can enhance the story's atmosphere, help the listener recreate the story's image, and increase their attention.

[0130] In one possible implementation, after obtaining the target text, steps S302 to S307 may be directly executed. In another possible implementation, after obtaining the target text, it may be detected whether a user's operation for announcing the target text is received. When a user's operation for announcing the target text is received (e.g., the user clicks the announcement icon with a finger or an input device), steps S302 to S307 are executed. After generating audio with sound effects corresponding to the target text, the audio with sound effects corresponding to the target text may be played.

[0131] The audio generation method provided by the embodiment of the present invention can determine the matching speaker information for each role involved in the target text, and can determine the role to which each sentence in the target text belongs, and then can synthesize multi-speaker audio. After synthesizing the multi-speaker audio, it can also determine the sound effect addition position and the corresponding sound effect type based on the target text, and filter the sound effects from the sound effect library based on the sound effect type, and then can add the filtered sound effects to the synthesized audio. The audio generation method provided by the embodiment of the present invention can generate multi-speaker audio with sound effects. Synthesizing multi-speaker audio can enhance the sound richness of the audio, and adding sound effects to the synthesized audio can enhance the listener's listening experience and attention.

[0132] In order to further enhance the listening experience of the listener, an embodiment of the present invention provides another audio generation method, which may include:

[0133] Step S401: Acquire target text.

[0134] Step S402: Determine the roles involved in the target text and obtain a number of roles.

[0135] Step S403: Determine the speaker information matching each role, and determine the role to which each sentence in the target text belongs.

[0136] Step S404: synthesize the audio corresponding to the target text according to the target text, the speaker information matching each role, and the role to which each sentence in the target text belongs.

[0137] The specific implementation process and related description of steps S401 to S404 can be found in the specific implementation process and related description of steps S101 to S104 in the above embodiment, and will not be repeated in this embodiment.

[0138] Step S405: Determine the sound effect adding position and the corresponding sound effect type according to the target text.

[0139] Step S406: Filter the sound effects from the sound effect library according to the sound effect type.

[0140] The specific implementation process and related description of steps S405 to S406 can be found in the specific implementation process and related description of steps S305 to S306 in the above embodiment, and will not be repeated in this embodiment. In addition, this embodiment does not limit steps S405 to S406 to be executed after step S404. Steps S405 to S406 can be executed after step S401 and before step S409.

[0141] Step S407: Determine the text type of the target text.

[0142] For example, if the target text is a story text, the text type of the target text may be one of the following types: children's story, idiom story, fairy tale, fable, modern novel, etc.

[0143] Optionally, the text type of the target text can be determined based on a pre-trained text classification model, wherein the text classification model uses training texts labeled with text types so that the text type predicted by the text classification model for the training texts is consistent with the text type labeled with the training texts as a training target. For example, if the target text is a story text, a story text classification model can be pre-trained using training story texts labeled with text types. The target text is then input into the story text classification model for classification to determine the text type of the target text.

[0144] Step S408: Filter background music for the target text from the background music library according to the text type of the target text.

[0145] In one possible implementation, background music can be selected from a background music library based solely on the target text's text type. The background music library includes background music that matches various text types, such as children's stories, idiom stories, and fairy tales. Assuming the target text is a story text and its text type is children's story, background music matching children's stories will be selected from the background music library.

[0146] In another possible implementation method, the target text can be divided into plots to obtain several plot texts, and the plot type of each plot text can be determined (such as relaxed, depressing, warm, sad). Then, background music can be selected from the background music library for each plot text in the target text based on the text type of the target text and the plot type of each plot text in the target text.

[0147] Optionally, the target text may be plot-divided based on a pre-trained plot division model to obtain a number of plot texts, wherein the plot division model is trained using training text annotated with plot division information, and the plot division information annotated in the training text may include attribute labels for each text unit (such as a character or word) in the training text, wherein the attribute label of a text unit is used to indicate whether the position of the text unit is the starting position, middle position, or end position of a plot. When the target text is plot-divided based on the pre-trained plot division model, the attribute label of each text unit in the target text is predicted based on the pre-trained plot division model, and then the plot text is determined based on the attribute label of each text unit.

[0148] Optionally, when dividing the target text into plots, the target text can be divided into plots according to the plot development process. For example, if the target text is a story text, the target text can be divided into four plot texts according to the plot development process of the story: narrative, conflict, climax, and ending.

[0149] After obtaining a number of plot texts, the plot type of each plot text can be determined based on a pre-trained plot type determination model. The plot type determination model is trained using training plot texts annotated with plot types, with the goal of ensuring that the plot types predicted by the plot type determination model for the training plot texts are consistent with the annotated plot types of the training plot texts. It should be noted that adjacent plot texts within the number of plot texts may have different plot types, while non-adjacent plot texts may have the same plot type.

[0150] After obtaining the text type of the target text and the plot type of each plot text in the target text, background music can be selected from the background music library based on this. The background music library includes background music that matches each text type, and the background music that matches each text type includes background music that matches each plot type. First, background music that matches the text type of the target text is selected from the background music library. Then, background music that matches the plot type of each plot text is selected from the background music that matches the text type of the target text, thereby obtaining background music for each plot in the target text.

[0151] Exemplarily, the target text is a story text, and the text type of the target text is a children's story. The target text is divided into four plot texts. The plot type of the first plot text is light-hearted, the plot type of the second plot text is depressing, the plot type of the third plot text is warm, and the plot type of the fourth plot text is sad. First, background music matching the children's story is screened from the background music library, and then background music matching the plot type of the first plot text (i.e., light-hearted) is screened from the background music matching the children's story as the background music for the first plot text. Background music matching the plot type of the second plot text (i.e., depressing) is screened from the background music matching the children's story as the background music for the second plot text. Background music matching the plot type of the third plot text (i.e., warm) is screened from the background music matching the children's story as the background music for the third plot text. Background music matching the plot type of the fourth plot text (i.e., sad) is screened from the background music matching the children's story as the background music for the fourth plot text.

[0152] It should be noted that this embodiment does not limit steps S407 to S408 to be executed after step S406 . Steps S407 to S408 only need to be executed after step S401 and before step S409 .

[0153] Step S409: The filtered background music and the filtered sound effects are merged with the synthesized audio to obtain audio with sound effects and background music.

[0154] When fusing sound effects, the selected sound effects are merged with the synthesized audio based on their placement. If the selected background music is for the entire target text, it is merged with the entire synthesized audio. If the selected background music is for each plot text in the target text, the background music for each plot text is merged with the corresponding audio segment.

[0155] In one possible implementation, after obtaining the target text, steps S402 to S409 may be directly executed. In another possible implementation, after obtaining the target text, it may be detected whether a user's operation for reading the target text is received. When a user's operation for reading the target text is received, steps S402 to S409 are executed, and after generating audio with sound effects and background music corresponding to the target text, the audio with sound effects and background music corresponding to the target text may be played.

[0156] The audio generation method provided by the embodiment of the present invention can determine the matching speaker information for each role involved in the target text, and can determine the role to which each sentence in the target text belongs, and then can synthesize multi-speaker audio. After synthesizing the multi-speaker audio, the sound effect addition position and the corresponding sound effect type can be determined according to the target text, and then the sound effects can be filtered from the sound effect library according to the sound effect type. The type of the target text and the plot type of each plot contained in the target text can also be determined, and background music can be filtered from the background music library according to the type of the target text and the plot type of each plot contained in the target text. Then, the filtered sound effects and background music can be merged with the synthesized audio. The audio generation method provided by the embodiment of the present invention can generate multi-speaker audio with sound effects and background music. Synthesizing multi-speaker audio can enhance the sound richness of the audio, and adding sound effects and background music to the synthesized audio can enhance the listener's listening experience and attention.

[0157] It should be noted that the audio generation method provided in the above embodiment can add sound effects and background music to the synthesized audio after synthesizing the audio corresponding to the target text. In another embodiment of the present invention, after synthesizing the audio corresponding to the target text, only background music can be added to the synthesized audio, that is, after synthesizing the audio corresponding to the target text, the text type of the target text is determined, and according to the text type of the target text, background music is filtered for the target text from the background music library (or, the text type of the target text is determined, the target text is divided into plots, and the plot type of each plot text obtained by the division is determined, and background music is filtered for each plot text in the target text according to the text type of the target text and the plot type of each plot text), and then the filtered background music is merged with the synthesized audio to obtain audio with background music corresponding to the target text.

[0158] An embodiment of the present invention further provides an audio generating device. The audio generating device provided by the embodiment of the present invention is described below. The audio generating device described below and the audio generating method described in the text above can be referenced to each other.

[0159] Please refer to Figure 5, which shows a structural diagram of the audio generation device provided by an embodiment of the present invention, which may include: a text acquisition module 501, a first role determination module 502, a speaker matching module 503, a second role determination module 504 and an audio synthesis module 505.

[0160] The text acquisition module 501 is used to acquire the target text.

[0161] The first role determination module 502 is configured to determine the roles involved in the target text and obtain a plurality of roles.

[0162] The speaker matching module 503 is used to determine speaker information that matches each role.

[0163] The second role determination module 504 is configured to determine the role to which each sentence in the target text belongs.

[0164] The audio synthesis module 505 is used to synthesize the audio corresponding to the target text as the target audio based on the target text, the speaker information matching each role, and the role to which each sentence in the target text belongs.

[0165] Optionally, the text acquisition module 501 may include: a text generation requirement acquisition module, a text generation element acquisition module and a text generation module.

[0166] The text generation requirement acquisition module is used to obtain the text generation requirement input by the user.

[0167] The text generation element acquisition module is used to extract text generation elements from the text generation requirements.

[0168] The text generation module is used to call a preset large language model and generate a target text according to the text generation elements.

[0169] Optionally, when determining the role involved in the target text, the text acquisition module 501 is specifically configured to determine the role involved in the target text when a broadcast operation for the target text is received.

[0170] The audio generation device provided in the embodiment of the present invention may further include: a text broadcast module.

[0171] The text broadcast module is used to play the target audio.

[0172] Optionally, the audio generating device provided by the embodiment of the present invention may further include: a sound effect information determining module, a sound effect screening module, and a sound effect adding module.

[0173] A sound effect information determination module, configured to determine a sound effect adding location and a corresponding sound effect type based on the target text;

[0174] The sound effect screening module is used to screen the sound effects from the sound effect library according to the sound effect type.

[0175] The sound effect adding module is used to add the screened sound effects to the synthesized audio according to the sound effect adding position to obtain the audio with the sound effect as the target audio.

[0176] Optionally, the audio generation device provided by the embodiment of the present invention may further include: a text type determination module, a background music screening module, and a background music fusion module.

[0177] The text type determination module is used to determine the text type of the target text.

[0178] The background music screening module is used to screen background music for the target text from a background music library according to the text type of the target text.

[0179] The background music fusion module is used to fuse the filtered background music with the synthesized audio to obtain audio with background music as the target audio.

[0180] Optionally, the audio generation device provided by the embodiment of the present invention may further include: a plot division module and a plot type determination module.

[0181] The plot division module is used to divide the target text into plots to obtain a plurality of plot texts.

[0182] The plot type determination module is used to determine the plot type of each plot text.

[0183] The background music screening module is specifically used to screen background music for the target text from the background music library according to the text type of the target text:

[0184] According to the text type of the target text and the plot type of each plot text in the target text, background music is selected from a background music library for each plot text in the target text.

[0185] Optionally, when determining the role to which each sentence in the target text belongs, the second role determination module 504 is specifically configured to:

[0186] Determining the type of each sentence in the target text, wherein the type of each sentence is one of narration and dialogue;

[0187] The role to which each sentence of the dialogue type belongs is determined from the plurality of roles.

[0188] Optionally, the audio generation device provided by the embodiment of the present invention may further include: an emotion type determination module.

[0189] The emotion type determination module is used to determine the emotion type of the sentence in the target text.

[0190] The audio synthesis module is specifically used to synthesize the audio corresponding to the target text based on the target text, the speaker information matching each role, and the role to which each sentence in the target text belongs:

[0191] The audio corresponding to the target text is synthesized according to the target text, the speaker information matching each role, the role to which each sentence in the target text belongs, and the emotion type of the sentence in the target text.

[0192] Optionally, when determining speaker information matching each role, the speaker matching module 503 is specifically configured to:

[0193] Determine the role attributes of each character;

[0194] According to the role attributes of each role, speaker information matching each role is screened from the speaker information database, or, according to the role attributes of each role and in combination with the type of the target text, speaker information matching each role is screened from the speaker information database.

[0195] Optionally, when determining the roles involved in the target text and obtaining a plurality of roles, the first role determination module 502 is specifically configured to:

[0196] Extracting roles from the target text to obtain a plurality of roles;

[0197] Multiple roles corresponding to the same object among the multiple roles are merged to obtain a final role.

[0198] The audio generation device provided by the embodiment of the present invention can determine the matching speaker information for each role involved in the target text, and can determine the role to which each sentence in the target text belongs, and then synthesize multi-speaker audio. That is, the audio generation device provided by the embodiment of the present invention can generate audio with rich sound based on the target text, and the listener experience is better.

[0199] In addition, after synthesizing multi-speaker audio, the audio generation device provided by the embodiment of the present invention can also determine the sound effect addition position and the corresponding sound effect type according to the target text, filter the sound effects from the sound effect library according to the sound effect type, and can also determine the type of the target text and the plot type of each plot contained in the target text, and filter the background music from the background music library according to the type of the target text and the plot type of each plot contained in the target text, and then, the filtered sound effects and background music can be merged with the synthesized audio. The audio generation device provided by the embodiment of the present invention can add sound effects and background music to the synthesized multi-speaker audio, and adding sound effects and background music to the synthesized audio can enhance the listener's listening experience and attention.

[0200] An embodiment of the present invention further provides an audio generating device. Please refer to Figure 6, which shows a structural diagram of the audio generating device. The audio generating device may include: at least one processor 601, at least one communication interface 602, at least one memory 603 and at least one communication bus 604.

[0201] In the embodiment of the present invention, there is at least one processor 601 , communication interface 602 , memory 603 , and communication bus 604 , and the processor 601 , communication interface 602 , and memory 603 communicate with each other via the communication bus 604 .

[0202] The processor 601 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0203] The memory 603 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0204] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0205] Get the target text;

[0206] Determine the roles involved in the target text to obtain a number of roles;

[0207] Determining speaker information matching each role, and determining the role to which each sentence in the target text belongs;

[0208] According to the target text, the speaker information matching each role, and the role to which each sentence in the target text belongs, the audio corresponding to the target text is synthesized as the target audio.

[0209] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0210] An embodiment of the present invention further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0211] Get the target text;

[0212] Determine the roles involved in the target text to obtain a number of roles;

[0213] Determining speaker information matching each role, and determining the role to which each sentence in the target text belongs;

[0214] According to the target text, the speaker information matching each role, and the role to which each sentence in the target text belongs, the audio corresponding to the target text is synthesized as the target audio.

[0215] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0216] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0217] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0218] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An audio generation method, characterized in that: include: Get the target text; Determine the roles involved in the target text to obtain a number of roles; Determine speaker information matching each role, and determine the role to which each sentence in the target text belongs; According to the target text, the speaker information matching each role, and the role to which each sentence in the target text belongs, the audio corresponding to the target text is synthesized as the target audio.

2. The audio generation method according to claim 1, characterized in that: The obtaining of the target text comprises: Get the text input by the user to generate requirements; Extracting text generation elements from the text generation requirements; The preset large language model is called to generate target text according to the text generation elements.

3. The audio generation method according to claim 1 or 2, characterized in that: The determining of the roles involved in the target text includes: When receiving a broadcast operation for the target text, determining the role involved in the target text; The audio generation method also includes: playing the target audio.

4. The audio generation method according to claim 1, characterized in that: Also includes: Determine the location where the sound effect is to be added and the corresponding sound effect type according to the target text; Filtering sound effects from a sound effect library according to the sound effect type; According to the sound effect adding position, the screened sound effect is added to the synthesized audio to obtain the audio with the sound effect as the target audio.

5. The audio generation method according to claim 1, characterized in that: Also includes: Determining the text type of the target text; According to the text type of the target text, selecting background music for the target text from a background music library; The filtered background music is fused with the synthesized audio to obtain audio with background music as the target audio.

6. The audio generation method according to claim 5, characterized in that: Before selecting background music for the target text from a background music library according to the text type of the target text, the method further includes: Performing plot division on the target text to obtain a plurality of plot texts; Determine the plot type for each plot text; The step of selecting background music for the target text from a background music library according to the text type of the target text comprises: According to the text type of the target text and the plot type of each plot text in the target text, background music is screened for each plot text in the target text from a background music library.

7. The audio generation method according to claim 1, characterized in that: Determining the role to which each sentence in the target text belongs includes: Determining the type of each sentence in the target text, wherein the type of each sentence is one of narration and dialogue; The role to which each sentence of the dialogue type belongs is determined from the plurality of roles.

8. The audio generation method according to claim 1, characterized in that: Also includes: Determining the sentiment type of the sentence in the target text; The step of synthesizing the audio corresponding to the target text according to the target text, speaker information matching each role, and the role to which each sentence in the target text belongs includes: The audio corresponding to the target text is synthesized according to the target text, information of a speaker matching each role, the role to which each sentence in the target text belongs, and the emotion type of the sentence in the target text.

9. The audio generation method according to claim 1, characterized in that: The step of determining speaker information matching each role includes: Determine the role attributes of each character; According to the role attributes of each role, filter the speakers that match each role from the speaker information database. Speaker information, or, based on the role attributes of each role and in combination with the type of the target text, speaker information matching each role is screened from a speaker information database.

10. The audio generation method according to claim 1, characterized in that: The determining of the roles involved in the target text to obtain a plurality of roles includes: Extracting roles from the target text to obtain a plurality of roles; Multiple roles corresponding to the same object among the several roles are merged to obtain a final role.

11. An audio generating device, characterized in that: include: A text acquisition module, a first role determination module, a speaker matching module, a second role determination module and an audio synthesis module; The text acquisition module is used to acquire the target text; The first role determination module is used to determine the roles involved in the target text to obtain a plurality of roles; The speaker matching module is used to determine speaker information matching each role; The second role determination module is used to determine the role to which each sentence in the target text belongs; The audio synthesis module is used to synthesize the audio corresponding to the target text as the target audio according to the target text, the speaker information matching each role and the role to which each sentence in the target text belongs.

12. An audio generating device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the audio generation method according to any one of claims 1 to 10.

13. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the audio generation method according to any one of claims 1 to 10 is implemented.

Images