Audio generation method and device based on large language model

Abstract

The present disclosure provides an audio generation method based on a large language model, which relates to artificial intelligence technology fields such as large language models, natural language processing, deep learning, and audio generation. The audio generation method based on a large language model includes: obtaining a text to be processed; parsing the text to be processed using a large language model to obtain character information and emotional information corresponding to the text to be processed; obtaining a target reference text and a target reference audio based on the character information and the emotional information; generating a target audio corresponding to the text to be processed based on the text to be processed, the target reference text, and the target reference audio. The present disclosure can improve the accuracy of the obtained character information and emotional information, so that the generated target audio is more consistent with the character corresponding to the text to be processed and the emotion of the character, thereby improving the accuracy of the target audio and enhancing the authenticity of the target audio.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to artificial intelligence technologies such as large language models, natural language processing, deep learning, and audio generation. A method, apparatus, electronic device, and readable storage medium for audio generation based on a large language model are provided. Background Art

[0002] When generating audio, it's crucial to ensure the highest possible accuracy and authenticity. Existing technologies typically combine an emotion classification model with a deep learning-based sound synthesis model. However, both models require extensive training with labeled data, resulting in high training costs and low training efficiency, which in turn reduces the efficiency and accuracy of audio generation. Summary of the Invention

[0003] According to a first aspect of the present disclosure, an audio generation method based on a large language model is provided, comprising: obtaining a text to be processed; parsing the text to be processed using a large language model to obtain character information and emotional information corresponding to the text to be processed; obtaining a target reference text and a target reference audio based on the character information and the emotional information; and generating a target audio corresponding to the text to be processed based on the text to be processed, the target reference text, and the target reference audio.

[0004] According to a second aspect of the present disclosure, an audio generation device based on a large language model is provided, including: an acquisition unit for acquiring a text to be processed; a parsing unit for parsing the text to be processed using a large language model to obtain character information and emotional information corresponding to the text to be processed; a processing unit for obtaining a target reference text and a target reference audio based on the character information and the emotional information; and a generation unit for generating a target audio corresponding to the text to be processed based on the text to be processed, the target reference text and the target reference audio.

[0005] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0006] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method as described above.

[0007] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method described above when executed by a processor.

[0008] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0010] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0011] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0012] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0013] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0014] Figure 5 It is a block diagram of an electronic device used to implement the audio generation method based on a large language model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, and various details of the embodiments of the present disclosure are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and mechanisms are omitted in the following description.

[0016] Figure 1 Schematic diagram of the first embodiment of the present disclosure. Figure 1 As shown, the audio generation method based on the large language model of this embodiment specifically includes the following steps:

[0017] S101, obtaining the text to be processed;

[0018] S102, parsing the text to be processed using a large language model to obtain character information and sentiment information corresponding to the text to be processed;

[0019] S103, obtaining a target reference text and a target reference audio according to the character information and the emotion information;

[0020] S104: Generate a target audio corresponding to the text to be processed according to the text to be processed, the target reference text and the target reference audio.

[0021] The audio generation method based on the large language model of this embodiment, on the one hand, parses the text to be processed through the large language model, and with the help of the large language model's own powerful text comprehension ability, can improve the accuracy of the obtained character information and emotional information, and thus improve the accuracy of the obtained target reference text and target reference audio. On the other hand, in addition to using the text to be processed, the target reference text and target reference audio are also combined to generate the target audio. Since the target reference text and target reference audio are obtained based on the character information and emotional information, the generated target audio can be made to better match the character corresponding to the text to be processed and the emotion of the character, thereby improving the accuracy of the generated target audio and enhancing the authenticity of the generated target audio.

[0022] In addition, since the character information and emotional information in this embodiment are obtained based on the current text to be processed, this embodiment can switch the reference text and reference audio used when generating audio when the character and / or emotion changes for different texts to be processed, thereby achieving the purpose of generating different audio according to different characters and / or different emotions.

[0023] The text to be processed obtained by executing S101 in this embodiment may be a single sentence, that is, the text to be processed includes only one sentence; the text to be processed obtained by executing S101 in this embodiment may also include multiple sentences.

[0024] If the text to be processed includes multiple sentences, this embodiment will process different sentences in the text to be processed separately and generate audio corresponding to each sentence; finally, this embodiment can use the generated multiple audios as target audio, or can use the merged result of the generated multiple audios as the target audio.

[0025] This embodiment does not limit the text type corresponding to the text to be processed. For example, the text to be processed may be one or more sentences in a novel text, or one or more sentences in a story text, etc.

[0026] Taking a novel text as an example, when executing S101 to obtain the text to be processed, this embodiment can first obtain the novel text, then segment the novel text at the sentence level (for example, ending with a period, exclamation mark, question mark, quotation mark, etc.), and finally obtain one or more sentences obtained by segmentation as the text to be processed; this embodiment generates the target audio corresponding to the novel text based on all the sentences in the novel text.

[0027] Since the large language model has a limit on the number of words in the text input each time, when the text to be processed includes multiple sentences, the number of words in the text to be processed obtained by executing S101 in this embodiment cannot exceed the preset word count threshold, for example, the number of words in the text to be processed is less than or equal to 2000.

[0028] After executing S101 to obtain the text to be processed, this embodiment executes S102 to parse the text to be processed using a large language model to obtain character information and sentiment information corresponding to the text to be processed.

[0029] If the text to be processed includes multiple sentences, this embodiment will use the large language model to parse each sentence in the text to be processed separately when executing S102, so as to obtain the role information and sentiment information corresponding to each sentence.

[0030] In this embodiment, the Large Language Model (LLM) is a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of natural language text; the LLM can handle a variety of natural language tasks, such as text classification, question-answering, and dialogue, and is an important path to artificial intelligence.

[0031] When executing S102 and using the large language model to parse the text to be processed, this embodiment can use the text to be processed as the input of the large language model, and obtain the character information and emotion information corresponding to the text to be processed based on the output result of the large language model.

[0032] Among them, the character information obtained by executing S102 in this embodiment corresponds to a certain character, such as character A, character B, character C or narration character in the novel; the emotional information corresponds to the emotional category of the character, such as neutral, happy, sad, angry, fearful, surprise, etc.

[0033] It is understandable that the role information corresponding to a sentence can be one or more (usually two); for example, the role information corresponding to a sentence can be role A, indicating that the sentence only includes the text corresponding to role A, or it can be role A and narration role, indicating that the sentence includes both the text corresponding to role A and the narration text.

[0034] Therefore, in this embodiment, when executing S102 , if the same sentence includes multiple role information, the large language model will output emotion information corresponding to different role information for each role information.

[0035] Furthermore, when executing S102 and using the large language model to parse the text to be processed, this embodiment can further obtain role labeling information based on the output result of the large language model.

[0036] In this embodiment, the role labeling information is used to reflect the age, gender, etc. of the role. For example, the role labeling information can be man, woman, boy, girl, old man, young man, etc.

[0037] For example, if the text to be processed is (character A said: "It's heavy, big sister can't hold it anymore."), then the character information obtained by executing S102 in this embodiment is character A, the emotion information is happy, and the character labeling information is woman.

[0038] If the text to be processed is (the little guy is smiling with his eyes curved in the arms of character A, and nodding his head vigorously, "Yeah, character B has grown taller again, it's this tall, so tall."), then this embodiment executes S102. For the part of the text "The little guy is smiling with his eyes curved in the arms of character A, and nodding his head vigorously," the character information obtained is the narration character, and the emotional information is neutral; for the part of the text "Yeah, character B has grown taller again, it's this tall, so tall.", the character information obtained is character B, the emotional information is happy, and the character labeling information is boy.

[0039] In addition, in this embodiment, when executing S102 to parse the text to be processed using the large language model, a preset prompt text (prompt) may be input into the large language model together with the text to be processed.

[0040] The preset prompt text in this embodiment may be "parse the input text and output the character information and emotion information corresponding to the text."

[0041] In order to further improve the accuracy of the large language model in parsing the text to be processed, the preset prompt text in this embodiment can also include more detailed information, such as role tasks, tool capability requirements and limitations, examples, etc.

[0042] Among them, the character tasks in the preset prompt text may include the following: As the character annotation function of the novel reader, your task is to receive the text input by the user and automatically annotate each sentence by which character reads it, as well as the emotion expressed by the sentence; you need to analyze the dialogue and narration in the text, identify the lines of different characters, and judge their emotions.

[0043] The tool capabilities in the preset prompt text may include the following: (1) Text analysis: You need to have strong text analysis capabilities and be able to identify role words and emotional words in sentences in order to accurately judge the character's lines and emotions; (2) Character identification: By identifying the dialogue and narration in the text, you need to be able to distinguish the lines of different characters and generate corresponding annotation information for each character; corresponding to each character, you need to provide role annotation information such as man, boy, woman, girl, etc., for example, role A+woman, role C+girl; (3) Emotional judgment: Based on the content and context of the sentence, you need to be able to judge the emotion expressed by the sentence, such as joy, sadness, anger, etc.

[0044] The requirements and restrictions for the preset text enhancement can include the following: (1) Accuracy: Your annotation results need to be highly accurate and can truly reflect the characters and emotions in the text; (2) Originality: Do not modify the input text, and the narration cannot reduce sentences; (3) Mergability: Adjacent sentences with the same characters and emotions will be merged.

[0045] Examples of preset prompt texts may include the following: Input: "My head hurts..." As soon as character C moves, he feels a sharp pain in his head, as if it is splitting apart; Output: Character C+Woman+Neutral: "My head hurts...", Narration+Woman+Neutral: As soon as character C moves, he feels a sharp pain in his head, as if it is splitting apart.

[0046] In this embodiment, after executing S102 to obtain the character information and emotion information corresponding to the text to be processed, executing S103 to obtain the target reference text and target reference audio according to the character information and emotion information.

[0047] When executing S103 in this embodiment to obtain the target reference text and target reference audio based on the character information and emotional information, the implementation method that can be adopted is: selecting a data set corresponding to the character information from multiple data sets as the target data set, and the target data set includes reference texts and reference audios corresponding to different emotions of the character; selecting reference text and reference audio corresponding to the emotional information from the target data set as the target reference text and target reference audio.

[0048] In this embodiment, different data sets correspond to different characters, and each data set is pre-set with reference texts and reference audios corresponding to different emotions of the corresponding characters. For example, in the data set corresponding to character A, reference texts and reference audios corresponding to the neutral emotion of character A, reference texts and reference audios corresponding to the happy emotion of character A, etc. are included; further, the timbre of the reference audio in the data set corresponding to character A is consistent with the timbre of character A.

[0049] In this embodiment, reference texts and reference audios corresponding to different emotions in a data set appear in pairs, and different reference audios correspond to different emotions; the reference audio is the audio corresponding to the reference text, that is, speech recognition is performed on the reference audio, and the recognition result is consistent with the reference text.

[0050] That is to say, this embodiment pre-sets different data sets to obtain the corresponding target reference text and target reference audio in real time based on the character information and emotional information corresponding to the current text to be processed. This allows this embodiment to generate audio corresponding to different characters (or different timbres) and emotions by switching reference texts and reference audios, which can simplify the audio generation steps and improve the audio generation efficiency.

[0051] When executing S103, this embodiment may further include the following: in response to determining that there is no dataset corresponding to the role information, obtaining role labeling information output by the large language model, for example, when the role information output by the large language model is an unknown role, determining a target dataset that does not have corresponding role information; and selecting a dataset corresponding to the role labeling information from multiple datasets as the target dataset.

[0052] For example, if the role labeling information output by the large language model is "woman", this embodiment will use the data set corresponding to "woman" as the target data set; that is, in addition to pre-setting data sets corresponding to different roles, this embodiment will also pre-set data sets corresponding to different role labeling information, such as a data set corresponding to "woman", a data set corresponding to "man", and so on.

[0053] That is to say, this embodiment can also select the target data set based on the role labeling information, so that even if the target data set cannot be determined based on the role information, the target data set selection can still be completed, thereby ensuring that the target reference text and target reference audio can be obtained to generate the target audio.

[0054] It can be understood that if this embodiment fails to select the reference text and reference audio corresponding to the emotional information when executing S103, the reference text and reference audio corresponding to the preset emotional information can be obtained as the target reference text and target reference audio. The preset emotional information can be a neutral emotion, etc.

[0055] In this embodiment, after executing S103 to obtain the target reference text and the target reference audio, executing S104 to generate the target audio corresponding to the text to be processed according to the text to be processed, the target reference text and the target reference audio.

[0056] When executing S104 in this embodiment, the text to be processed, the obtained target reference text and the target reference audio can be input into a pre-trained audio generation model, and the output result of the audio generation model can be used as the target audio corresponding to the text to be processed.

[0057] The audio generation model in this embodiment is pre-trained, and it can output the target audio corresponding to the text to be processed based on the input text to be processed, reference text and reference audio. The timbre, emotion and other information of the target audio are consistent with those of the reference audio.

[0058] That is to say, when generating the target audio corresponding to the text to be processed, this embodiment uses the target reference text corresponding to the target reference audio in addition to the target reference audio, which can further improve the similarity between the target audio and the target reference audio and obtain higher quality target audio.

[0059] In this embodiment, after executing S104 to generate the target audio, the generated target audio may be played, thereby achieving the purpose of real-time reading of the text to be processed.

[0060] Figure 2 Schematic diagram of the second embodiment of the present disclosure. Figure 2 As shown, in this embodiment, when executing S104 "generating a target audio corresponding to the text to be processed according to the text to be processed, the target reference text, and the target reference audio", the following contents may be included:

[0061] S201, obtaining a fusion feature vector of at least one phoneme based on a phoneme feature vector of at least one phoneme in a text and a semantic feature vector of a character to which at least one phoneme belongs, wherein the text includes the text to be processed and the target reference text;

[0062] S202: Encode the target reference audio to obtain at least one reference audio feature vector;

[0063] S203. Obtain at least one predicted audio feature vector based on the fused feature vector of the at least one phoneme and the at least one reference audio feature vector;

[0064] S204: Decode the at least one predicted audio feature vector to obtain target audio corresponding to the text to be processed.

[0065] That is to say, in addition to the reference audio feature vectors obtained using the target reference audio, this embodiment also obtains predicted audio feature vectors based on the fusion feature vectors of each phoneme in the text to be processed and the target reference text, and then obtains the target audio based on the predicted audio vectors. This embodiment fuses phoneme feature vectors and semantic feature vectors, can make full use of the semantic information of the text, and there is a corresponding relationship between the target reference audio and the target reference text, so that the similarity of timbre and emotion between the target audio and the target reference audio can be improved based on semantic information, and a more accurate (for example, more accurate timbre and emotion) and more realistic target audio can be obtained.

[0066] When this embodiment executes S201 to obtain phoneme feature vectors, it can first convert the text (the text to be processed and the target reference text) into a phoneme sequence, and then perform embedding processing on at least one phoneme in the phoneme sequence to obtain the phoneme feature vectors of at least one phoneme.

[0067] For example, if the text is "一共", the corresponding phoneme sequence is "y i2 g ong4", then the phonemes corresponding to the character "一" in the text are "y" and "i2", and the phonemes corresponding to the character "共" are "g" and "ong4" (the numbers represent tones).

[0068] When this embodiment executes S201 to perform embedding processing on at least one phoneme in the phoneme sequence, it can first convert at least one phoneme into at least one phoneme identifier, and then use a preset phoneme vocabulary to perform embedding processing on at least one phoneme identifier to obtain the phoneme feature vectors of at least one phoneme; where different phonemes correspond to different phoneme identifiers.

[0069] For example, for the character "一", if the phoneme identifier corresponding to the phoneme "y" is "1" and the phoneme identifier corresponding to the phoneme "i2" is "2", this embodiment can obtain the phoneme identifier sequence [1, 2]; then perform embedding processing on the phoneme identifier sequence to obtain the phoneme feature vector sequence [1_v, 2_v]; where 1_v is the phoneme feature vector corresponding to the phoneme "y", and 2_v is the phoneme feature vector corresponding to the phoneme "i2".

[0070] That is to say, this embodiment converts phonemes into phoneme identifiers and then performs embedding processing on the phoneme identifiers to obtain phoneme feature vectors, which can improve the accuracy of the phoneme feature vectors.

[0071] When obtaining the semantic feature vector of a character in this embodiment by executing S201, the semantic representation of the character can be obtained first. For example, the text is input into a bert model, and the semantic representation of at least one character in the text is obtained according to the output result of the bert model. Then, the semantic representation of at least one character is embedded to obtain the semantic feature vector of at least one character.

[0072] For example, if the text is "一共", this text includes two characters, "一" and "共". The semantic representation corresponding to the character "一" can be "3", and the semantic representation corresponding to the character "共" can be "4". Using a preset semantic vocabulary to embed the semantic representation, the semantic feature vector 3_v corresponding to the character "一" and the semantic feature vector 4_v corresponding to the character "共" are obtained.

[0073] That is to say, in this embodiment, by converting a character into a semantic representation and then embedding the semantic representation to obtain a semantic feature vector, the accuracy of the semantic feature vector can be improved.

[0074] When obtaining the fusion feature vector of a phoneme according to the phoneme feature vector of the phoneme and the semantic feature vector of the character to which the phoneme belongs by executing S201 in this embodiment, the fusion feature vector can be obtained by adding or concatenating the phoneme feature vector and the semantic feature vector.

[0075] For example, for the phoneme "y" in the character "一", if the phoneme feature vector of this phoneme is 1_v and the semantic feature vector of the character "一" to which this phoneme belongs is 3_v, then in this embodiment, "1_v" and "3_v" are fused, and the fusion result is used as the fusion feature vector of the corresponding phoneme "y".

[0076] That is to say, in this embodiment, the semantic feature vector of a character is fused with the phoneme feature vectors of at least one phoneme corresponding to this character, so that when generating the target audio, the semantic information and phoneme information of the text (including the target reference text and the text to be processed) can be more fully utilized.

[0077] When encoding the target reference audio to obtain at least one reference audio feature vector by executing S202 in this embodiment, the target reference audio can be encoded first, and at least one reference audio representation is obtained according to the encoding result. The audio representation in this embodiment can be a digital character, and different audio representations are related to timbre, emotion, etc. Then, at least one reference audio representation is embedded to obtain at least one reference audio feature vector.

[0078] For example, if at least one reference audio representation obtained in this embodiment is [10, 11], using a preset audio vocabulary to embed the reference audio representation, at least one reference audio vector [10_v, 11_v] is obtained.

[0079] In this embodiment, when executing S203 to obtain at least one predicted audio feature vector based on the fused feature vector of at least one phoneme and at least one reference audio feature vector, the fused feature vector of at least one phoneme can be first fused with the at least one reference audio feature vector to obtain at least one feature vector to be processed, and then the at least one feature vector to be processed can be encoded to obtain at least one predicted audio feature vector.

[0080] In addition, when executing S203, this embodiment can also input the fusion feature vector of at least one phoneme and at least one reference audio feature vector into a pre-trained neural network audio coding model, and use the output result of the neural network audio coding model as at least one predicted audio feature vector.

[0081] In this embodiment, when performing S204 to decode at least one predicted audio feature vector, the at least one predicted audio feature vector may be decoded according to a decoding method corresponding to an encoding method of the target reference audio, so as to correspond to the target audio of the text to be processed.

[0082] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure. Figure 3 FIG shows a flowchart of the embodiment of the present invention when generating a target audio corresponding to the text to be processed according to the text to be processed, the target reference text and the target reference audio; Figure 3 As shown in , this embodiment first obtains a fused feature vector of at least one phoneme corresponding to the target reference text, a fused feature vector of at least one phoneme corresponding to the text to be processed, and at least one reference audio feature vector corresponding to the target reference audio; then, through a neural network audio coding model, at least one predicted audio feature vector is obtained based on the fused feature vector of at least one phoneme and at least one reference audio feature vector; finally, the target audio corresponding to the text to be processed is obtained based on the at least one predicted audio feature vector.

[0083] Figure 4 Schematic diagram of the fourth embodiment of the present disclosure. Figure 4 As shown, the audio generation device 400 based on the large language model of this embodiment includes:

[0084] An acquisition unit 401 is used to acquire a text to be processed;

[0085] The parsing unit 402 is configured to parse the text to be processed using a large language model to obtain character information and sentiment information corresponding to the text to be processed;

[0086] The processing unit 403 is configured to obtain a target reference text and a target reference audio according to the character information and the emotion information;

[0087] The generating unit 404 is configured to generate a target audio corresponding to the text to be processed according to the text to be processed, the target reference text and the target reference audio.

[0088] The text to be processed acquired by the acquisition unit 401 may be a single sentence, that is, the text to be processed includes only one sentence; the text to be processed acquired by executing S101 in this embodiment may also include multiple sentences.

[0089] If the text to be processed includes multiple sentences, this embodiment will process different sentences in the text to be processed separately and generate audio corresponding to each sentence; finally, this embodiment can use the generated multiple audios as target audio, or can use the merged result of the generated multiple audios as the target audio.

[0090] This embodiment does not limit the text type corresponding to the text to be processed. For example, the text to be processed may be one or more sentences in a novel text, or one or more sentences in a story text, etc.

[0091] Since the large language model has a limit on the number of words in the text input each time, when the text to be processed includes multiple sentences, the number of words in the text to be processed obtained by the acquisition unit 401 cannot exceed the preset word count threshold, for example, the number of words in the text to be processed is less than or equal to 2000.

[0092] In this embodiment, after the acquisition unit 401 acquires the text to be processed, the parsing unit 402 uses the large language model to parse the text to be processed to obtain the character information and emotion information corresponding to the text to be processed.

[0093] If the text to be processed includes multiple sentences, the parsing unit 402 will use the large language model to parse each sentence in the text to be processed separately, so as to obtain the role information and sentiment information corresponding to each sentence.

[0094] In this embodiment, the Large Language Model (LLM) is a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of natural language text; the LLM can handle a variety of natural language tasks, such as text classification, question-answering, and dialogue, and is an important path to artificial intelligence.

[0095] When parsing the text to be processed using the large language model, the parsing unit 402 can use the text to be processed as the input of the large language model, and obtain the role information and emotion information corresponding to the text to be processed based on the output result of the large language model.

[0096] Among them, the character information obtained by the parsing unit 402 corresponds to a certain character, such as character A, character B, character C or narration character in the novel; the emotional information corresponds to the emotional category of the character, such as neutral, happy, sad, angry, fearful, surprise, etc.

[0097] Furthermore, when parsing the text to be processed using the large language model, the parsing unit 402 may further obtain role labeling information based on the output result of the large language model.

[0098] In this embodiment, the role labeling information is used to reflect the age, gender, etc. of the role. For example, the role labeling information can be man, woman, boy, girl, old man, young man, etc.

[0099] In addition, when the parsing unit 402 uses the large language model to parse the text to be processed, it can also input a preset prompt text (prompt) into the large language model together with the text to be processed.

[0100] The preset prompt text in this embodiment may be "parse the input text and output the character information and emotion information corresponding to the text."

[0101] In order to further improve the accuracy of the large language model in parsing the text to be processed, the preset prompt text in this embodiment can also include more detailed information, such as role tasks, tool capability requirements and limitations, examples, etc.

[0102] In this embodiment, after the parsing unit 402 obtains the character information and emotion information corresponding to the text to be processed, the processing unit 403 obtains the target reference text and target reference audio according to the character information and emotion information.

[0103] When the processing unit 403 obtains the target reference text and target reference audio based on the character information and emotional information, the implementation method that can be adopted is: selecting a data set corresponding to the character information from multiple data sets as the target data set; selecting a reference text and reference audio corresponding to the emotional information from the target data set as the target reference text and target reference audio.

[0104] In this embodiment, different data sets correspond to different characters, and reference texts and reference audios of different emotions corresponding to the corresponding characters are pre-set in each data set.

[0105] In this embodiment, reference texts and reference audios corresponding to different emotions in a data set appear in pairs, and different reference audios correspond to different emotions; the reference audio is the audio corresponding to the reference text, that is, speech recognition is performed on the reference audio, and the recognition result is consistent with the reference text.

[0106] That is to say, in this embodiment, by pre-setting different data sets, the processing unit 403 obtains the corresponding target reference text and target reference audio in real time according to the obtained character information and emotional information corresponding to the current text to be processed, so that this embodiment generates audio corresponding to different characters (or different timbres) and emotions by switching reference texts and reference audios, which can simplify the audio generation steps and improve the audio generation efficiency.

[0107] The processing unit 403 may further include the following: in response to determining that there is no dataset corresponding to the role information, obtaining role labeling information output by the large language model; and selecting a dataset corresponding to the role labeling information from multiple datasets as a target dataset.

[0108] That is to say, the processing unit 403 can also select the target data set based on the role labeling information, so that when the target data set cannot be determined based on the role information, the target data set selection can still be completed, thereby ensuring that the target reference text and target reference audio can be obtained to generate the target audio.

[0109] It can be understood that if the processing unit 403 is unable to select the reference text and reference audio corresponding to the emotional information, it can obtain the reference text and reference audio corresponding to the preset emotional information as the target reference text and target reference audio. The preset emotional information can be neutral emotion, etc.

[0110] In this embodiment, after the processing unit 403 obtains the target reference text and the target reference audio, the generating unit 404 generates the target audio corresponding to the text to be processed based on the text to be processed, the target reference text and the target reference audio.

[0111] The generation unit 404 may input the text to be processed, the obtained target reference text and the target reference audio into a pre-trained audio generation model, and then use the output result of the audio generation model as the target audio corresponding to the text to be processed.

[0112] The audio generation model in this embodiment is pre-trained, and it can output the target audio corresponding to the text to be processed based on the input text to be processed, reference text and reference audio. The timbre, emotion and other information of the target audio are consistent with those of the reference audio.

[0113] That is to say, when generating the target audio corresponding to the text to be processed, the generation unit 404 uses the target reference text corresponding to the target reference audio in addition to the target reference audio, which can further improve the similarity between the target audio and the target reference audio and obtain higher quality target audio.

[0114] After generating the target audio, the generating unit 404 may also play the generated target audio, thereby achieving the purpose of real-time reading of the text to be processed.

[0115] In addition, when the generation unit 404 generates the target audio corresponding to the text to be processed based on the text to be processed, the target reference text and the target reference audio, it can also include the following contents: obtaining a fusion feature vector of at least one phoneme based on the phoneme feature vector of at least one phoneme in the text and the semantic feature vector of the character to which at least one phoneme belongs, the text includes the text to be processed and the target reference text; encoding the target reference audio to obtain at least one reference audio feature vector; obtaining at least one predicted audio feature vector based on the fusion feature vector of at least one phoneme and at least one reference audio feature vector; decoding at least one predicted audio feature vector to obtain the target audio corresponding to the text to be processed.

[0116] That is to say, in addition to using the reference audio feature vector obtained by the target reference audio, the generation unit 404 also obtains a predicted audio feature vector based on the fused feature vector of each phoneme in the text to be processed and the target reference text, and then obtains the target audio based on the predicted audio vector. This embodiment fuses the phoneme feature vector with the semantic feature vector, which can fully utilize the semantic information of the text, and there is a corresponding relationship between the target reference audio and the target reference text, so that the similarity of timbre and emotion between the target audio and the target reference audio can be improved based on the semantic information, so as to obtain a more accurate (for example, more accurate timbre and emotion) and more realistic target audio.

[0117] When obtaining the phoneme feature vector, the generation unit 404 may first convert the text (the text to be processed and the target reference text) into a phoneme sequence, and then perform embedding processing on at least one phoneme in the phoneme sequence to obtain the phoneme feature vector of the at least one phoneme.

[0118] When the generation unit 404 performs embedding processing on at least one phoneme in a phoneme sequence, it can first convert the at least one phoneme into at least one phoneme identifier, and then use a preset phoneme vocabulary to perform embedding processing on the at least one phoneme identifier to obtain a phoneme feature vector of the at least one phoneme; wherein different phonemes correspond to different phoneme identifiers.

[0119] That is, the generating unit 404 converts phonemes into phoneme identifiers and then performs embedding processing on the phoneme identifiers to obtain phoneme feature vectors, thereby improving the accuracy of the phoneme feature vectors.

[0120] When obtaining the semantic feature vector of a character, the generating unit 404 may first obtain the semantic representation of the character, and then perform embedding processing on the semantic representation of at least one character, thereby obtaining the semantic feature vector of at least one character.

[0121] That is, the generating unit 404 converts characters into semantic representations and then performs embedding processing on the semantic representations to obtain semantic feature vectors, thereby improving the accuracy of the semantic feature vectors.

[0122] When obtaining the fused feature vector of the phoneme according to the phoneme feature vector of the phoneme and the semantic feature vector of the character to which the phoneme belongs, the generating unit 404 may obtain the fused feature vector by adding or concatenating the phoneme feature vector and the semantic feature vector.

[0123] That is, the generation unit 404 fuses the semantic feature vector of the character with the phoneme feature vector of at least one phoneme corresponding to the character, so that when generating the target audio, the semantic information and phoneme information of the text (including the target reference text and the text to be processed) can be more fully utilized.

[0124] When the generation unit 404 encodes the target reference audio to obtain at least one reference audio feature vector, it can first encode the target reference audio and obtain at least one reference audio representation based on the encoding result. The audio representation in this embodiment can be a digital character, and different audio representations are related to timbre, emotion, etc.; then, the at least one reference audio representation is embedded to obtain at least one reference audio feature vector.

[0125] When the generation unit 404 obtains at least one predicted audio feature vector based on the fused feature vector of at least one phoneme and at least one reference audio feature vector, it can first fuse the fused feature vector of at least one phoneme with the at least one reference audio feature vector to obtain at least one feature vector to be processed, and then encode the at least one feature vector to be processed to obtain at least one predicted audio feature vector.

[0126] In addition, the generation unit 404 can also input the fusion feature vector of at least one phoneme and at least one reference audio feature vector into a pre-trained neural network audio coding model, and use the output result of the neural network audio coding model as at least one predicted audio feature vector.

[0127] When decoding the at least one predicted audio feature vector, the generating unit 404 may decode the at least one predicted audio feature vector according to a decoding method corresponding to an encoding method of the target reference audio, so as to correspond to the target audio of the text to be processed.

[0128] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0129] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0130] like Figure 5 , is a block diagram of an electronic device for an audio generation method based on a large language model according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0131] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the device 500 can also be stored in the RAM 503. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0132] Various components in device 500 are connected to I / O interface 505, including: an input unit 506, such as a keyboard, mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a magnetic disk, optical disk, etc.; and a communication unit 509, such as a network card, modem, wireless communication transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0133] The computing unit 501 can be various general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the audio generation method based on the large language model. For example, in some embodiments, the audio generation method based on the large language model can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 508.

[0134] In some embodiments, part or all of the computer program can be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the audio generation method based on the large language model described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform the audio generation method based on the large language model by any other appropriate means (e.g., by means of firmware).

[0135] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0136] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable large language model-based audio generation device, such that when the program code is executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0137] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0138] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0139] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0140] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship is established by computer programs running on the respective computers and establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service system that addresses the management difficulties and poor business scalability of traditional physical hosts and VPS services ("Virtual Private Servers," or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0141] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0142] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. An audio generation method based on a large language model, comprising: Get the text to be processed; Parsing the text to be processed using a large language model to obtain character information and sentiment information corresponding to the text to be processed; Obtaining a target reference text and a target reference audio according to the character information and the emotion information; generating a target audio corresponding to the text to be processed according to the text to be processed, the target reference text and the target reference audio; The step of obtaining a target reference text and a target reference audio according to the character information and the emotion information includes: Selecting a data set corresponding to the role information from multiple data sets as the target data set; Selecting reference text and reference audio corresponding to the emotional information from the target data set as the target reference text and the target reference audio; Generating the target audio corresponding to the text to be processed according to the text to be processed, the target reference text, and the target reference audio includes: Obtaining a fusion feature vector of at least one phoneme based on a phoneme feature vector of at least one phoneme in a text and a semantic feature vector of a character to which at least one phoneme belongs, wherein the text includes the text to be processed and the target reference text; Encoding the target reference audio to obtain at least one reference audio feature vector; Obtaining at least one predicted audio feature vector based on the fused feature vector of the at least one phoneme and the at least one reference audio feature vector; The at least one predicted audio feature vector is decoded to obtain target audio corresponding to the text to be processed.

2. The method according to claim 1, further comprising: In response to determining that there is no data set corresponding to the role information, obtaining role labeling information output by the large language model; A dataset corresponding to the role labeling information is selected from multiple datasets as the target dataset.

3. The method according to claim 1, wherein The encoding of the target reference audio to obtain at least one reference audio feature vector includes: encoding the target reference audio to obtain at least one reference audio representation; Embedding processing is performed on the at least one reference audio representation to obtain the at least one reference audio feature vector.

4. The method according to claim 1, wherein The obtaining, according to the fused feature vector of the at least one phoneme and the at least one reference audio feature vector, at least one predicted audio feature vector comprises: Fusing the fused feature vector of the at least one phoneme with the at least one reference audio feature vector to obtain at least one feature vector to be processed; The at least one feature vector to be processed is encoded to obtain the at least one predicted audio feature vector.

5. The method according to claim 1, wherein The decoding of the at least one predicted audio feature vector comprises: The at least one predicted audio feature vector is decoded according to a decoding method corresponding to an encoding method of the target reference audio.

6. An audio generation device based on a large language model, comprising: An acquisition unit, used to acquire the text to be processed; A parsing unit, configured to parse the text to be processed using a large language model to obtain character information and sentiment information corresponding to the text to be processed; a processing unit, configured to obtain a target reference text and a target reference audio according to the character information and the emotion information; a generating unit, configured to generate a target audio corresponding to the text to be processed based on the text to be processed, the target reference text, and the target reference audio; When the processing unit obtains the target reference text and the target reference audio according to the character information and the emotion information, the processing unit specifically performs the following steps: Selecting a data set corresponding to the role information from multiple data sets as the target data set; Selecting reference text and reference audio corresponding to the emotional information from the target data set as the target reference text and the target reference audio; When the generating unit generates the target audio corresponding to the text to be processed according to the text to be processed, the target reference text, and the target reference audio, the generating unit specifically performs: Obtaining a fusion feature vector of at least one phoneme based on a phoneme feature vector of at least one phoneme in a text and a semantic feature vector of a character to which at least one phoneme belongs, wherein the text includes the text to be processed and the target reference text; Encoding the target reference audio to obtain at least one reference audio feature vector; Obtaining at least one predicted audio feature vector based on the fused feature vector of the at least one phoneme and the at least one reference audio feature vector; The at least one predicted audio feature vector is decoded to obtain target audio corresponding to the text to be processed.

7. The apparatus according to claim 6, wherein the processing unit is further configured to execute: In response to determining that there is no data set corresponding to the role information, obtaining role labeling information output by the large language model; A dataset corresponding to the role labeling information is selected from multiple datasets as the target dataset.

8. The device according to claim 6, wherein When encoding the target reference audio to obtain at least one reference audio feature vector, the generating unit specifically performs: encoding the target reference audio to obtain at least one reference audio representation; Embedding processing is performed on the at least one reference audio representation to obtain the at least one reference audio feature vector.

9. The device according to claim 6, wherein When the generating unit obtains at least one predicted audio feature vector according to the fused feature vector of the at least one phoneme and the at least one reference audio feature vector, the generating unit specifically performs: Fusing the fused feature vector of the at least one phoneme with the at least one reference audio feature vector to obtain at least one feature vector to be processed; The at least one feature vector to be processed is encoded to obtain the at least one predicted audio feature vector.

10. The device according to claim 6, wherein When decoding the at least one predicted audio feature vector, the generating unit specifically performs: The at least one predicted audio feature vector is decoded according to a decoding method corresponding to an encoding method of the target reference audio.

11. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

13. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images