Method for training audio book large model and related product

Abstract

The invention discloses a method for training an audio book large model and a related product. The method comprises the steps that a first audio is acquired, the first audio is a mixed audio of dry voice and music elements of a single person, and the music elements comprise sound effects and / or background music; preprocessing the first audio to obtain text information, first description information, a first quantization code, second description information and a second quantization code of the first audio; combining the text information, the first description information, the first quantization code, the second description information and the second quantization code according to the thinking chain to obtain a model cue word; and inputting the model cue word into the audio reading material large model so as to train the audio reading material large model. By using the trained audio book large model, the audio book audio with the sound effect and the background music can be obtained, the content is more vivid and interesting, and the quality of the audio book and the auditory experience of the user are improved.

Description

Technical Field

[0001] The present disclosure generally relates to the field of speech generation technology. More specifically, the present disclosure relates to a method, electronic device, and computer-readable storage medium for training a large model of an audio book. Further, the present disclosure also relates to a method, electronic device, and computer-readable storage medium for generating an audio book. Background Art

[0002] In the traditional audiobook production process, the generation of sounds, sound effects and background music mainly relies on manual dubbing and manual soundtracking, which is not only time-consuming and labor-intensive, but also expensive to produce.

[0003] With the rapid development of artificial intelligence (AI) technology, AI-based audiobook generation solutions have emerged. These solutions use advanced text-to-speech (TTS) technology to convert written text into audible speech. However, the audiobooks generated by existing solutions often have the characteristics of machine voices and lack the emotional and stylistic expressions of human dubbing.

[0004] In view of this, there is an urgent need to provide a solution for generating audio books so as to improve the production efficiency of audio books, reduce the production cost, and generate audio book audio with emotional human voice, sound effects and background music. Summary of the invention

[0005] In order to at least solve one or more of the technical problems mentioned above, the present disclosure proposes a solution for generating an audiobook in the following aspects.

[0006] In a first aspect, the present disclosure provides a method for training an audio book big model, comprising: obtaining a first audio, the first audio being a mixed audio of a single person's dry vocal sound and music elements, the music elements comprising sound effects and / or background music; preprocessing the first audio to obtain text information, first description information, first quantization code, second description information and second quantization code of the first audio, wherein the first description information is a description of the emotion and style expressed by the dry vocal sound, the first quantization code is a quantification of the semantic features of the dry vocal sound, the second description information is a description of the scene and atmosphere created by the music elements, and the second quantization code is a quantification of the semantic features of the music elements; combining the text information, the first description information, the first quantization code, the second description information and the second quantization code according to a thinking chain to obtain a model prompt word; inputting the model prompt word into the audio book big model to realize training of the audio book big model.

[0007] In some embodiments, preprocessing the first audio to obtain text information, first description information, second description information and second quantization code of the first audio includes: performing speech recognition on the first audio to obtain the text information; performing sound separation on the first audio to obtain the dry vocal sound and the music element in the first audio; using natural language to describe the emotion and style expressed by the dry vocal sound to obtain the first description information, and using natural language to describe the scene and atmosphere created by the music element to obtain the second description information; respectively inputting the dry vocal sound and the music element into an audio quantization model to obtain the first quantization code and the second quantization code.

[0008] In some embodiments, the audio quantization model includes an audio processing model and a quantization module, the audio quantization model is a model obtained by training using a generative adversarial network framework, the generative adversarial network framework includes the audio quantization model, an audio decoder and a discriminator; and obtaining the audio quantization model using the generative adversarial network framework includes: obtaining a second audio, the second audio is a mixed audio of a single person's dry vocal sound and musical elements; performing sound separation on the second audio to obtain the dry vocal sound and the musical elements in the second audio; based on the dry vocal sound and the musical elements in the second audio, using the generative adversarial network framework to train the audio quantization model to obtain a trained audio quantization model.

[0009] In some embodiments, based on the dry vocal sounds and the music elements in the second audio, the audio quantization model is trained using the generative adversarial network framework to obtain a trained audio quantization model, including: training the audio processing model, the quantization module and the audio decoder based on the dry vocal sounds and the music elements in the second audio to obtain a trained audio processing model, a trained quantization module and a trained audio decoder; training the discriminator based on the dry vocal sounds and the music elements in the second audio, the trained audio processing model, the trained quantization module and the trained audio decoder until the discriminator training is completed to obtain the trained audio quantization model.

[0010] In some embodiments, obtaining the first audio includes: obtaining a third audio, wherein the third audio is a mixed audio of dry vocal sounds of multiple people and musical elements; and segmenting the third audio according to the rule of each person per sentence to obtain multiple first audios.

[0011] In a second aspect, the present disclosure provides a method for generating an audio book, comprising: obtaining a text to be generated; inputting the text to be generated into an audio book large model trained by the method described in the first aspect and its multiple embodiments to perform a quantization coding generation operation to output a human voice quantization code and a music element quantization code, wherein the music element includes sound effects and / or background music; based on the human voice quantization code and the music element quantization code, using an audio decoder to perform an audio book generation operation to obtain an audio book audio containing dry human voice and music elements.

[0012] In a third aspect, the present disclosure provides a method for generating an audio book, comprising: obtaining a text to be generated and an emotional style description information; inputting the text to be generated and the emotional style description information into a large audio book model trained by the method described in the first aspect and its multiple embodiments to perform a quantization coding generation operation to output a human voice quantization code and a music element quantization code, wherein the music element includes sound effects and / or background music; based on the human voice quantization code and the music element quantization code, using an audio decoder to perform an audio book generation operation to obtain an audio book audio containing dry human voice and music elements.

[0013] In a fourth aspect, the present disclosure provides a method for generating an audio book, comprising: obtaining a text to be generated, emotional style description information, and music element description information, wherein the music element includes sound effects and / or background music; inputting the text to be generated and the emotional style description information into a first audio book large model trained by the method described in the first aspect and its multiple embodiments to perform a quantization coding generation operation to output a human voice quantization code; inputting the text to be generated, the emotional style description information, and the human voice quantization code into a second audio book large model trained by the method described in the first aspect and its multiple embodiments to perform a quantization coding generation operation to output a music element quantization code; based on the human voice quantization code and the music element quantization code, using an audio decoder to perform an audio book generation operation to obtain an audio book audio containing dry human voice and music elements.

[0014] In a fifth aspect, the present disclosure provides an electronic device comprising: a processor; and a memory storing program instructions for training a large audio book model or for generating an audio book, wherein when the program instructions are executed by the processor, the method described in the aforementioned first aspect, second aspect, third aspect or fourth aspect and its multiple embodiments are implemented.

[0015] In a sixth aspect, the present disclosure provides a computer-readable storage medium having stored thereon program instructions for training a large audio book model or for generating an audio book, which, when executed by a processor, implements the method described in the aforementioned first aspect, second aspect, third aspect or fourth aspect and its multiple embodiments.

[0016] Through the scheme for generating audio books provided above, the model prompt words composed according to the thought chain give the audio book large model the ability to think step by step, so that it can decompose complex problems into multi-step sub-problems, significantly increasing the reasoning ability of the large model. In addition, the model prompt words composed according to the thought chain enable the audio book large model to display the reasoning process, so that users can directly influence the problem-solving process of the large model. Therefore, when using the trained audio book large model to generate audio books, multiple modes of reasoning can be expanded, improving the ease of use and flexibility of the audio book large model.

[0017] Thereafter, in the application stage of the audiobook model, based on the trained audiobook model and audio decoder, it is only necessary to input the text to be generated, or the combination of the text to be generated and the emotional style description information and / or the music element description information into the trained audiobook model, to obtain the audiobook audio containing the dry human voice and the music elements, which greatly improves the production efficiency of the audiobook and reduces the manufacturing cost. At the same time, the solution disclosed herein can be used to obtain the audiobook audio with emotional human voice, sound effects and background music, and the content is more vivid and interesting, which not only improves the quality of the audiobook, but also improves the user's auditory experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0019] Figure 1 An exemplary flow chart of a method for training a large audiobook model according to an embodiment of the present disclosure is shown;

[0020] Figure 2 An exemplary flow chart of a method for preprocessing a first audio according to an embodiment of the present disclosure is shown;

[0021] Figure 3 An exemplary model prompt word of the audiobook large model according to the embodiment of the present disclosure is shown;

[0022] Figure 4A schematic diagram of a generative adversarial network framework of an embodiment of the present disclosure is shown;

[0023] Figure 5 An exemplary flow chart showing a method for generating an audio book according to some embodiments of the present disclosure;

[0024] Figure 6 An exemplary flow chart showing a method for generating an audio book according to some other embodiments of the present disclosure;

[0025] Figure 7 An exemplary flow chart showing a method for generating an audio book according to some other embodiments of the present disclosure is shown;

[0026] Figure 8 An exemplary structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0028] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0029] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0030] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0031] The specific implementation of the present disclosure is described in detail below with reference to the accompanying drawings.

[0032] Figure 1 An exemplary flow chart of a method 100 for training a large audiobook model according to an embodiment of the present disclosure is shown. It is understood that the method 100 may be executed by any appropriate device with data processing capabilities, such as but not limited to a terminal device, a processor, and a server.

[0033] like Figure 1 As shown, at step S101, method 100 may obtain audio for training (for distinction, may be referred to as first audio). Here, the first audio is a mixed audio of a single person's dry vocal sound and a musical element, and the musical element may include sound effects and / or background music. Then, at step S102, method 100 may pre-process the first audio to obtain text information, first description information, first quantization code, second description information, and second quantization code of the first audio.

[0034] Next, at step S103, method 100 may combine the text information, the first description information, the first quantization code, the second description information, and the second quantization code to obtain a model prompt word. Finally, at step S104, method 100 may input the model prompt word into the audio book big model to train the audio book big model.

[0035] At the aforementioned step S101, when method 100 obtains the first audio, specifically, it can first obtain audio with multiple characters and emotions and sound effects and background music (for the sake of distinction, it can be called the third audio), that is, the third audio is a mixed audio of multiple people's dry vocal sounds and musical elements, and then, the third audio can be segmented according to the rule of each person per sentence to obtain multiple first audios. Thereafter, each first audio can be used as training data for training the audiobook large model. As an example, assume that three characters appear in the third audio, and each character says three sentences. Then, the third audio can be segmented into nine first audios according to the rule of each person per sentence.

[0036] At the aforementioned step S102, the method 100 can perform preprocessing such as speech recognition, sound separation, feature extraction, feature quantification, and natural speech description on the first audio. By performing speech recognition on the first audio, the text information contained in the first audio can be obtained, and by manually using natural language to describe the emotion and style (or speech style) expressed by the human voice dry sound in the first audio, the first description information can be obtained, that is, the first description information is a description of the emotion and style expressed by the human voice dry sound. By manually using natural speech to describe the scene and atmosphere created by the music elements in the first audio, the second description information can be obtained, that is, the second description information is a description of the scene and atmosphere created by the music elements.

[0037] Here, the emotion expressed by the human voice dry sound can be any appropriate emotion, for example, it can include but not limited to happiness, sadness, anger, fear and surprise, etc. The speech style expressed by the human voice dry sound can be any appropriate speech style, for example, it can include but not limited to humor, seriousness, encouragement, criticism and command, etc. The scene created by the music element can include any appropriate scene such as natural scene, sports scene and festival scene. Further, the natural scene can include but not limited to birdsong, barking, rustling leaves, rolling waves, gurgling streams and howling winds, etc. Sports scenes can include but not limited to cheering and victory cheers, etc. Festival scenes can further include Christmas bells, Christmas carols, New Year firecrackers and New Year bells, etc. The atmosphere created by the music element can include but not limited to relaxed and happy, sad and melancholy, nervous and exciting, lively and cheerful, etc.

[0038] In addition, by performing sound separation on the first audio, a human voice dry sound and a musical element can be obtained. By performing feature extraction on the human voice dry sound and the musical element respectively, their semantic features can be obtained. Further, by performing feature quantization on their semantic features respectively, a first quantization code and a second quantization code can be obtained. In other words, the first quantization code is a quantization of the semantic features of the human voice dry sound, and the second quantization code is a quantization of the semantic features of the musical element.

[0039] In practical applications, single codebook quantization or multi-codebook quantization can be used when quantizing semantic features. Single codebook quantization uses a global codebook to quantize all features, and this codebook contains representative encodings of all possible feature vectors. Unlike single codebook quantization, multi-codebook quantization uses multiple codebooks to quantize features, and each codebook can focus on a specific part or dimension of the feature space. In the scheme disclosed in the present invention, in order to reduce the difficulty of training the subsequent large model of audiobooks, single codebook quantization is selected when quantizing language features.

[0040] In addition, in order to cover the diversity of the feature space, a single codebook may require a larger codebook to improve the quantization effect. However, the size of the single codebook will directly affect the difficulty and speed of training the subsequent large audiobook model. Therefore, it is necessary to select an appropriate size of the single codebook to balance the quantization effect with the difficulty and speed of training. It is understandable that a technician in this field can select a specific value of the size of the single codebook according to actual needs. For example, the size of the single codebook can be set to 1024bit, 2048bit, 4096bit or 8192bit, etc., and this disclosure does not make specific limitations on this.

[0041] To facilitate understanding, how to obtain the text information of the first audio, the first description information, the first quantization code, the second description information and the second quantization code will be described later. Figure 2 The method 200 for preprocessing the first audio is described in detail.

[0042] At the aforementioned step S103, method 100 can combine the text information, the first description information, the first quantization code, the second description information and the second quantization code according to the Chain-of-Thought (CoT). CoT is an improved prompt word prompt technology, which aims to improve the performance of large models on complex reasoning tasks. It is difficult for large models to directly give correct answers to complex problems. The chain of thought can help the large model to explicitly output the intermediate step-by-step reasoning steps before outputting the final answer, which helps to enhance the arithmetic and reasoning ability of the large model. Under the guidance of CoT, the model prompt words of the large audio book model designed in the present disclosure are such as Figure 3 shown.

[0043] exist Figure 3 In the process of combining the text information, the first description information, the first quantitative code, the second description information and the second quantitative code according to the thought chain, the text information can be placed in <text>and <\text>, place the first description information between<speech_caption> and <\speech_caption>, placing the first quantization code<speech_code> and <\speech_code>, place the second description information between<sound_caption> and <\sound_caption>, place the second quantization code<sound_code> and <\sound_code>. It can be seen that in each reasoning step, a set of corresponding symbols (such as <> and <\>) are used to mark the beginning and end of a reasoning step, and to prompt the audiobook model to solve the problem step by step.

[0044] At the aforementioned step S104, the audio book big model is trained by inputting the model prompt words into it, so that the audio book big model can learn the ability to generate quantization codes corresponding to the dry voice of the human voice and the quantization codes corresponding to the musical elements. In short, the trained audio book big model can perform the operation of generating quantization codes. In practical applications, those skilled in the art can select appropriate open source models for the audio book big model according to actual needs, such as Qwen2.5 series models, Llama series models or GPT-4 series models, etc., and the present disclosure does not make specific limitations on this. Preferably, Qwen2.5-72B in the Qwen2.5 series model can be selected for the audio book big model.

[0045] After obtaining the quantization codes corresponding to the dry human voice and the quantization codes corresponding to the music elements, the dry human voice can be generated using the quantization codes corresponding to the dry human voice and the feature vector corresponding to the quantization codes in the single codebook, and the music elements can be generated using the quantization codes corresponding to the music elements and the feature vector corresponding to the quantization codes in the single codebook. Furthermore, the dry human voice and the music elements can be mixed to obtain an audio book.

[0046] Combination of the above Figure 1 A method for training a large audio book model is described. The model prompt words composed according to the thought chain give the large audio book model the ability to think step by step, so that it can decompose complex problems into multi-step sub-problems, significantly increasing the reasoning ability of the large model. In addition, the model prompt words composed of the thought chain allow the large audio book model to display the reasoning process, so that users can directly influence the problem-solving process of the large model. Therefore, when using the trained large audio book model to generate audio books, multiple modes of reasoning can be expanded, which improves the ease of use and flexibility of the large audio book model.

[0047] After that, in the application stage of the audio book model, based on the trained audio book model and audio decoder, the text to be generated, the emotional style description information and the music element description information are input into the trained audio book model, and the audio book audio containing the human voice dry sound and music elements can be obtained, which greatly improves the production efficiency of audio books and reduces the manufacturing cost. At the same time, the solution disclosed in this disclosure can be used to obtain audio book audio with emotional human voice, sound effects and background music, and the content is more vivid and interesting, which not only improves the quality of audio books, but also improves the user's listening experience.

[0048] Figure 2 FIG. 2 shows an exemplary flow chart of a method 200 for preprocessing a first audio according to an embodiment of the present disclosure. Figure 2 The description is a specific implementation of the aforementioned step S102. Figure 1 The features described can analogously apply here.

[0049] like Figure 2 As shown, at step S201, method 200 may use any appropriate speech recognition model, such as WeNet, Flan-T5 or WavLM, to perform speech recognition on the first audio to obtain text information. At step S202, method 200 may use any appropriate speech separation model, such as MDX-Net, Demucs or Spleeter, to perform sound separation on the first audio to obtain the dry vocal sound and the musical elements in the first audio. Then, at step S203, method 200 may use natural language to describe the emotion and style expressed by the dry vocal sound to obtain the first description information, and use natural language to describe the scene and atmosphere created by the musical elements to obtain the second description information. Finally, at step S204, method 200 may input the dry vocal sound and the musical elements into the audio quantization model respectively to obtain the first quantization code and the second quantization code.

[0050] In the scheme disclosed herein, the audio quantization model includes an audio processing model and a quantization module. The audio processing model is used to extract semantic features from the input audio. The quantization module quantizes the semantic features extracted by the audio processing model into a series of quantization codes, and the quantization module calculates a quantization loss Loss_q, which is used to back-propagate and update parameters. In practical applications, the self-supervised pre-trained model Hubert can be selected as the audio processing model, and it can be trained using a generative adversarial network framework. Therefore, in addition to the audio quantization model, the generative adversarial network framework can also include an audio decoder and a discriminator.

[0051] Here, you can refer to Figure 4 Understand the generative adversarial network framework disclosed in this disclosure for training the audio quantization model. Figure 4 As shown, the audio decoder can restore the feature vector corresponding to the quantized code to a new audio under the timbre reference, and will generate a Mel spectrum loss Loss_mel. In the framework of the generative adversarial network, the audio decoder, the audio processor and the quantization module constitute a generator, and the discriminator can distinguish the true and false of the new audio generated by the generator, and generate adversarial damage Loss_d. Based on this, the total loss Loss for training the large audio book model under the framework of the generative adversarial network is Loss=Loss_q+Loss_mel+Loss_d. Here, the timbre reference can be a timbre list, which includes multiple sound names and timbre features corresponding to the sound names. The audio decoder restores the feature vector to a new audio under the timbre reference, which is actually based on the timbre features in the timbre list. Sound cloning is performed to imitate the sound corresponding to the timbre features to generate new audio.

[0052] Thus, the audio quantization model is obtained using the generative adversarial network framework. Specifically, the audio for training (for distinction, it can be called the second audio) can be obtained first, and the second audio is a mixed audio of a single person's dry vocal sound and musical elements. Then, the second audio can be sound separated to obtain the dry vocal sound and musical elements in the second audio. Then, based on the dry vocal sound and musical elements in the second audio, the audio quantization model can be trained using the generative adversarial network framework to obtain a trained audio quantization model.

[0053] In practical applications, when the audio quantization model is trained using a generative adversarial network framework based on the dry vocal sounds and music elements in the second audio to obtain a trained audio quantization model, the following operations can be performed specifically: the audio processing model, the quantization module, and the audio decoder are trained based on the dry vocal sounds and music elements in the second audio to obtain a trained audio processing model, a trained quantization module, and a trained audio decoder; the discriminator is trained based on the dry vocal sounds and music elements in the second audio, the trained audio processing model, the trained quantization module, and the trained audio decoder until the discriminator training is completed to obtain a trained audio quantization model.

[0054] In addition, after completing the aforementioned training of the audiobook model, in order to further improve the stability of the audiobook model, a portion of high-quality training data can be selected to fine-tune the audiobook model. High-quality training data refers to training data with high signal clarity and low noise, that is, training data with a high signal-to-noise ratio. Using high-quality training data to train the audiobook model helps optimize the model training process, reduce the risk of overfitting, and improve the accuracy and stability of the fine-tuned audiobook model.

[0055] Combination of the above Figures 1 to 4 A method for training a large model of an audio book is described. Accordingly, the present disclosure also provides a method for generating an audio book. Figure 5 , which shows an exemplary flow chart of a method 500 for generating an audio book according to some embodiments of the present disclosure. It is understood that the method 500 may be executed by any appropriate device with data processing capabilities, such as but not limited to a processor, a terminal device, and a server.

[0056] like Figure 5 As shown, at step S501, method 500 may obtain text to be generated.

[0057] Next, at step S502, method 500 may input the text to be generated into the audio book model trained according to the method for training the audio book model described in the previous embodiment to perform a quantization code generation operation, so as to output a human voice quantization code and a music element quantization code. The music element here includes sound effects and / or background music.

[0058] Finally, at step S503, method 500 can use an audio decoder to perform an audio book generation operation based on the human voice quantization coding and the music element quantization coding to obtain an audio book audio containing human voice dry sound and music elements. Specifically, the audio decoder can first generate human voice dry sound based on the human voice quantization coding and the timbre reference, and generate music elements based on the music element quantization coding and the timbre reference. Then, the human voice dry sound is mixed with the music element to obtain the audio book.

[0059] As described above, the model prompt words composed according to the thought chain enable the audiobook model to show the reasoning steps, so that technicians can directly influence the problem-solving process of the big model and obtain multiple modes of reasoning. Figure 6 Method 600 for generating an audio book according to some other embodiments of the present disclosure is similar to the method 500 described above, and the method 600 can also be executed by any appropriate device with data processing capability.

[0060] like Figure 6 As shown, at step S601, method 600 may obtain text to be generated and emotional style description information.

[0061] Next, at step S602, method 600 may input the text to be generated and the emotional style description information into the audio book big model trained according to the method for training the audio book big model described in the previous embodiment to perform a quantization coding generation operation to output the human voice quantization code and the music element quantization code.

[0062] Finally, at step S603, method 600 can use an audio decoder to perform an audio book generation operation based on the human voice quantization coding and the music element quantization coding to obtain an audio book audio containing human voice dry sound and music elements. Specifically, the audio decoder can first generate human voice dry sound based on the human voice quantization coding and the timbre reference, and generate music elements based on the music element quantization coding and the timbre reference. Then, the human voice dry sound is mixed with the music element to obtain the audio book.

[0063] In practical applications, in addition to the application scenarios of generating audiobook audio based on the text to be generated and the application scenarios of generating audiobook audio based on the text to be generated and the emotional style description information, there is also an application scenario of generating audiobook audio based on the text to be generated, the emotional style description information and the music element description information. In this application scenario, in order to make full use of the known information such as the text to be generated, the emotional style description information and the music element description information, reasoning can be performed twice to generate audiobook audio.

[0064] Here, you can refer to Figure 7 It is understood that the method 700 for generating an audio book according to some other embodiments of the present disclosure is shown. Similar to the methods 500 and 600 described above, the method 700 can also be executed by any appropriate device with data processing capabilities. Figure 7 As shown, at step S701, method 700 may obtain text to be generated, emotional style description information, and music element description information.

[0065] Next, at step S702, method 700 may input the text to be generated and the emotional style description information into the audio book large model (for the sake of distinction, it may be referred to as the first audio book large model) trained according to the method for training the audio book large model described in the previous embodiment to perform a quantization coding generation operation to output the human voice quantization code.

[0066] Next, at step S703, method 700 may input the text to be generated, the emotional style description information and the voice quantization code into the audio book large model (for the sake of distinction, it may be referred to as the second audio book large model) trained according to the method for training the audio book large model described in the previous embodiment to perform a quantization code generation operation to output the music element quantization code.

[0067] Finally, at step S704, method 700 can use an audio decoder to perform an audio book generation operation based on the human voice quantization coding and the music element quantization coding to obtain an audio book audio containing human voice dry sound and music elements. Specifically, the audio decoder can first generate human voice dry sound based on the human voice quantization coding and the timbre reference, and generate music elements based on the music element quantization coding and the timbre reference. Then, the human voice dry sound is mixed with the music element to obtain the audio book.

[0068] Next, combine Figure 8 An electronic device 800 provided in an embodiment of the present application is exemplarily introduced. Figure 8 As shown, the electronic device 800 of the embodiment of the present application may include a processor 801 , a memory 802 and a communication bus 803 .

[0069] In the process of a specific embodiment, the processor 801 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing image processing device (DSPD), a programmable logic image processing device (PLD), a field programmable gate array (FPGA), a CPU, a controller, a microcontroller, and a microprocessor. It can be understood that for different devices, the electronic device used to implement the function of the processor may also be other, which is not specifically limited in this embodiment.

[0070] In the embodiment of the present application, the communication bus 803 is used to realize the connection and communication between the processor 801 and the memory 802; the memory 802 stores program instructions for adjusting the scanning electron microscope image; when the processor 801 executes the program instructions stored in the memory 802, the present application is realized. Figures 1 to 4 The method described for training a large model of audiobooks or a combination thereof Figures 5 to 7 A method for generating an audio book is described.

[0071] Combination of the above Figure 8 An electronic device for training a large audiobook model or for generating an audiobook that can be used to execute the present application is described. It should be understood that the device structure or architecture here is merely exemplary, and the implementation method and implementation entity of the present application are not limited thereto, but can be changed without departing from the spirit of the present application. It can be understood that the description of each embodiment in the present disclosure emphasizes the differences between the various embodiments, and the same or corresponding parts can be referenced to each other. For the purpose of brevity, the present disclosure will not go into details one by one.

[0072] According to the above description in combination with the accompanying drawings, those skilled in the art can also understand that the embodiments of the present application can also be implemented by a software program. Therefore, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores program instructions for training a large audio book model or for generating an audio book. The program instructions can be used to implement the present application in combination with Figures 1 to 4 The method described for training a large model of audiobooks or a combination thereof Figures 5 to 7 A method for generating an audio book is described.

[0073] It should be noted that although the operations of the method of the present application are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0074] Although multiple embodiments of the present application have been shown and described herein, it is obvious to those skilled in the art that such embodiments are provided only by way of example. Those skilled in the art can think of many changes, modifications and alternatives without departing from the thought and spirit of the present application. It should be understood that in the process of practicing the present application, various alternatives to the embodiments of the present application described herein can be adopted. The attached claims are intended to limit the scope of protection of the present application, and therefore cover equivalents or alternatives within the scope of these claims.< / text>

Claims

1. A method for training a large audiobook model, comprising: Acquire a first audio, where the first audio is a mixed audio of a single person's dry vocal sound and a musical element, where the musical element includes a sound effect and / or background music; Preprocessing the first audio to obtain text information, first description information, first quantization code, second description information and second quantization code of the first audio, wherein the first description information is a description of the emotion and style expressed by the dry vocal sound, the first quantization code is a quantization of the semantic features of the dry vocal sound, the second description information is a description of the scene and atmosphere created by the music element, and the second quantization code is a quantization of the semantic features of the music element; Combining the text information, the first description information, the first quantitative code, the second description information and the second quantitative code according to a thought chain to obtain a model prompt word; The model prompt words are input into the audio book large model to train the audio book large model.

2. The method according to claim 1, wherein: Preprocessing the first audio to obtain text information, first description information, second description information, and second quantization code of the first audio includes: Performing speech recognition on the first audio to obtain the text information; Performing sound separation on the first audio to obtain the dry human voice and the music element in the first audio; Describing the emotion and style expressed by the dry vocal sound using natural language to obtain the first description information, and describing the scene and atmosphere created by the musical element using natural language to obtain the second description information; The dry vocal sound and the musical element are respectively input into an audio quantization model to obtain the first quantization code and the second quantization code.

3. The method according to claim 2, wherein: The audio quantization model includes an audio processing model and a quantization module. The audio quantization model is a model obtained by training using a generative adversarial network framework. The generative adversarial network framework includes the audio quantization model, an audio decoder and a discriminator. as well as Using the generative adversarial network framework to obtain the audio quantization model includes: Acquire a second audio, where the second audio is a mixed audio of a dry vocal sound of a single person and a musical element; Performing sound separation on the second audio to obtain a dry human voice and the music element in the second audio; Based on the dry vocal sound and the musical elements in the second audio, the audio quantization model is trained using the generative adversarial network framework to obtain a trained audio quantization model.

4. The method according to claim 3, wherein: Based on the dry human voice and the music element in the second audio, the audio quantization model is trained using the generative adversarial network framework to obtain the trained audio quantization model, including: Training the audio processing model, the quantization module, and the audio decoder based on the dry human voice and the music elements in the second audio to obtain a trained audio processing model, a trained quantization module, and a trained audio decoder; The discriminator is trained based on the dry vocal sound and the musical elements in the second audio, the trained audio processing model, the trained quantization module and the trained audio decoder until the discriminator training is completed, thereby obtaining the trained audio quantization model.

5. The method according to claim 1, wherein: Acquiring the first audio includes: Acquire a third audio, where the third audio is a mixed audio of dry vocal sounds of multiple persons and music elements; The third audio is segmented according to the rule of one person per sentence to obtain multiple first audios.

6. A method for generating an audio book, comprising: Get the text to be generated; Inputting the text to be generated into the audio book model trained by the method according to any one of claims 1 to 5 to perform a quantization coding generation operation, so as to output a human voice quantization coding and a music element quantization coding, wherein the music element includes sound effects and / or background music; Based on the human voice quantization coding and the music element quantization coding, an audio book generation operation is performed using an audio decoder to obtain audio book audio containing human voice dry sound and music elements.

7. A method for generating an audio book, comprising: Obtain the text to be generated and the emotional style description information; Inputting the text to be generated and the emotional style description information into the audio book model trained by the method according to any one of claims 1 to 5 to perform a quantization coding generation operation, so as to output a human voice quantization code and a music element quantization code, wherein the music element includes sound effects and / or background music; Based on the human voice quantization coding and the music element quantization coding, an audio book generation operation is performed using an audio decoder to obtain audio book audio containing human voice dry sound and music elements.

8. A method for generating an audio book, comprising: Acquire the text to be generated, the emotional style description information and the music element description information, wherein the music element includes sound effects and / or background music; Inputting the text to be generated and the emotional style description information into the first audio book model trained according to the method of any one of claims 1 to 5 to perform a quantization code generation operation to output a human voice quantization code; Inputting the text to be generated, the emotional style description information and the human voice quantization code into the second audio book large model trained by the method according to any one of claims 1 to 5 to perform a quantization code generation operation, so as to output the music element quantization code; Based on the human voice quantization coding and the music element quantization coding, an audio book generation operation is performed using an audio decoder to obtain audio book audio containing human voice dry sound and music elements.

9. An electronic device, comprising: processor; as well as A memory storing program instructions for training a large audiobook model or for generating an audiobook, wherein when the program instructions are executed by a processor, the method according to any one of claims 1 to 5 or the method according to any one of claims 6 to 8 is implemented.

10. A computer-readable storage medium storing program instructions for training a large audio book model or for generating an audio book, wherein the program instructions, when executed by a processor, implement the method according to any one of claims 1 to 5 or the method according to any one of claims 6 to 8.

Images