Audio content generation method, related device, equipment and storage medium

Abstract

The invention discloses an audio content generation method, a related device, equipment and a storage medium, and the audio content generation method comprises the steps: obtaining to-be-synthesized text content; synthesizing a content audio of the text content, and predicting a text fragment needing a sound effect and a first description text needing the sound effect in the text content; generating sound effect audio conforming to the first description text; and combining the audio effect audio at the audio position corresponding to the text segment in the content audio to obtain the audio content. According to the scheme, automatic generation of the audio content can be realized, and the requirements of diversity of sound effects and multi-event complex sound effects are met.

Description

Technical Field

[0001] The present application relates to the technical field of content generation, and in particular to an audio content generation method and related devices, equipment and storage media. Background Art

[0002] Thanks to the development of information technology, the technology for generating audio content such as audio novels has increasingly attracted the attention of researchers and creators.

[0003] In order to enhance the liveliness and fun of audio content, auxiliary music such as sound effects are usually added to the audio content including dialogues, narrations, etc. In the prior art, suitable music is generally selected manually from the music library where it is needed to be added to merge with the content audio to obtain audio content. This method cannot achieve the automatic generation of audio content, and is limited by the existing resources of the music library, and cannot meet the needs of the diversity of sound effects and complex sound effects of multiple events. In view of this, how to achieve the automatic generation of audio content and meet the needs of the diversity of sound effects and complex sound effects of multiple events has become a problem that needs to be solved urgently. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a method for generating audio content and related devices, equipment and storage media, which can realize the automatic generation of audio content and meet the needs of diverse sound effects and multi-event complex sound effects.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a method for generating audio content, including: obtaining text content to be synthesized; synthesizing content audio of the text content, and predicting a text segment requiring sound effects and a first description text of the required sound effects in the text content; generating sound effect audio that conforms to the first description text; and merging the sound effect audio at the audio position corresponding to the text segment in the content audio to obtain audio content.

[0006] In order to solve the above technical problems, the second aspect of the present application provides an audio content generation device, including: a text acquisition module, a synthesis prediction module, a sound effect generation module and an audio merging module, the text acquisition module is used to acquire the text content to be synthesized; the synthesis prediction module is used to synthesize the content audio of the text content, and predict the text fragment requiring sound effects and the first description text of the required sound effects in the text content; the sound effect generation module is used to generate sound effect audio that conforms to the first description text; the audio merging module is used to merge the sound effect audio at the audio position corresponding to the text fragment in the content audio to obtain the audio content.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, which at least includes a memory and a processor coupled to each other, the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the audio content generation method in the above first aspect.

[0008] In order to solve the above technical problem, the fourth aspect of the present application provides a computer-readable storage medium, which stores program instructions that can be executed by a processor, and the program instructions are used to implement the audio content generation method of the above first aspect.

[0009] The above scheme obtains the text content to be synthesized, then synthesizes the content audio of the text content, and predicts the text segment that needs the sound effect and the first description text of the required sound effect in the text content, so as to generate the sound effect audio that conforms to the first description text, and then merges the sound effect audio at the audio position corresponding to the text segment in the content audio to obtain the audio content. Therefore, on the one hand, the automatic generation of the audio content can be achieved without manual selection through the above-mentioned synthesis, prediction, generation, merging and other series of processes. On the other hand, by predicting the first description text of the required sound effect and generating the corresponding sound effect audio accordingly, compared with selecting the appropriate sound effect from the music library, it can break through the resource limitation of the music library, and adaptively generate the sound effect audio according to the actual needs, which helps to meet the diversity of sound effects and the needs of multi-event complex sound effects. Therefore, it is possible to achieve the automatic generation of audio content and meet the diversity of sound effects and the needs of multi-event complex sound effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of an embodiment of the method for generating audio content of the present application; Figure 2a It is a schematic diagram of an embodiment of the present application; Figure 2b It is a schematic diagram of an embodiment of the sound effect generation of the present application; Figure 2c is a schematic diagram of an embodiment of background generation of the present application; Figure 2d It is a process diagram of an embodiment of the method for generating audio content of the present application; Figure 3 It is a schematic diagram of the framework of an embodiment of an audio content generating device of the present application; Figure 4 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 5 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0011] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0012] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0013] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0014] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the audio content generation method of the present application. Specifically, it may include the following steps: Step S11: Obtain the text content to be synthesized.

[0015] In an implementation scenario, as a possible implementation example, the text content to be synthesized can come from text materials such as novels, scripts, poems, etc. For example, part or the whole can be selected from the text material as the text content to be synthesized. Taking the generation of an audio novel as an example, a chapter, a section or a paragraph can be extracted from the novel to be synthesized as the text content to be synthesized; or, the novel to be synthesized can be directly used as the text content to be synthesized. Of course, the above example is only one possible scenario of extracting the text content to be synthesized from the text material in the actual application process, and other possible scenarios will not be given one by one here.

[0016] In another implementation scenario, different from the aforementioned implementation, as another possible implementation example, the text content to be synthesized can be generated based on relevant factors such as the theme of the audio content to be generated. Taking the generation of an audio novel as an example, the relevant factors may include the theme "detective solving a case", the role "protagonist: Inspector A, supporting role: Baron B, supporting role: Servant C", etc., and the specific content of the relevant factors is not limited here, and further based on the relevant factors, a prompt instruction is constructed, such as the prompt instruction may include but is not limited to: "Please generate a novel based on the following factors. The relevant factors include: theme "detective solving a case", protagonist: Inspector A, supporting role: Baron B, supporting role: Servant C", and the specific content of the prompt instruction is not limited here. On this basis, the prompt instruction can be input into the generative large model to obtain the output text of the generative large model as the novel to be synthesized, that is, the text content to be synthesized. Of course, the above example is only a possible example of obtaining text content when generating an audio novel, and other possible situations will not be given one by one here.

[0017] Step S12: synthesizing the content audio of the text content, and predicting the text segment requiring sound effects and the first description text of the required sound effects in the text content.

[0018] In one implementation scenario, see Figure 2a , Figure 2a This is a schematic diagram of an embodiment of the present application. Figure 2a As shown in Scheme 1, as a possible implementation example, in order to synthesize the content audio of the text content, the text content can be first analyzed to obtain a phoneme sequence, and then the phoneme sequence can be sent to an acoustic model for modeling to obtain acoustic features, and finally the acoustic features can be sent to a vocoder, and the output audio signal is the content audio of the text content.

[0019] In another implementation scenario, different from the aforementioned implementation, as another possible implementation example, in order to synthesize the content audio of the text content, such as Figure 2a As shown in Solution 2, after performing text analysis on the text content, audio sampling points can be directly generated through an end-to-end model, which can be combined to form an audio signal as the content audio of the text content.

[0020] It should be noted that the above examples are merely several possible implementation examples of synthesizing audio signals of text content to obtain content audio. Other possible methods of synthesizing audio are not limited here and will not be given one by one.

[0021] In an implementation scenario, the text segments that require sound effects in the text content may specifically include, but are not limited to: text segments that describe actions and produce sounds when executing the actions (e.g., "someone slams the door", "someone knocks on the door carefully", "someone scrapes the window glass vigorously", etc.), and even multi-event situations such as "someone hits the table hard and makes the cutlery jingle", "oncoming vehicles collide violently at the intersection, causing gasoline to leak and explode due to sparks", and text segments that describe the environment and usually have sounds in the environment (the sounds may be relatively simple or may be multi-event situations) (e.g., "the vegetable market is crowded with people buying vegetables early in the morning", "the waves gently hit the beach in the evening", "the road is already full of cars waiting to pass during the rush hour", etc.). Of course, the above examples are only a few possible situations of text segments that require sound effects in actual applications, and other possible situations are not limited here, and no examples are given one by one.

[0022] In an implementation scenario, the first description text of the required sound effect is a text that describes the characteristics of the required sound effect and contains content, so as to guide the generation of sound effect audio accordingly. Exemplarily, taking the aforementioned text segment that requires sound effects "someone slams the door hard" as an example, its first description text may include but is not limited to "the sound of a heavy door slamming"; or, taking the aforementioned text segment that requires sound effects "someone knocks on the door carefully" as an example, its first description text may include but is not limited to "the sound of a gentle knock on the door"; or, taking the aforementioned text segment that requires sound effects "the vegetable market is crowded with people coming to buy vegetables early in the morning" as an example, its first description text may include but is not limited to "the sound of a lively crowd hawking and shouting"; or, taking the aforementioned text segment that requires sound effects "it is the peak of commuting and the road is already full of cars waiting to pass" as an example, its first description text may include but is not limited to "the sound of car horns coming one after another". Of course, the above examples are only several possible examples of the first description text in the actual application process, and other possible situations are not limited here, and no examples are given one by one.

[0023] In one implementation scenario, as a possible implementation example, prediction information related to sound effects can be obtained by predicting text content by a first deep neural network, and the first deep neural network can be trained using first sample text content and the annotation information of the first sample text content as a target, and the annotation of the first sample text content can be related to the sound effects required by the first sample text content, thereby forcing the first deep neural network to learn from the first sample text content and its annotation information how to predict information related to the required sound effects from text information through the training process.

[0024] In a specific implementation scenario, the first deep neural network may include but is not limited to: a convolutional neural network, a recurrent neural network, a long short-term memory network, a Transformer, etc., and the network result of the first deep neural network is not limited here.

[0025] In a specific implementation scenario, taking the prediction information related to sound effects including the text segment that needs sound effects in the text content and the first description text of the required sound effects as an example, the annotation information of the first sample text content may include the text segment that needs sound effects in the first sample text and the sample description text of the required sound effects. For example, taking the first sample text content "the vegetable market is crowded with people who come to buy vegetables early in the morning, so-and-so has bought vegetables from the vegetable market and arrived at the customer's door, he knocks on the door carefully, and after the door is opened, he passes the vegetables into the door, and finally closes the door gently" as an example, its annotation information may include: the text segment that needs sound effects "the vegetable market is crowded with people who come to buy vegetables early in the morning" and its sample description text "the lively crowd hawking and shouting", the text segment that needs sound effects "he knocks on the door carefully" and its sample description text "the sound of knocking gently", and the text segment that needs sound effects "finally close the door gently" and its sample description text "the sound of closing the door gently". Of course, the above example is merely an example of predictive information related to sound effects, including a text segment requiring sound effects in the text content and a first description text of the required sound effects. It is one possible example of the annotation information of the first sample text content. The specific content of the annotation information is not limited here, and examples are not given one by one.

[0026] In a specific implementation scenario, after obtaining the first sample text content, the first deep neural network can be trained. Specifically, the first sample text content can be predicted based on the first deep neural network to obtain the predicted segment of the first sample text content that requires sound effects and the predicted description text of the required sound effects, and the network parameters of the first deep neural network can be adjusted based on the difference between the predicted segment of the first sample text content that requires sound effects and the text segment in the annotation information, and the difference between the predicted description text of the required sound effects and the sample description text in the annotation information.

[0027] In a specific implementation scenario, please refer to Figure 2b , Figure 2b Schematic diagram of an embodiment of the sound effect generation of the present application. Figure 2b In the example on the left side of the dotted line, the text content "In the evening...sit down to work" can be predicted by the first deep neural network to obtain prediction information related to sound effects, namely, the text segment that needs sound effects "The streets of the town are still bustling with people" and its first description text "The noisy sound of the crowd and the sound of the engine turning", the text segment that needs sound effects "Hans ran home" and its first description text "The sound of hurried running", the text segment that needs sound effects "Hans ran home again" and its first description text "Quick footsteps", the text segment that needs sound effects "Close the door" and its first description text "The sound of the wooden door closing". Of course, the above example is only a possible example of the text content to be synthesized and its related information predicted by the first deep neural network for sound effects, and other possible situations will not be given one by one here.

[0028] In another implementation scenario, different from the aforementioned implementation, as another possible implementation example, the prediction information related to the sound effects can also be obtained by the first large language model predicting the text content under the first prompt instruction, and the first prompt instruction is used to instruct the first large language model to output the prediction information related to the text content and the required sound effects, thereby utilizing the general understanding ability of the large language model to predict information related to the required sound effects from the text information.

[0029] In a specific implementation scenario, the first language model may include but is not limited to open source large models such as Llama and Bloom, or the first language model may be obtained by fine-tuning parameters of a specific data set (such as a small amount of the aforementioned first sample text content and its annotation information), or the first language model may be a custom large model. The specific source of the first language model is not limited here.

[0030] In a specific implementation scenario, please refer to Figure 2b , Figure 2b Schematic diagram of an embodiment of the sound effect generation of the present application. Figure 2b In the example on the right side of the dotted line, the text content "In the evening...sit down to work" is predicted by the first language model under the instruction of the first prompt instruction "predict which segments in the above paragraph need sound effects, and the description text of the required sound effects", and the prediction information related to the sound effects can be obtained, that is, the text segment that needs sound effects "The streets of the town are still bustling with people" and its first description text "The noisy sound of the crowd and the sound of the engine turning", the text segment that needs sound effects "Hans ran home" and its first description text "The sound of rapid running", and the text segment that needs sound effects "Hans ran home again" and its first description text "Quick footsteps", the text segment that needs sound effects "Close the door" and its first description text "The sound of the wooden door closing". Of course, the above example is only a possible example of the text content to be synthesized and its prediction of sound effect related information by the first language model, and other possible situations will not be given one by one here.

[0031] Step S13: Generate sound effect audio that conforms to the first description text.

[0032] In one implementation scenario, as a possible implementation example, sound effect audio can be generated by a second deep neural network according to a first description text. The second deep neural network can be trained using sample sound effect description text and with the annotation information of the sample sound effect description text as a target. The annotation information of the sample sound effect description text may include sample sound effect audio that conforms to the sample sound effect description text. In this way, the training process can force the second deep neural network to learn how to generate sound effect audio that conforms to the description text as indicated.

[0033] In a specific implementation scenario, the second deep neural network may include but is not limited to: a convolutional neural network, a recurrent neural network, a long short-term memory network, a Transformer, etc. The network structure of the second deep neural network is not limited here.

[0034] In a specific implementation scenario, after obtaining the sample sound effect description text and its annotation information, the second deep neural network can be trained based on this. Specifically, the second deep neural network can generate audio according to the sample sound effect description text to obtain the predicted sound effect audio, and then adjust the network parameters of the second deep neural network based on the difference between the predicted sound effect audio and the sample sound effect audio.

[0035] In another implementation scenario, different from the aforementioned implementation, as another possible implementation example, the sound effect audio can also be generated by the audio diffusion model according to the first description text. Exemplarily, the audio diffusion model can include but is not limited to AudioLDM, etc., and the specific structure of the audio diffusion model is not limited here.

[0036] It should be noted that the above examples are merely several possible implementation examples of generating sound effect audio according to the first description text. Other possible implementation methods are not limited here and will not be given one by one.

[0037] Step S14: merge the sound effect audio at the audio position corresponding to the text segment in the content audio to obtain the audio content.

[0038] In one implementation scenario, as a possible implementation example, the sound effect audio of the text segment can be directly inserted at the audio position corresponding to the text segment in the content audio to achieve track merging and obtain the audio content.

[0039] In another implementation scenario, different from the aforementioned implementation, as another possible implementation example, the prediction information related to the sound effect may include not only the text segment in the text content that requires the sound effect and the first description text of the required sound effect, but also the type of integration of the sound effect required by the text segment, that is, the way to integrate with the content audio. Before integrating the sound effect audio with the content audio, the type of integration of the sound effect required by the text segment can be predicted, and the integration type is one of several types, and several types can at least include insertion and superposition. On this basis, the sound effect audio can be integrated at the audio position according to the integration type to obtain the audio content, so that the integration of the sound effect can be more reasonable and natural.

[0040] In a specific implementation scenario, when the track type is overlay, the starting character and ending character of the sound effect required by the text segment can also be predicted, and the aforementioned audio position can specifically be the starting time corresponding to the starting character to the ending time corresponding to the ending character in the content audio. Figure 2b , the merge type of the predicted text segment "The streets of the town are still bustling with people" is "insertion", the merge type of the text segment "Hans runs home" is "overlay", and the starting character of the required sound effect is "Han", and the ending character is "Jia" (that is, the sound effect audio is superimposed with the audio of "Hans runs home" in the content audio), the merge type of the text segment "Hans runs home again" is "overlay", and the starting character of the required sound effect is "Han", and the ending character is "Jia" (that is, the sound effect audio is superimposed with the audio of "Hans runs home again" in the content audio), and the merge type of the text segment "Close the door" is "insertion". Of course, the above example is only a possible example when the prediction information further includes the merge type and the merge type is superposition in the actual application process, and other possible situations are not given examples one by one here.

[0041] In a specific implementation scenario, when the prediction information related to sound effects also includes the type of sound effects required for the text fragment and the starting and ending characters when the type of sound effects is superposition, the prediction information can still be obtained by predicting the text content by the first deep neural network. At this time, the first deep neural network can still use the first sample text content and train with the annotation information of the first sample text content as the target. The difference is that at this time, the annotation information of the first sample text content can further include the type of sound effects required for the first sample text content and the starting and ending characters when the type of sound effects is superposition, so as to force the first deep neural network to learn through the training process to further predict the type of sound effects required for the text fragment requiring sound effects and the starting and ending characters when the type of sound effects is superposition from the text content. Other similarities or similarities can be found in the aforementioned related descriptions, which will not be repeated here. Alternatively, when the prediction information related to the sound effect also includes the type of sound effect required for the text segment and the starting character and ending character when the type of sound effect is superposition, the prediction information can still be obtained by the first large language model predicting the text content under the first prompt instruction, and the first prompt instruction is used to instruct the first large language model to output the prediction information related to the text content and the required sound effect. The difference is that at this time the first prompt instruction not only instructs the first large language model to analyze the text segment requiring sound effects and the description text of the required sound effects from the text content, but also instructs the first large language model to further analyze the type of sound effect required and the starting character and ending character when the type of sound effect is superposition. For other similarities or similarities, please refer to the aforementioned related description, which will not be repeated here.

[0042] In one implementation scenario, before combining the sound effect audio to obtain the audio content, the second description text of the background sound required by the text content can also be predicted, and background audio that conforms to the second description text can be generated, and then the sound effect audio can be combined at the audio position in the content audio, and the background audio and the content audio can be combined as a whole to obtain the audio content, so that the final synthesized audio content can not only have sound effects in the relevant segments, but also have an overall background to create an atmosphere. It should be noted that the background audio is the audio information used to express the background and create an atmosphere under the content audio in the audio content, that is, the background audio can be different from the sound effect audio, but can be combined with the content audio as a whole.

[0043] In a specific implementation scenario, the second description text of the required background sound may be a text describing the elements of the required background sound and its atmosphere, so as to guide the generation of background audio accordingly. Figure 2b Taking the Chinese text as an example, the second description text may include but is not limited to: "The overall atmosphere is a relaxed and pleasant one, with relaxing melodies and bright instrumental performances, interspersed with some mysterious and suspenseful feelings to increase the suspense of the story." Of course, the above examples are only several possible examples of the second description text in the actual application process. Other possible situations are not limited here, and no examples are given one by one.

[0044] In a specific implementation scenario, as a possible implementation example, the second description text can be obtained by predicting the text content by a third deep neural network, and the third deep neural network can be trained using the second sample text content and the annotation information of the second sample text content, and the annotation information of the second sample text content can include the sample description text of the background sound required for the second sample text content. It should be noted that the third deep neural network can include but is not limited to: convolutional neural network, recurrent neural network, long short-term memory network, Transformer, etc., and the network results of the third deep neural network are not limited here. In addition, the specific process of training the third deep neural network with the second sample text content can refer to the related description of training the first deep neural network with the first sample text content, which will not be repeated here. Please refer to Figure 2c , Figure 2c Schematic diagram of an embodiment of background generation of this application. Figure 2c The left side of the dotted line indicates that the text content "In the evening...sit down to work" can be predicted by the third deep neural network to obtain the second description text of the background sound required by the text content "on the whole, it is a relaxed and pleasant atmosphere, with relaxing melodies and bright instrumental performances, interspersed with some mysterious and suspenseful feelings to increase the suspense of the story". Of course, the above example is only a possible example of the text content to be synthesized and its related information of the background sound predicted by the third deep neural network, and other possible situations will not be given one by one here.

[0045] In a specific implementation scenario, different from the aforementioned implementation, as another possible implementation example, the second description text can also be obtained by the second largest language model predicting the text content under the second prompt instruction, and the second prompt instruction can be used to instruct the second largest language model to output the description text of the background sound required for the text content. It should be noted that the second largest language model may include but is not limited to open source large models such as Llama and Bloom, or the second largest language model may also be obtained by fine-tuning parameters of a specific data set (such as a small amount of the aforementioned second sample text content and its annotation information), or the second largest language model may also be a custom large model. The specific source of the second largest language model is not limited here. Please refer to Figure 2c , Figure 2c Schematic diagram of an embodiment of background generation of this application. Figure 2c The right side of the dotted line indicates that the text content "In the evening... sit down to work" is predicted by the second largest language model under the instruction of the second prompt instruction "predict what kind of background music should be matched with the above paragraph", and the prediction information related to the background sound can be obtained, that is, the second description text "is generally in a relaxed and pleasant atmosphere, with relaxed melodies and bright instrumental performances, interspersed with some mysterious and suspenseful feelings to increase the suspense of the story". Of course, the above example is only a possible example of the text content to be synthesized and its prediction of the background sound related information by the second largest language model, and other possible situations will not be given one by one here.

[0046] In a specific implementation scenario, as a possible implementation example, after obtaining the second description text, the background audio can be generated by the fourth deep neural network according to the second description text, and the fourth deep neural network can be trained using the sample background description text and the annotation information of the sample background description text as the target, and the annotation information of the sample background description text may include sample background audio that conforms to the sample background description text. It should be noted that the fourth deep neural network may include but is not limited to: convolutional neural network, recurrent neural network, long short-term memory network, Transformer, etc., and the network structure of the fourth deep neural network is not limited here. In addition, after obtaining the sample background description text and its annotation information, the fourth deep neural network can be trained based on this. Specifically, audio can be generated according to the sample background description text based on the fourth deep neural network to obtain predicted background audio, and then the network parameters of the fourth deep neural network can be adjusted based on the difference between the predicted background audio and the sample background audio.

[0047] In another specific implementation scenario, different from the aforementioned implementation, as another possible implementation example, after obtaining the second description text, the background audio can also be generated by the audio diffusion model according to the second description text. Exemplarily, the audio diffusion model can include but is not limited to AudioLDM, etc., and the specific structure of the audio diffusion model is not limited here.

[0048] It should be noted that the above examples are only several possible implementation examples of generating background audio according to the second description text, and other possible implementation methods are not limited here and will not be given one by one. Figure 2d , Figure 2d 1 is a process diagram of an embodiment of the audio content generation method of the present application. Figure 2d As shown, the text content to be synthesized is divided into three paths, one path predicts the sound effect related information and generates the sound effect audio based on it, one path performs audio synthesis to obtain the content audio, and one path predicts the background related information to generate the background audio based on it. Finally, the sound effect audio, content audio and background audio obtained from the above three paths are combined to obtain the audio content.

[0049] In one implementation scenario, in order to further increase the fun, a picture sequence that matches the content of each segment in the text content can also be generated. Figure 2b Taking the text content shown as an example, the generated picture sequence may include the following pictures in sequence: the first picture "depicting the scene of people coming and going on the streets of a small town in the evening", the second picture "depicting Hans waiting for Elsa in the house", the third picture "depicting Hans going out", the fourth picture "depicting Hans coming to the wheat field", the fifth picture "depicting that there is no wheat being cut in the wheat field, and Elsa is lying on the field and snoring", the sixth picture "depicting Hans running home to get the bird net with a small bell tied to it", the seventh picture "depicting Hans covering Elsa with the bird net, but Elsa still didn't wake up", the eighth picture "depicting Hans running home again", the ninth picture "depicting Hans closing the door", and the tenth picture "depicting Hans sitting down to work". It should be noted that the above examples are only based on Figure 2b The text content shown is a possible example of an example picture sequence, and other possible situations are not given one by one here. In addition, the picture sequence can be generated by a diffusion model based on text-generated images according to the text content, or the picture sequence can also be generated by a generative large model according to the text content, which is not limited here. On this basis, video content can be generated based on audio content and picture sequences. In other words, the audio track of the video content can be audio content, and the frame sequence of the video content can be a picture sequence. In addition, in the process of generating video content by combining picture sequences and audio content, the audio content and the picture sequence can also be semantically aligned based on the text content.

[0050] The above scheme obtains the text content to be synthesized, then synthesizes the content audio of the text content, and predicts the text segment that needs the sound effect and the first description text of the required sound effect in the text content, so as to generate the sound effect audio that conforms to the first description text, and then merges the sound effect audio at the audio position corresponding to the text segment in the content audio to obtain the audio content. Therefore, on the one hand, the automatic generation of the audio content can be achieved without manual selection through the above-mentioned synthesis, prediction, generation, merging and other series of processes. On the other hand, by predicting the first description text of the required sound effect and generating the corresponding sound effect audio accordingly, compared with selecting the appropriate sound effect from the music library, it can break through the resource limitation of the music library, and adaptively generate the sound effect audio according to the actual needs, which helps to meet the diversity of sound effects and the needs of multi-event complex sound effects. Therefore, it is possible to achieve the automatic generation of audio content and meet the diversity of sound effects and the needs of multi-event complex sound effects.

[0051] See also Figure 3 , Figure 3 It is a schematic diagram of the framework of an embodiment of the audio content generation device of the present application. The audio content generation device 30 includes: a text acquisition module 31, a synthesis prediction module 32, a sound effect generation module 33 and an audio merging module 34. The text acquisition module 31 is used to acquire the text content to be synthesized; the synthesis prediction module 32 is used to synthesize the content audio of the text content, and predict the text segment that needs the sound effect and the first description text of the required sound effect in the text content; the sound effect generation module 33 is used to generate the sound effect audio that conforms to the first description text; the audio merging module 34 is used to merge the sound effect audio at the audio position corresponding to the text segment in the content audio to obtain the audio content.

[0052] In the above scheme, the audio content generation device 30 obtains the text content to be synthesized, synthesizes the content audio of the text content, and predicts the text segment that needs the sound effect and the first description text of the required sound effect in the text content, thereby generating the sound effect audio that conforms to the first description text, and then merges the sound effect audio at the audio position corresponding to the text segment in the content audio to obtain the audio content. Therefore, on the one hand, the automatic generation of audio content can be achieved without manual selection through the above-mentioned synthesis, prediction, generation, merging and other series of processes. On the other hand, by predicting the first description text of the required sound effect and generating the corresponding sound effect audio accordingly, compared with selecting the appropriate sound effect from the music library, it can break through the resource limitation of the music library, and adaptively generate the sound effect audio according to the actual needs, which helps to meet the diversity of sound effects and the needs of multi-event complex sound effects. Therefore, it is possible to achieve the automatic generation of audio content and meet the diversity of sound effects and the needs of multi-event complex sound effects.

[0053] In some disclosed embodiments, the audio content generation device 30 includes a type prediction module for predicting the type of sound effects required for a text segment; wherein the type of sound effects is one of several types, and the several types include at least insertion and overlay; the audio sound effects module 34 is specifically used to combine the sound effects at the audio position according to the type of sound effects to obtain audio content.

[0054] In some disclosed embodiments, the type prediction module is also used to predict the starting character and ending character of the sound effect required for the text segment when the track-merging type is overlay; wherein the audio position is the starting time corresponding to the starting character to the ending time corresponding to the ending character in the content audio.

[0055] In some disclosed embodiments, the prediction information related to the sound effects is obtained by predicting the text content by a first deep neural network, and the first deep neural network is trained using the first sample text content and with the annotation information of the first sample text content as a target, and the annotation information of the first sample text content is related to the sound effects required for the first sample text content; or, the prediction information related to the sound effects is obtained by predicting the text content by a first large language model under a first prompt instruction, and the first prompt instruction is used to instruct the first large language model to output the prediction information related to the text content and the required sound effects.

[0056] In some disclosed embodiments, the sound effect audio is generated by a second deep neural network according to the first description text, the second deep neural network uses sample sound effect description text and is trained with the annotation information of the sample sound effect description text as a target, and the annotation information of the sample sound effect description text includes sample sound effect audio that conforms to the sample sound effect description text; or, the sound effect audio is generated by an audio diffusion model according to the first description text.

[0057] In some disclosed embodiments, the audio content generating device 30 includes a description prediction module for predicting a second description text of the background sound required for the text content; the audio content generating device 30 includes a background generation module for generating background audio that conforms to the second description text; the audio merging module 34 is specifically used to merge the sound effect audio at the audio position in the content audio, and merge the background audio with the content audio as a whole to obtain the audio content.

[0058] In some disclosed embodiments, the second descriptive text is obtained by predicting the text content by a third deep neural network, and the third deep neural network adopts the second sample text content and is trained with the annotation information of the second sample text content as a target, and the annotation information of the second sample text content includes the sample descriptive text of the background sound required for the second sample text content; or, the second descriptive text is obtained by predicting the text content by a second large language model under a second prompt instruction, and the second prompt instruction is used to instruct the second large language model to output the descriptive text of the background sound required for the text content.

[0059] In some disclosed embodiments, the background audio is generated by a fourth deep neural network according to the second description text, the fourth deep neural network adopts the sample background description text and is trained with the annotation information of the sample background description text as a target, and the annotation information of the sample background description text includes the sample background audio that conforms to the sample background description text; or, the background audio is generated by an audio diffusion model according to the second description text.

[0060] In some disclosed embodiments, the audio content generating device 30 includes a sequence generating module for generating a sequence of pictures corresponding to each fragment content in the text content; the audio content generating device 30 includes a video generating module for generating video content based on the audio content and the picture sequence.

[0061] See also Figure 4 , Figure 4 : is a schematic diagram of the framework of an embodiment of an electronic device of the present application. The electronic device 40 includes at least a memory 41 and a processor 42 coupled to each other, the memory 41 stores at least program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-mentioned audio content generation method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 40 may include but is not limited to a smart phone, a tablet computer, a learning machine, an office book, a server, etc., and the specific type of the electronic device 40 is not limited here.

[0062] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above-mentioned embodiments of the audio content generation method. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0063] In the above scheme, the electronic device 40 obtains the text content to be synthesized, synthesizes the content audio of the text content, and predicts the text segment that needs the sound effect and the first description text of the required sound effect in the text content, thereby generating the sound effect audio that conforms to the first description text, and then merges the sound effect audio at the audio position corresponding to the text segment in the content audio to obtain the audio content. Therefore, on the one hand, the automatic generation of the audio content can be achieved without manual selection through the above-mentioned synthesis, prediction, generation, merging and other series of processes. On the other hand, by predicting the first description text of the required sound effect and generating the corresponding sound effect audio accordingly, compared with selecting the appropriate sound effect from the music library, it can break through the resource limitation of the music library, and adaptively generate the sound effect audio according to the actual needs, which helps to meet the diversity of sound effects and the needs of multi-event complex sound effects. Therefore, it is possible to achieve the automatic generation of audio content and meet the diversity of sound effects and the needs of multi-event complex sound effects.

[0064] See also Figure 5 , Figure 5 The schematic diagram is a schematic diagram of a computer-readable storage medium 50 of the present application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor, and the program instructions 51 are used to implement the steps in any of the above-mentioned audio content generation method embodiments.

[0065] In the above scheme, the computer-readable storage medium 50 obtains the text content to be synthesized, then synthesizes the content audio of the text content, and predicts the text segment that needs the sound effect and the first description text of the required sound effect in the text content, thereby generating the sound effect audio that conforms to the first description text, and then merges the sound effect audio at the audio position corresponding to the text segment in the content audio to obtain the audio content. Therefore, on the one hand, the automatic generation of the audio content can be achieved without manual selection through the above-mentioned synthesis, prediction, generation, merging and other series of processes. On the other hand, by predicting the first description text of the required sound effect and generating the corresponding sound effect audio accordingly, compared with selecting the appropriate sound effect from the music library, it can break through the resource limitation of the music library, and adaptively generate the sound effect audio according to the actual needs, which helps to meet the diversity of sound effects and the needs of multi-event complex sound effects. Therefore, it is possible to achieve the automatic generation of audio content and meet the diversity of sound effects and the needs of multi-event complex sound effects.

[0066] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0067] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0068] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0069] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0070] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0071] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0072] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A method for generating audio content, characterized in that: include: Get the text content to be synthesized; synthesizing content audio of the text content, and predicting a text segment in the text content that requires a sound effect and a first description text of the required sound effect; Generating sound effect audio that conforms to the first description text; The sound effect audio is combined with the audio position corresponding to the text segment in the content audio to obtain the audio content.

2. The method according to claim 1, characterized in that Before combining the sound effect audio at the audio position corresponding to the text segment in the content audio to obtain the audio content, the method further includes: Predicting a type of sound effects required for the text segment; wherein the type of sound effects is one of several types, and the several types at least include insertion and superposition; The step of combining the sound effect audio at the audio position corresponding to the text segment in the content audio to obtain the audio content includes: The sound effect audio is merged at the audio position according to the merge type to obtain the audio content.

3. The method according to claim 2, characterized in that When the track merging type is superposition, the method further includes: Predict the starting character and ending character of the sound effect required for the text segment; wherein the audio position is the time from the starting moment corresponding to the starting character to the ending moment corresponding to the ending character in the content audio.

4. The method according to any one of claims 1 to 3, characterized in that: The prediction information related to the sound effect is obtained by predicting the text content by a first deep neural network, and the first deep neural network is trained using the first sample text content and with the annotation information of the first sample text content as a target, and the annotation information of the first sample text content is related to the sound effect required by the first sample text content; or The prediction information related to the sound effect is obtained by the first large language model predicting the text content under the first prompt instruction, and the first prompt instruction is used to instruct the first large language model to output the prediction information related to the text content and the required sound effect.

5. The method according to claim 1, characterized in that The sound effect audio is generated by a second deep neural network according to the first description text, the second deep neural network adopts a sample sound effect description text and is trained with the annotation information of the sample sound effect description text as a target, and the annotation information of the sample sound effect description text includes the sample sound effect audio that conforms to the sample sound effect description text; or The sound effect audio is generated by an audio diffusion model according to the first description text.

6. The method according to claim 1, characterized in that Before combining the sound effect audio at the audio position corresponding to the text segment in the content audio to obtain the audio content, the method further includes: A second description text for predicting background sound required for the text content; generating background audio that conforms to the second description text; The step of combining the sound effect audio at the audio position corresponding to the text segment in the content audio to obtain the audio content includes: The sound effect audio is integrated at the audio position in the content audio, and the background audio is integrated with the content audio as a whole to obtain the audio content.

7. The method according to claim 6, characterized in that The second description text is obtained by predicting the text content by a third deep neural network, and the third deep neural network is trained using the second sample text content and taking the annotation information of the second sample text content as a target, and the annotation information of the second sample text content includes a sample description text of the background sound required for the second sample text content; or The second description text is obtained by predicting the text content by the second largest language model under the second prompt instruction, and the second prompt instruction is used to instruct the second largest language model to output the description text of the background sound required for the text content.

8. The method according to claim 6, characterized in that The background audio is generated by a fourth deep neural network according to the second description text, the fourth deep neural network adopts a sample background description text and is trained with the annotation information of the sample background description text as a target, and the annotation information of the sample background description text includes the sample background audio that conforms to the sample background description text; or The background audio is generated by an audio diffusion model according to the second description text.

9. The method according to claim 1, characterized in that: The method further comprises: Generate a sequence of pictures that match the contents of each segment in the text content; Video content is generated based on the audio content and the picture sequence.

10. An audio content generating device, characterized in that: include: A text acquisition module, used to acquire the text content to be synthesized; A synthesis prediction module, used for synthesizing the content audio of the text content, and predicting the text segment requiring sound effects and the first description text of the required sound effects in the text content; A sound effect generation module, used to generate sound effect audio that conforms to the first description text; The audio merging module is used to merge the sound effect audio at the audio position corresponding to the text segment in the content audio to obtain the audio content.

11. An electronic device, characterized in that: The method comprises at least a memory and a processor coupled to each other, wherein the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the audio content generation method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the audio content generation method according to any one of claims 1 to 9.

Images