Audio generation method and device, equipment, storage medium and program product
By obtaining natural language text and performing attribute prediction and encoding and decoding, a unified audio attribute description text is generated, and the problem of inconsistent themes of audio content in the existing technology is solved, and the unified theme generation and quality improvement of audio content is achieved.
Patent Information
- Application Number
- CN202411080437.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-22
AI Technical Summary
The audio content generated by the existing intelligent audio production solution is not uniform in the theme, and the generation effect does not meet expectations. The audio attributes such as style, sound source, volume, etc. cannot be effectively controlled, resulting in mismatch in the patchwork results and requires manual adjustment.
By obtaining the user's natural language text, performing attribute prediction to obtain audio attribute description text with unified text configuration format, using this description text to generate audio data related to user needs, and using multi-task multi-classification header and text feature extraction model for encoding and decoding. Generate a unified audio attribute description and generate audio content that matches the topic through the audio generation model.
It realizes the unified audio content needs input by users from the text analysis level, and generates thematic unified audio content, which improves the generation quality and reduces the need for manual adjustment.
Smart Images

Figure CN119007707A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent music technology, and in particular to an audio generation method, device, equipment, storage medium and program product. Background Art
[0002] Existing intelligent audio production solutions are all aimed at generating a single type of audio. For example, voice content is generated by using models such as text-to-speech / Voice Conversion (TTS / VC), sound effect content is generated by using sound effect generation models, and music content is generated by using music generation models. Each audio content is generated using different solutions or models, and finally the final audio content is obtained by piecing together the generated content. This will result in the problem that the theme of the final generated content is not unified, and the audio content generation effect does not meet expectations. Summary of the invention
[0003] The purpose of the present invention is to provide an audio generation method, device, equipment, storage medium and program product, which are used to solve the problem that the theme of audio content generated by the existing technology is not unified and the audio content generation effect does not meet expectations.
[0004] In order to solve the above technical problem, an embodiment of the present invention provides an audio generation method, which includes:
[0005] Get the text corresponding to the user's natural language;
[0006] Performing attribute prediction based on the text corresponding to the natural language of the user to obtain an audio attribute description text with a unified text configuration format, wherein the audio attribute description text is used to represent the audio information prompted by the text corresponding to the natural language of the user;
[0007] The audio attribute description text is used to generate audio data related to the text corresponding to the natural language of the user.
[0008] In some embodiments, the attribute prediction based on the text corresponding to the natural language of the user to obtain the audio attribute description text with a unified text configuration format includes:
[0009] Encoding the text corresponding to the natural language of the user to obtain a first encoded content;
[0010] Based on the pre-created multi-task multi-classification header, the first encoded content is decoded to obtain a prediction probability value corresponding to each audio attribute, wherein each header has a corresponding audio attribute;
[0011] The predicted probability value corresponding to each audio attribute is converted and processed to obtain an audio attribute description text with a unified text configuration format.
[0012] In some embodiments, the attribute prediction based on the text corresponding to the natural language of the user to obtain the audio attribute description text with a unified text configuration format includes:
[0013] Inputting the text corresponding to the user's natural language into a text feature extraction model to obtain an audio attribute description text with a unified text configuration format;
[0014] The text feature extraction model is used to perform attribute prediction based on the text corresponding to the user's natural language, and obtain an audio attribute description text with a unified text configuration format.
[0015] In some embodiments, the method further comprises:
[0016] Obtaining a training set, wherein each training data in the training set includes an audio sample, a natural language description corresponding to the audio sample, and an attribute annotation;
[0017] Inputting the natural language description corresponding to the audio sample into the first preset model to obtain the predicted probability value corresponding to each audio attribute related to the audio sample; wherein the first preset model includes an encoder and a decoder, the encoder is used to tokenize the natural language description corresponding to the audio sample; the decoder includes a pre-created multi-task multi-classification head, each head includes a fully connected layer and a classification layer, and is used to classify the encoded content output by the encoder to obtain the predicted probability value of the corresponding audio attribute;
[0018] The first preset model is trained using a first loss function to obtain a trained text feature extraction model; wherein the first loss function is determined by the predicted probability value corresponding to each audio attribute related to the audio sample and the attribute label corresponding to the audio sample.
[0019] In some embodiments, the step of using the audio attribute description text to generate audio data related to the text corresponding to the natural language of the user includes:
[0020] Performing unified encoding and mapping on the audio attribute description text to obtain a second encoded content;
[0021] Decoding the second encoded content to obtain a Mel spectrum;
[0022] Audio data related to text corresponding to the natural language of the user is generated according to the Mel-spectrogram.
[0023] In some embodiments, the step of using the audio attribute description text to generate audio data related to the text corresponding to the natural language of the user includes:
[0024] The audio attribute description text is input into an audio generation model to obtain audio data related to the text corresponding to the natural language of the user.
[0025] In some embodiments, the method further comprises:
[0026] Obtaining a training set, wherein each training data in the training set includes an audio sample, a natural language description corresponding to the audio sample, and an attribute annotation;
[0027] Inputting the attribute annotation corresponding to the audio sample into a second preset model to obtain predicted audio data;
[0028] The second preset model is trained using a second loss function to obtain a trained audio generation model; wherein the second loss function is determined by the predicted audio data and the audio sample.
[0029] The embodiment of the present invention further provides an audio generation device, comprising:
[0030] A first acquisition module is used to acquire text corresponding to the user's natural language;
[0031] A first processing module, configured to perform attribute prediction based on the text corresponding to the natural language of the user, and obtain an audio attribute description text with a unified text configuration format, wherein the audio attribute description text is used to represent the audio information prompted by the text corresponding to the natural language of the user;
[0032] The audio generation module is used to use the audio attribute description text to generate audio data related to the text corresponding to the natural language of the user.
[0033] An embodiment of the present invention further provides an audio generating device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; when the processor executes the program, the above-mentioned audio generating method is implemented.
[0034] An embodiment of the present invention further provides a readable storage medium on which a program is stored. When the program is executed by a processor, the steps in the audio generation method described above are implemented.
[0035] An embodiment of the present invention further provides a computer program product, comprising computer instructions, which implement the steps in the above-mentioned audio generation method when executed by a processor.
[0036] The beneficial effects of the above technical solution of the present invention are as follows:
[0037] In the above scheme, the text corresponding to the user's natural language is obtained; then, based on the text corresponding to the user's natural language, attribute prediction is performed to obtain an audio attribute description text with a unified text configuration format, and the audio attribute description text is used to characterize the audio information prompted by the text corresponding to the user's natural language; finally, the audio attribute description text is used to generate audio data related to the text corresponding to the user's natural language, so that the audio content requirements to be generated corresponding to the text input by the user are unified from the text parsing level, and then the audio attribute description text with a unified text configuration can be used to generate audio content with a unified theme. The generated audio content is more in line with user expectations and the generation quality is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A schematic diagram of a flow chart of an audio generation method according to an embodiment of the present invention;
[0039] Figure 2 It is an overall framework diagram of the text feature extraction model of an embodiment of the present invention;
[0040] Figure 3 A schematic diagram of the architecture of an audio generation model according to an embodiment of the present invention;
[0041] Figure 4 A schematic diagram of the implementation flow of the audio generation method according to an embodiment of the present invention;
[0042] Figure 5 A schematic diagram of product interaction formed by the method of an embodiment of the present invention;
[0043] Figure 6 Schematic diagram of the structure of a frequency generating device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0044] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0045] The existing audio content is generated using different solutions. The inconsistency of the models will inevitably lead to inconsistency in the themes of the generated content. For example, "soothing" music and "soothing" sound effects will be generated using different solutions, which will be ambiguous and result in mismatched patchwork results, requiring a lot of manual adjustment work.
[0046] Audio content consists of multiple elements such as voice, music, and sound effects. Each element has its own attributes, such as style, sound source, volume, time point, duration, etc. Existing solutions have failed to independently control these attributes while forming a reasonable theme to meet user expectations.
[0047] In order to solve the above technical problems, the present invention provides an audio generation method, device, equipment, storage medium and program product, wherein the method and device are based on the same application concept. Since the principles of solving the problems by the method and device are similar, the implementation of the device and the method can refer to each other, and the repeated parts will not be repeated.
[0048] like Figure 1 FIG. 1 is a flow chart of an audio generation method provided by an embodiment of the present invention. The method may include:
[0049] Step 101, obtaining the text corresponding to the user's natural language;
[0050] Here, the text corresponding to the user's natural language can be obtained through user input. The user input can be the text corresponding to the user's natural language; it can also be the voice corresponding to the user's natural language. In this case, the voice can be converted into text through voice recognition technology to obtain the text corresponding to the user's natural language.
[0051] In order to standardize user input, the original text corresponding to the user's natural language is normalized. Specifically, grammatical errors, redundant words, etc. in the original text are preprocessed, such as correcting grammatical errors, deleting redundant words, etc., and finally the standardized text corresponding to the user's natural language is obtained.
[0052] It should be noted that the text representation corresponding to the user's natural language represents the audio content demand to be generated.
[0053] Step 102, performing attribute prediction based on the text corresponding to the natural language of the user to obtain an audio attribute description text with a unified text configuration format, wherein the audio attribute description text is used to represent the audio information prompted by the text corresponding to the natural language of the user;
[0054] The audio attributes include, but are not limited to: audio type, sound source type, sound source volume, start time (the time when the sound source starts playing), sound source length (the reference length of the sound source), and sound source content (only valid for voice content, text content to be synthesized).
[0055] The audio types include speech, sound effects, and music. The sound source types include speech types (referring to speaker types), sound effect types (referring to different sound effect types), and music types (referring to different music types).
[0056] Here, the audio attribute description text with a unified text configuration format should be understood as a structured audio attribute description text. The present invention performs a structured design on the audio attributes, and the details can be referred to in Table 1 below.
[0057]
[0058]
[0059] By predicting the attributes of the text corresponding to the user's natural language, the audio attributes can be predicted, that is, predicting which keywords the text corresponding to the user's natural language contains, that is, predicting which audio attributes the text contains, that is, extracting keywords, that is, audio attributes, from the text corresponding to the user's natural language; then, according to the pre-designed text configuration format, obtain the audio attribute description text with a unified text configuration format.
[0060] Through the above processing, the audio content requirements to be generated corresponding to the text input by the user are unified from the text analysis level.
[0061] Step 103: using the audio attribute description text, generate audio data related to the text corresponding to the natural language of the user.
[0062] Here, by using the audio attribute description text with a unified text configuration, it is possible to generate audio content with a unified theme. The generated audio content is more in line with user expectations and the generation quality is improved.
[0063] In an optional embodiment, in the above step 102, attribute prediction is performed based on the text corresponding to the natural language of the user to obtain an audio attribute description text with a unified text configuration format, including:
[0064] 1) Encoding the text corresponding to the natural language of the user to obtain a first encoded content;
[0065] Here, the encoding of the text corresponding to the natural language of the user may be a token-based encoding, and the obtained first encoding content may capture the semantic information of each word in the text.
[0066] 2) decoding the first encoded content based on a pre-created multi-task multi-classification head to obtain a prediction probability value corresponding to each audio attribute, wherein each head has a corresponding audio attribute;
[0067] Here, multi-task multi-classification refers to more than two classification tasks. The classification tasks refer to the different audio attributes mentioned above. A classification task may also include multiple classification tasks. For example, the classification task corresponding to the audio type also includes multiple sub-classification tasks, such as sub-classification tasks corresponding to voice, sub-classification tasks corresponding to sound effects, sub-classification tasks corresponding to music, etc.
[0068] Based on the pre-created multi-task multi-classification header, the first encoded content is decoded to classify the corresponding audio attributes, and different headers correspond to different audio attributes.
[0069] 3) Based on the predicted probability value corresponding to each audio attribute, a conversion process is performed to obtain an audio attribute description text with a unified text configuration format.
[0070] That is to say, the predicted probability value corresponding to the audio attribute can be converted into an audio attribute description text with a unified text configuration format.
[0071] It should be noted that the processing logic of obtaining the audio attribute description text with a unified text configuration format can be implemented by a text feature extraction model. In an optional embodiment, the above step 102 may include:
[0072] Inputting the text corresponding to the user's natural language into a text feature extraction model to obtain an audio attribute description text with a unified text configuration format;
[0073] The text feature extraction model is used to perform attribute prediction based on the text corresponding to the user's natural language, and obtain an audio attribute description text with a unified text configuration format.
[0074] That is, the processing logic of obtaining the audio attribute description text with a unified text configuration format is completed through the text feature extraction model, which is beneficial to practical applications.
[0075] Here, the text feature extraction model is obtained by pre-training, and the specific training process can be implemented by the following embodiments. In an optional embodiment, the method of the present invention also includes:
[0076] (1) obtaining a training set, wherein each training data in the training set includes an audio sample, a natural language description corresponding to the audio sample, and an attribute label;
[0077] Here, the training data set can be obtained through data collection / labeling. For example, about 100 hours of audio data can be collected from audio books, film and television dramas, including the audio content of the types defined in Table 1 above, and natural language text descriptions and attribute labels can be performed according to the defined fields, and a segment sample is formed every 30 seconds to form a training set. in:
[0078] is a natural language set, and the i-th audio sample is described in natural language, that is, Represents the natural language description corresponding to an audio sample.
[0079] It is the annotation of the attribute structure, corresponding to the attribute text structure in the example of Table 1, where each attribute structure element contains a defined field description:
[0080]
[0081] Yaudio It is a collection of audio samples after collection.
[0082] It should be noted that the training set TrainData will be used to train the text feature extraction model and the audio generation model described later.
[0083] (2) inputting the natural language description corresponding to the audio sample into a first preset model to obtain a predicted probability value corresponding to each audio attribute related to the audio sample; wherein the first preset model includes an encoder and a decoder, the encoder is used to tokenize the natural language description corresponding to the audio sample; the decoder includes a pre-created multi-task multi-classification head, each head includes a fully connected layer and a classification layer, and is used to classify the encoded content output by the encoder to obtain the predicted probability value of the corresponding audio attribute;
[0084] Here, only TrainData is used in text feature extraction model training. Set, so that the model through the input Predict the corresponding The overall architecture of the first preset model (also the trained text feature extraction model) is shown in Figure 2 . Wherein, the first preset model includes an encoder Encoder and a decoder Decoder. Wherein, the encoder Encoder includes a text encoder Text Encoder and a token encoder TokenEncoder (the encoder can be the encoder in the T5 model), wherein the Text Encoder is used to encode the ID (i.e., text ID) of the natural language description corresponding to the audio sample, and this encoding can be regarded as a primary mapping; then, the output content of the TextEncoder is input into the Token Encoder for encoding, and this encoding can be regarded as a secondary mapping, and the final output encoding content can represent the semantic information TextEmbedding of each word in the natural language description corresponding to the audio sample. A multi-task and multi-classification head is created in the decoder Decoder, each head consists of a fully connected layer fc and a classification layer softmax, and finally outputs the predicted probability X′ corresponding to each audio attribute. attr_probability .
[0085] (3) Using a first loss function, the first preset model is trained to obtain a trained text feature extraction model; wherein the first loss function is determined by a predicted probability value corresponding to each audio attribute related to the audio sample and an attribute label corresponding to the audio sample.
[0086] Here, if the data is encoded in a non-one-hot token format, the first loss function can be a sparse cross entropy loss function. The first loss function is used to supervise the model training. attr as follows:
[0087]
[0088] Among them, X′ attr_probability is the predicted attribute probability, that is, the predicted probability value corresponding to the audio attribute related to the audio sample; is the annotated label attribute value, that is, the attribute annotation corresponding to the audio sample.
[0089] Here, using the first loss function to train the first preset model specifically includes: using the loss value calculated by the first loss function to adjust the network parameters of the first preset model (i.e., adjusting the parameters); then updating the first preset model to the first preset model after parameter adjustment, and then, using the data in the training set to train the first preset model after parameter adjustment, and then again using the loss value calculated by the first loss function to adjust the parameters of the model until the model is stable (such as the loss value no longer changes or the number of training times reaches the preset number of times, etc.), thereby obtaining a trained text feature extraction model.
[0090] It should be noted that when training the model, hyperparameters such as the learning rate are appropriately selected and adjusted based on the specific collected data.
[0091] In an optional embodiment, the above step 103, using the audio attribute description text to generate audio data related to the text corresponding to the natural language of the user, includes:
[0092] ① uniformly encoding and mapping the audio attribute description text to obtain a second encoding content;
[0093] Here, the audio attribute description text is uniformly encoded and mapped to a uniform space, and the obtained second encoding content is a uniform latent space vector of the audio content to be generated.
[0094] ② Decoding the second encoded content to obtain a Mel spectrum;
[0095] Here, decoding the second encoded content can restore it to the Mel spectrum.
[0096] ③ Based on the Mel-spectrogram, generate audio data related to the text corresponding to the natural language of the user.
[0097] In this way, the audio content requirements to be generated input by the user are unified at the encoding level, and different types of audio content can be adaptively generated from the unified input encoding.
[0098] The processing logic for generating audio data can be implemented by an audio generation model. In an optional embodiment, the step 103, using the audio attribute description text to generate audio data related to the text corresponding to the natural language of the user, includes:
[0099] The audio attribute description text is input into an audio generation model to obtain audio data related to the text corresponding to the natural language of the user.
[0100] Here, the audio attribute description text obtained in step 102 is input into the audio generation model to obtain audio data related to the text corresponding to the natural language of the user.
[0101] The audio generation model uses the Encoder-Decoder architecture, see Figure 3 . The audio attribute description text of the input by Encoder The data is uniformly encoded and mapped to a uniform space. The Decoder decodes the data and restores it to a Mel spectrum. The Vocoder then restores the Mel spectrum to audio data.
[0102] Here, the audio generation model is obtained by pre-training, and the specific training process can be implemented by the following embodiments. In an optional embodiment, the method of the present invention also includes:
[0103] a1, obtaining a training set, wherein each training data in the training set includes an audio sample, a natural language description corresponding to the audio sample, and an attribute annotation;
[0104] The implementation process of obtaining the training set here can be found in the explanation of the training part of the text feature extraction model, which will not be repeated here.
[0105] a2, inputting the attribute annotation corresponding to the audio sample into a second preset model to obtain predicted audio data;
[0106] During training, accept the TrainData Attribute annotations corresponding to the audio samples in the collection Get the predicted audio data Y a ' udio .
[0107] a3. Using a second loss function, train the second preset model to obtain a trained audio generation model; wherein the second loss function is determined by the predicted audio data and the audio sample.
[0108] Here, the second loss function principle is based on the predicted audio data Y a ' udioand the true value, that is, the audio sample Y audio Make loss constraints and supervise the model training based on the second loss function. The specific steps of training the second preset model using the second loss function are as follows: using the loss value calculated by the second loss function to adjust the network parameters of the second preset model (i.e., adjust the parameters); then update the second preset model to the second preset model after parameter adjustment, and then use the data in the training set to train the second preset model after parameter adjustment, and then use the loss value calculated by the second loss function to adjust the model parameters again until the model is stable (such as the loss value no longer changes or the number of training times reaches the preset number, etc.), and obtain a trained audio generation model.
[0109] The audio generation method of the present invention generates diverse (different types) audio content based on the user's text description, and the generated audio content includes but is not limited to audio content such as voice, sound effects, and music, so that the user can use natural language to describe the end-to-end generation of audio content with a unified theme. For example, in one example, the user input is as follows:
[0110] "Generate an audio clip of a romantic date between a man and a woman under the moonlight. The first two minutes are a conversation between the man and woman sharing their daily fun, accompanied by soothing piano music. The last minute is a conversation between the man and woman imagining a beautiful future, accompanied by brisk string music, and the whole process is accompanied by bird calls and cicada chirps."
[0111] The generated content is: a 3-minute audio file that meets the above description, including original voice, sound effects, and background music.
[0112] The method of the present invention can help generate intelligent audio content, allowing users to produce copyright-exclusive and diverse audio content with a low threshold, which can cover a large number of potential application areas such as audio drama production, educational audio production, product display audio production, etc. The overall implementation process can be seen in Figure 4 .
[0113] Finally, the product interaction diagram formed by the method of the present invention can be seen in Figure 5 .
[0114] like Figure 6 As shown, an embodiment of the present invention further provides an audio generation device, wherein the device includes:
[0115] The first acquisition module 601 is used to acquire the text corresponding to the user's natural language;
[0116] A first processing module 602 is used to perform attribute prediction based on the text corresponding to the natural language of the user, and obtain an audio attribute description text with a unified text configuration format, wherein the audio attribute description text is used to represent the audio information prompted by the text corresponding to the natural language of the user;
[0117] The audio generation module 603 is used to generate audio data related to the text corresponding to the natural language of the user by using the audio attribute description text.
[0118] In some embodiments, the first processing module 602 includes:
[0119] A first encoding unit, used to encode the text corresponding to the natural language of the user to obtain a first encoded content;
[0120] A first decoding unit, configured to decode the first encoded content based on a pre-created multi-task multi-classification header to obtain a prediction probability value corresponding to each audio attribute, wherein each header has a corresponding audio attribute;
[0121] The first processing unit is used to perform conversion processing based on the predicted probability value corresponding to each audio attribute to obtain an audio attribute description text with a unified text configuration format.
[0122] In some embodiments, the first processing module 602 includes:
[0123] A second processing unit is used to input the text corresponding to the natural language of the user into a text feature extraction model to obtain an audio attribute description text with a unified text configuration format;
[0124] The text feature extraction model is used to perform attribute prediction based on the text corresponding to the user's natural language, and obtain an audio attribute description text with a unified text configuration format.
[0125] In some embodiments, the apparatus of the present invention further comprises:
[0126] A second acquisition module is used to acquire a training set, wherein each training data in the training set includes an audio sample, a natural language description corresponding to the audio sample, and an attribute label;
[0127] The second processing module is used to input the natural language description corresponding to the audio sample into the first preset model to obtain the predicted probability value corresponding to each audio attribute related to the audio sample; wherein the first preset model includes an encoder and a decoder, the encoder is used to tokenize the natural language description corresponding to the audio sample; the decoder includes a pre-created multi-task multi-classification head, each head includes a fully connected layer and a classification layer, and is used to classify the encoded content output by the encoder to obtain the predicted probability value of the corresponding audio attribute;
[0128] The first model training module is used to train the first preset model using a first loss function to obtain a trained text feature extraction model; wherein the first loss function is determined by the predicted probability value corresponding to each audio attribute related to the audio sample and the attribute label corresponding to the audio sample.
[0129] In some embodiments, the audio generation module 603 includes:
[0130] A second encoding unit, used for uniformly encoding and mapping the audio attribute description text to obtain a second encoded content;
[0131] A second decoding unit, used for decoding the second coded content to obtain a Mel spectrum;
[0132] The first audio generation unit is used to generate audio data related to the text corresponding to the natural language of the user according to the Mel spectrum.
[0133] In some embodiments, the audio generation module 603 includes:
[0134] The second audio generation unit is used to input the audio attribute description text into the audio generation model to obtain audio data related to the text corresponding to the natural language of the user.
[0135] In some embodiments, the apparatus of the present invention further comprises:
[0136] A third acquisition module is used to acquire a training set, where each training data in the training set includes an audio sample, a natural language description corresponding to the audio sample, and an attribute annotation;
[0137] A third processing module, used for inputting the attribute annotation corresponding to the audio sample into a second preset model to obtain predicted audio data;
[0138] The second model training module is used to train the second preset model using a second loss function to obtain a trained audio generation model; wherein the second loss function is determined by the predicted audio data and the audio sample.
[0139] Among them, the implementation embodiments of the above-mentioned audio generation method are all applicable to the embodiments of the audio generation device and can achieve the same technical effects.
[0140] The embodiment of the present invention further provides an audio generation device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; when the processor executes the program, the following is achieved: Figure 1 The audio generation method shown.
[0141] Among them, the implementation embodiments of the above-mentioned audio generation method are all applicable to the embodiments of the audio generation device and can achieve the same technical effects.
[0142] The embodiment of the present invention further provides a readable storage medium on which a program is stored. When the program is executed by a processor, the steps in the above-mentioned audio generation method are implemented.
[0143] Among them, the implementation embodiments of the above-mentioned audio generation method are all applicable to the embodiments of the readable storage medium and can achieve the same technical effect.
[0144] The embodiment of the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the following Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.
[0145] It should be noted that many functional components described in this specification are referred to as modules in order to more particularly emphasize the independence of their implementation methods.
[0146] In the embodiment of the present invention, module can be implemented with software so that it can be executed by various types of processors. For example, an executable code module of an identification can include one or more physical or logical blocks of computer instructions, for example, it can be constructed as an object, process or function. Nevertheless, the executable code of the identified module does not need to be physically located together, but can include different instructions stored in different positions, and when these instructions are logically combined together, it constitutes a module and realizes the specified purpose of the module.
[0147] In fact, executable code module can be a single instruction or many instructions, and can even be distributed on a plurality of different code segments, distributed among different programs, and distributed across a plurality of memory devices. Similarly, operating data can be identified in the module, and can be implemented and organized in the data structure of any appropriate type according to any appropriate form. The operating data can be collected as a single data set, or can be distributed in different locations (including on different storage devices), and can only be present on a system or network as an electronic signal at least in part.
[0148] When a module can be implemented by software, considering the level of existing hardware technology, a person skilled in the art can build a corresponding hardware circuit to implement the corresponding function of the module that can be implemented by software without considering the cost. The hardware circuit includes a conventional very large scale integration (VLSI) circuit or gate array and existing semiconductors such as logic chips, transistors, or other discrete components. The module can also be implemented by a programmable hardware device, such as a field programmable gate array, a programmable array logic, a programmable logic device, etc.
[0149] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary personnel in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. An audio generation method, characterized in that: include: Get the text corresponding to the user's natural language; Performing attribute prediction based on the text corresponding to the natural language of the user to obtain an audio attribute description text with a unified text configuration format, wherein the audio attribute description text is used to represent the audio information prompted by the text corresponding to the natural language of the user; The audio attribute description text is used to generate audio data related to the text corresponding to the natural language of the user.
2. The method according to claim 1, characterized in that: The method of performing attribute prediction based on the text corresponding to the natural language of the user to obtain an audio attribute description text with a unified text configuration format includes: Encoding the text corresponding to the natural language of the user to obtain first encoded content; Based on the pre-created multi-task multi-classification header, the first encoded content is decoded to obtain a prediction probability value corresponding to each audio attribute, wherein each header has a corresponding audio attribute; The predicted probability value corresponding to each audio attribute is converted and processed to obtain an audio attribute description text with a unified text configuration format.
3. The method according to claim 1, characterized in that The method of performing attribute prediction based on the text corresponding to the natural language of the user to obtain an audio attribute description text with a unified text configuration format includes: Inputting the text corresponding to the user's natural language into a text feature extraction model to obtain an audio attribute description text with a unified text configuration format; The text feature extraction model is used to perform attribute prediction based on the text corresponding to the user's natural language, and obtain an audio attribute description text with a unified text configuration format.
4. The method according to claim 3, characterized in that The method further comprises: Obtaining a training set, wherein each training data in the training set includes an audio sample, a natural language description corresponding to the audio sample, and an attribute annotation; Inputting the natural language description corresponding to the audio sample into the first preset model to obtain the predicted probability value corresponding to each audio attribute related to the audio sample; wherein the first preset model includes an encoder and a decoder, the encoder is used to tokenize the natural language description corresponding to the audio sample; the decoder includes a pre-created multi-task multi-classification head, each head includes a fully connected layer and a classification layer, and is used to classify the encoded content output by the encoder to obtain the predicted probability value of the corresponding audio attribute; The first preset model is trained using a first loss function to obtain a trained text feature extraction model; wherein the first loss function is determined by the predicted probability value corresponding to each audio attribute related to the audio sample and the attribute label corresponding to the audio sample.
5. The method according to claim 1, characterized in that The step of using the audio attribute description text to generate audio data related to the text corresponding to the natural language of the user includes: Performing unified encoding and mapping on the audio attribute description text to obtain a second encoded content; Decoding the second encoded content to obtain a Mel spectrum; Audio data related to text corresponding to the natural language of the user is generated according to the Mel-spectrogram.
6. The method according to claim 1, characterized in that The step of using the audio attribute description text to generate audio data related to the text corresponding to the natural language of the user includes: The audio attribute description text is input into an audio generation model to obtain audio data related to the text corresponding to the natural language of the user.
7. The method according to claim 6, characterized in that The method further comprises: Obtaining a training set, wherein each training data in the training set includes an audio sample, a natural language description corresponding to the audio sample, and an attribute annotation; Inputting the attribute annotation corresponding to the audio sample into a second preset model to obtain predicted audio data; The second preset model is trained using a second loss function to obtain a trained audio generation model; wherein the second loss function is determined by the predicted audio data and the audio sample.
8. An audio generating device, characterized in that: include: A first acquisition module is used to acquire text corresponding to the user's natural language; A first processing module, configured to perform attribute prediction based on the text corresponding to the natural language of the user, and obtain an audio attribute description text with a unified text configuration format, wherein the audio attribute description text is used to represent the audio information prompted by the text corresponding to the natural language of the user; The audio generation module is used to use the audio attribute description text to generate audio data related to the text corresponding to the natural language of the user.
9. An audio generating device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that: When the processor executes the program, the audio generation method according to any one of claims 1 to 7 is implemented.
10. A readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the audio generation method according to any one of claims 1 to 7 are implemented.
11. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps in the audio generation method according to any one of claims 1 to 7.
