A method and system for generating audio from text

Abstract

The present invention provides a method and system for generating audio from text. The method comprises the steps of diffusion model training and audio inference. In the diffusion model training step, Mel conversion is performed on training audio data to obtain a first Mel spectrum, the first Mel spectrum is constructed as a first Mel image, and the first Mel image is encoded into an image feature vector through an image encoder; training text data corresponding to the training audio data is obtained, and the training text data is encoded into a training text feature vector through a text encoder; the image feature vector, the training text feature vector and a preset training noise vector are input into a diffusion model to train the diffusion model; in the audio inference step, a text feature vector to be generated is obtained based on the text to be generated, the text feature vector to be generated and a preset Gaussian noise vector are input into the diffusion model to obtain a target image feature vector, and a target audio is obtained based on the target image feature vector.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a method and system for generating audio from text. Background Art

[0002] Text-to-Audio (TTA) generation is an emerging application that aims to synthesize diverse audio outputs based on textual prompts. With the integration of artificial intelligence in the field of generative AI (AIGC), the scope of TTA applications has significantly expanded to cover areas such as film dubbing and music composition.

[0003] Early TTA models mainly relied on a single label, resulting in monotonous audio generation, which was limited by the limited label space and generation capacity. In contrast, natural descriptive text provides more comprehensive and detailed information. Therefore, subsequent work has developed models based on text content. Some studies use the diffusion model framework to generate audio from text, while other models use autoregressive models based on the transformer decoder structure.

[0004] In the process of synthesizing audio through text in the prior art, it is usually necessary to first convert the reference audio into a Mel spectrum, obtain the Mel implicit features through a Mel encoder, and then generate the synthesized audio by combining the Mel implicit features and the text. Although the Mel spectrum is the data for comparing the original audio, the Mel spectrum does not further process the original audio data, resulting in poor final synthesized audio effect. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method for generating audio from text to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present invention provides a method for generating audio from text, the method comprising the steps of diffusion model training and audio reasoning;

[0007] In the diffusion model training step, training audio data is obtained, Mel conversion is performed on the training audio data to obtain a first Mel spectrum, the first Mel spectrum is constructed into a first Mel image, and the first Mel image is encoded into an image feature vector through an image encoder;

[0008] Acquire training text data corresponding to the training audio data, and encode the training text data into a training text feature vector through a text encoder;

[0009] Inputting the image feature vector, the training text feature vector and the preset training noise vector into the diffusion model to train the diffusion model;

[0010] In the audio inference step, a feature vector of the text to be generated is obtained based on the text to be generated, the feature vector of the text to be generated and a preset Gaussian noise vector are input into a diffusion model to obtain a target image feature vector, and the target audio is obtained based on the target image feature vector.

[0011] The above scheme is adopted. This scheme is different from the method of encoding the Mel spectrum adopted in the prior art. Based on the Mel spectrum, this scheme further constructs the first Mel spectrum into a first Mel image, and then encodes it into an image feature vector through an image encoder. The image encoder can encode based on each pixel in the image. Compared with the original Mel spectrum encoding method, the image feature vector in this scheme can combine more data to obtain an image feature vector combining more audio features, thereby making the final synthesized audio effect better.

[0012] In some embodiments of the present invention, in the step of obtaining the target audio based on the target image feature vector, the target image feature vector is decoded by an image decoder to obtain a corresponding target Mel image, the target Mel image is constructed into a target Mel spectrum, and the target Mel spectrum is constructed into a target audio through a vocoder.

[0013] In some embodiments of the present invention, in the step of constructing the target Mel image as a target Mel spectrum, the target Mel image is compressed into a single-channel image to obtain a target Mel spectrum corresponding to the single-channel target Mel image.

[0014] In some embodiments of the present invention, the step of obtaining a target mel spectrum corresponding to a single-channel target mel image further includes performing inverse normalization on each spectral point in the target mel spectrum corresponding to the single-channel target mel image, and combining each inverse normalized spectral point to obtain a target mel spectrum after inverse normalization.

[0015] In some embodiments of the present invention, the step of constructing the first Mel-spectrogram into a first Mel-image includes:

[0016] The first Mel-spectrum correspondence is constructed as an initial Mel-image, where the initial Mel-image is a single-channel image;

[0017] The channels of the initial Mel image are copied to obtain a multi-channel first Mel image.

[0018] In some embodiments of the present invention, the diffusion model includes a plurality of diffusion networks arranged in sequence. In the step of inputting an image feature vector, a training text feature vector and a preset training noise vector into the diffusion model and training the diffusion model, the image feature vector is subjected to a plurality of noise fusion steps to add a plurality of training noise vectors to the image feature vector so that the image feature vector is fused with the training noise vector. The fused image feature vector and the training text feature vector are input into the diffusion model. Each diffusion network of the diffusion model outputs a predicted training noise vector respectively. A loss function is constructed based on the predicted training noise vector and the actual training noise vector to train the diffusion model.

[0019] In some embodiments of the present invention, in the step of constructing a loss function based on the predicted training noise vector and the actual training noise vector, the loss function value is calculated using the following formula:

[0020]

[0021] Among them, l θ Represents the loss function value, ∈ θ represents the predicted training noise vector, and ∈ represents the actual training noise vector.

[0022] In some embodiments of the present invention, the step of constructing the first Mel spectrum as a first Mel image includes standardizing the first Mel spectrum. In the step of standardizing the first Mel spectrum, the first Mel spectrum includes multiple spectrum points, and the values ​​of the spectrum points are standardized to obtain a standardized value of each spectrum point. The standardized values ​​of all spectrum points are combined to obtain the first Mel spectrum after standardization.

[0023] In some embodiments of the present invention, in the step of inputting the text feature vector to be generated and a preset Gaussian noise vector into a diffusion model to obtain a target image feature vector, each diffusion network of the diffusion model outputs a predicted noise vector, and the preset Gaussian noise vector is subtracted from the predicted noise vector to obtain the target image feature vector.

[0024] In some embodiments of the present invention, the diffusion model training step further includes performing data enhancement on the first Mel image of the training audio data and the training text data;

[0025] In the step of performing data enhancement on the first Mel image of the training audio data, each first Mel image of the training data is stretched based on time, and is cropped after stretching, and the cropped first Mel images are randomly spliced ​​or randomly weighted averaged to obtain the first Mel image after data enhancement;

[0026] In the step of performing data enhancement on the training text data, verbs or nouns are randomly replaced in the training text data, and the word order of the training text data is randomly disrupted, and then randomly spliced ​​to obtain the data-enhanced training text data.

[0027] The second aspect of the present invention also provides a text-generated audio system, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0028] The third aspect of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps implemented by the aforementioned method for generating audio from text.

[0029] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention can be specifically pointed out and obtained in the specification and the accompanying drawings.

[0030] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0032] Figure 1 A schematic diagram of an implementation of a method for generating audio from text according to the present invention;

[0033] Figure 2 This is a schematic diagram of the implementation architecture of audio reasoning in this solution;

[0034] Figure 3 Schematic diagram of the conversion process of the spectrum image converter;

[0035] Figure 4 This is a schematic diagram of the processing of Mel image enhancement in the present invention;

[0036] Figure 5 This is a schematic diagram of the process of text enhancement in the present invention;

[0037] Figure 6 This is a schematic diagram of the implementation architecture of the diffusion model training for this scheme. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0039] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0040] Introduction to existing technology:

[0041] The existing technology includes the following parts: Mel encoder, Mel decoder, diffusion model, vocoder, audio encoder and text encoder. During training, confirm that text and audio are used as input, and the audio is used as input condition and input into the audio encoder to obtain audio features; then the audio is converted into a Mel spectrum and passed through the Mel encoder to obtain Mel implicit features; the Mel implicit features, audio features and Gaussian noise are added as input and sent to the diffusion model, and then the diffusion model needs to predict Gaussian noise. During reasoning, take the text as input and input into the text encoder to obtain text features; then add Gaussian noise and text features and send them to the diffusion model, continuously predict Gaussian noise from complete Gaussian noise sampling, gradually restore the samples, and obtain Mel implicit features; then send the Mel implicit features to the Mel decoder to obtain the Mel spectrum, and then send it to the pre-trained vocoder to obtain audio.

[0042] Existing technologies require a lot of computing resources to train from scratch, including the Mel encoder, Mel decoder, and diffusion model. These modules all need to be trained from scratch and require a large amount of data sets to get results.

[0043] In order to solve the problems described above, the method of the present invention uses a transfer learning method to migrate the pre-trained text generation model to the text generation speech model, and can achieve good results using a small amount of data sets and computing resources.

[0044] The steps of the present invention specifically include:

[0045] like Figure 1 As shown, the present invention proposes a method for generating audio from text, the steps of the method including diffusion model training and audio reasoning;

[0046] like Figure 6As shown, the diffusion model training step includes: step S100, acquiring training audio data, performing Mel conversion on the training audio data to obtain a first Mel spectrum, constructing the first Mel spectrum into a first Mel image, and encoding the first Mel image into an image feature vector through an image encoder;

[0047] In a specific implementation process, the training audio data is pre-produced and recorded training audio data.

[0048] In the specific implementation process, the Mel conversion is a scale that simulates the human ear's perception of audio frequencies, dividing the frequency axis into several intervals of equal width. In the low frequency band, the Mel scale is linearly related to the frequency; while in the high frequency band, the Mel scale is logarithmically related to the frequency. This division method is more in line with the human ear's perception of audio frequencies. In practical applications, the Mel scale can be converted into the corresponding frequency or the frequency can be converted into the Mel scale.

[0049] In a specific implementation process, the first Mel spectrum is a spectrum diagram obtained by performing Mel conversion on the training audio data.

[0050] like Figure 3 As shown, in a specific implementation process, the first Mel spectrum is constructed into a first Mel image through a spectrum image converter.

[0051] Step S200, obtaining training text data corresponding to the training audio data, and encoding the training text data into a training text feature vector through a text encoder;

[0052] In the specific implementation process, the image encoder and the text encoder in this solution are both preset encoders for encoding images and texts.

[0053] In a specific implementation process, the training text data is the text corresponding to the training audio data.

[0054] Step S300, inputting the image feature vector, the training text feature vector and the preset training noise vector into the diffusion model to train the diffusion model;

[0055] In the specific implementation process, the diffusion model includes a UNet structure and an attention structure. The input features are integrated into the diffusion model through the Q, K, and V mechanisms in the attention structure.

[0056] like Figure 2 As shown, the audio inference steps include: step S400, obtaining a feature vector of the text to be generated based on the text to be generated, inputting the feature vector of the text to be generated and a preset Gaussian noise vector into a diffusion model to obtain a target image feature vector, and obtaining a target audio based on the target image feature vector.

[0057] The above scheme is adopted. This scheme is different from the method of encoding the Mel spectrum adopted in the prior art. Based on the Mel spectrum, this scheme further constructs the first Mel spectrum into a first Mel image, and then encodes it into an image feature vector through an image encoder. The image encoder can encode based on each pixel in the image. Compared with the original Mel spectrum encoding method, the image feature vector in this scheme can combine more data to obtain an image feature vector combining more audio features, thereby making the final synthesized audio effect better.

[0058] In some embodiments of the present invention, in the step of obtaining the target audio based on the target image feature vector, the target image feature vector is decoded by an image decoder to obtain a corresponding target Mel image, the target Mel image is constructed into a target Mel spectrum, and the target Mel spectrum is constructed into a target audio through a vocoder.

[0059] like Figure 3 As shown, in a specific implementation process, the target Mel image is constructed into a target Mel spectrum through a spectrum image converter.

[0060] In the specific implementation process, the Mel spectrum is a scale that simulates the human ear's perception of audio frequencies and is used to divide the frequency axis of the audio signal nonlinearly. In the Mel spectrum, the frequency axis is divided into several intervals of equal width, which are called the Mel scale. Unlike the traditional linear spectrum, the low frequency band of the Mel spectrum has a linear relationship with the frequency, while the high frequency band has a logarithmic relationship with the frequency. This division method is more in line with the human ear's perception of audio frequencies, and therefore can better simulate the human ear's perception of audio signals.

[0061] In a specific implementation process, a vocoder is a speech codec that simulates a human voice signal and is used to analyze and synthesize audio data of a human voice signal.

[0062] In some embodiments of the present invention, in the step of constructing the target Mel image as a target Mel spectrum, the target Mel image is compressed into a single-channel image to obtain a target Mel spectrum corresponding to the single-channel target Mel image.

[0063] In a specific implementation process, corresponding to the step of generating the first Mel image, the target Mel image is compressed into a single-channel image.

[0064] In some embodiments of the present invention, the step of obtaining a target mel spectrum corresponding to a single-channel target mel image further includes performing inverse normalization on each spectral point in the target mel spectrum corresponding to the single-channel target mel image, and combining each inverse normalized spectral point to obtain a target mel spectrum after inverse normalization.

[0065] In the specific implementation process, in the step of performing inverse normalization on each spectral point in the target Mel spectrum corresponding to the single-channel target Mel image, corresponding to the step of performing normalization on the values ​​of the spectral points, if the spectral points with a numerical range of [-12, 5] are standardized to standardized values ​​in the numerical range of [-1, 1] during the normalization process, then in the step of inverse normalization, the standardized values ​​of the spectral points with a numerical range of [-1, 1] are inversely standardized within the numerical range of [-12, 5].

[0066] In some embodiments of the present invention, the step of constructing the first Mel-spectrogram into a first Mel-image includes:

[0067] The first Mel-spectrum correspondence is constructed as an initial Mel-image, where the initial Mel-image is a single-channel image;

[0068] The channels of the initial Mel image are copied to obtain a multi-channel first Mel image.

[0069] Using the above scheme, this scheme replicates the single-channel image channel by channel, so that the final first Mel image can carry more features and improve the accuracy of audio synthesis.

[0070] In some embodiments of the present invention, the diffusion model includes a plurality of diffusion networks arranged in sequence. In the step of inputting an image feature vector, a training text feature vector and a preset training noise vector into the diffusion model and training the diffusion model, the image feature vector is subjected to a plurality of noise fusion steps to add a plurality of training noise vectors to the image feature vector so that the image feature vector is fused with the training noise vector. The fused image feature vector and the training text feature vector are input into the diffusion model. Each diffusion network of the diffusion model outputs a predicted training noise vector respectively. A loss function is constructed based on the predicted training noise vector and the actual training noise vector to train the diffusion model.

[0071] In some embodiments of the present invention, in the step of constructing a loss function based on the predicted training noise vector and the actual training noise vector, the loss function value is calculated using the following formula:

[0072]

[0073] Among them, l θ Represents the loss function value, ∈ θ represents the predicted training noise vector, and ∈ represents the actual training noise vector.

[0074] In some embodiments of the present invention, the step of constructing the first Mel spectrum as a first Mel image includes standardizing the first Mel spectrum. In the step of standardizing the first Mel spectrum, the first Mel spectrum includes multiple spectrum points, and the values ​​of the spectrum points are standardized to obtain a standardized value of each spectrum point. The standardized values ​​of all spectrum points are combined to obtain the first Mel spectrum after standardization.

[0075] In some embodiments of the present invention, in the step of normalizing the values ​​of the spectrum points, the value range of the original spectrum points is [-12, 5], and the spectrum points in the value range [-12, 5] are normalized to standardized values ​​in the value range [-1, 1].

[0076] In some implementations of the present invention, in the step of normalizing the values ​​of the spectrum points, the following formula is used for normalization:

[0077] δ = (i-mean) / var;

[0078] Among them, δ represents the normalized value after normalization, i represents the value of the spectrum point before normalization, and mean and var are both preset calculation parameters.

[0079] In some embodiments of the present invention, if in the step of normalizing the spectrum points in the value range of [-12, 5] to normalized values ​​in the value range of [-1, 1], mean is -5.5 and var is 6.5.

[0080] In some embodiments of the present invention, in the step of inputting the text feature vector to be generated and a preset Gaussian noise vector into a diffusion model to obtain a target image feature vector, each diffusion network of the diffusion model outputs a predicted noise vector, and the preset Gaussian noise vector is subtracted from the predicted noise vector to obtain the target image feature vector.

[0081] like Figure 4 and 5 As shown, in some embodiments of the present invention, the diffusion model training step further includes performing data enhancement on the first Mel image of the training audio data and the training text data;

[0082] In the step of performing data enhancement on the first Mel image of the training audio data, each first Mel image of the training data is stretched based on time, and is cropped after stretching, and the cropped first Mel images are randomly spliced ​​or randomly weighted averaged to obtain the first Mel image after data enhancement;

[0083] In the step of performing data enhancement on the training text data, verbs or nouns are randomly replaced in the training text data, and the word order of the training text data is randomly disrupted, and then randomly spliced ​​to obtain the data-enhanced training text data.

[0084] In the specific implementation process, in the step of stretching each Mel image of the training data based on time, the original Mel image with a length of 10s can be stretched to 20s, and then randomly cropped, so that the Mel image with a length of 20s can be stretched and cropped into multiple segments, specifically, it can be cropped into 4 Mel images with a length of 5s.

[0085] In the specific implementation process, in the step of randomly splicing or randomly weighted averaging the cropped Mel images to obtain the data enhanced Mel images, the number of Mel images of the training data can be increased by randomly splicing or randomly weighted averaging the originally different Mel images after stretching and cropping, thereby completing the data enhancement of the Mel images.

[0086] In the specific implementation process, in the step of randomly replacing verbs or nouns in the text, randomly disrupting the word order of the text, and then randomly splicing to obtain the data enhanced text, the verbs or nouns in the text of the training data are replaced, the word order of the text is randomly disrupted, different texts of the training data are randomly spliced, the number of texts in the training data is increased, and the data enhancement of the text is completed.

[0087] An embodiment of the present invention also provides a text-generated audio system, which includes a computer device, wherein the computer device includes a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0088] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps implemented by the aforementioned method for generating audio from text are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0089] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0090] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0091] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating audio from text, characterized in that: The steps of the method include diffusion model training and audio inference; In the diffusion model training step, training audio data is obtained, Mel conversion is performed on the training audio data to obtain a first Mel spectrum, the first Mel spectrum is constructed as a first Mel image, the first Mel spectrum is correspondingly constructed as an initial Mel image, and the initial Mel image is a single-channel image; the channel of the initial Mel image is copied to obtain a multi-channel first Mel image, and the first Mel image is encoded into an image feature vector through an image encoder; Acquire training text data corresponding to the training audio data, and encode the training text data into a training text feature vector through a text encoder; Inputting the image feature vector, the training text feature vector and the preset training noise vector into the diffusion model to train the diffusion model; In the audio inference step, a feature vector of the text to be generated is obtained based on the text to be generated, the feature vector of the text to be generated and a preset Gaussian noise vector are input into a diffusion model to obtain a target image feature vector, a target audio is obtained based on the target image feature vector, the target image feature vector is decoded by an image decoder to obtain a corresponding target Mel image, the target Mel image is constructed as a target Mel spectrum, the target Mel image is compressed into a single-channel image, a target Mel spectrum corresponding to the single-channel target Mel image is obtained, the target Mel spectrum is constructed as a target audio through a vocoder, each diffusion network of the diffusion model outputs a predicted noise vector, the preset Gaussian noise vector is subtracted from the predicted noise vector, and the target image feature vector is obtained.

2. The method for generating audio from text according to claim 1, characterized in that: The step of obtaining a target Mel spectrum corresponding to the single-channel target Mel image also includes performing inverse normalization processing on each spectrum point in the target Mel spectrum corresponding to the single-channel target Mel image, and combining each inverse normalized spectrum point to obtain a target Mel spectrum after inverse normalization processing.

3. The method for generating audio from text according to claim 1, characterized in that: The diffusion model includes a plurality of diffusion networks arranged in sequence. In the step of inputting an image feature vector, a training text feature vector and a preset training noise vector into the diffusion model and training the diffusion model, the image feature vector fuses a plurality of training noise vectors through a plurality of noise fusion steps, and the fused image feature vector and the training text feature vector are input into the diffusion model. Each diffusion network of the diffusion model outputs a predicted training noise vector respectively, and a loss function is constructed based on the predicted training noise vector and the actual training noise vector to train the diffusion model.

4. The method for generating audio from text according to claim 3, characterized in that: In the step of constructing the loss function based on the predicted training noise vector and the actual training noise vector, the loss function value is calculated using the following formula: Among them, l θ Represents the loss function value, ∈ θ represents the predicted training noise vector, and ∈ represents the actual training noise vector.

5. The method for generating audio from text according to claim 1, characterized in that: The step of constructing the first Mel spectrum into a first Mel image includes standardizing the first Mel spectrum. In the step of standardizing the first Mel spectrum, the first Mel spectrum includes a plurality of spectrum points, and the values ​​of the spectrum points are standardized to obtain a standardized value of each spectrum point. The standardized values ​​of all spectrum points are combined to obtain the first Mel spectrum after standardization.

6. A text-generated audio system, characterized in that: The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions. The processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method as described in any one of claims 1 to 5.

Images