Audio generation method, device, and computer-readable medium
By extracting entity information from the input text and generating spectral information by combining Gaussian distribution and cross-attention mechanism, the problem of content monotony and consistency in existing audio synthesis technology is solved, and random variation and quality assurance of audio content are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI MIDU INFORMATION TECH CO LTD
- Filing Date
- 2023-07-14
- Publication Date
- 2026-07-14
AI Technical Summary
Existing audio synthesis technologies produce monotonous content, have poor results, and generate identical audio content with the same input text each time.
Entity information is extracted from the input text, and features are extracted using BERT, LSTM, and CRF models. Gaussian distribution sampling and cross-attention mechanisms are combined to generate spectral information through UNet and VAE models. Finally, the audio signal is restored by the HiFiGAN model.
It enables random variations in audio content, avoiding the problem of generating completely identical content each time, while controlling the variations within a reasonable range to ensure audio quality.
Smart Images
Figure CN116778906B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of information technology, and in particular to an audio generation method, device and computer-readable medium. Background Technology
[0002] With the development of artificial intelligence, audio synthesis technology will be used in more and more scenarios. In the current conventional audio synthesis scheme, users need to input a text segment, and then an artificial intelligence model generates a corresponding audio segment based on the input text. However, the current scheme has the following problems: most of them can only synthesize human speech audio based on text, and the effect is not good when synthesizing audio of other content. In addition, the audio content generated by the same input text is consistent every time, without any change. Summary of the Invention
[0003] One objective of this application is to provide an audio generation method, device, and computer-readable medium to address the problems of limited content and poor synthesis quality in existing solutions.
[0004] To achieve the above objectives, embodiments of this application provide an audio generation method, the method comprising:
[0005] Extract entity information from the input text to describe the audio;
[0006] Feature extraction is performed on entity information to obtain text entity features;
[0007] Sampling is performed based on a preset Gaussian distribution to obtain Gaussian features;
[0008] The first feature is obtained by summing the text entity features and the Gaussian features.
[0009] Based on the cross-attention mechanism, the first feature is optimized to obtain the second feature;
[0010] Spectral information is generated based on the second feature, and the spectral information is then converted back into an audio signal.
[0011] Furthermore, the entity information includes nouns, adjectives, and verbs in the input text.
[0012] Furthermore, entity information describing the audio is extracted from the input text, including:
[0013] BERT, LSTM, and CRF models are used to extract entity information describing audio from input text.
[0014] Furthermore, feature extraction is performed on the entity information to obtain text entity features, including:
[0015] The BERT model is used to extract features from entity information and obtain text entity features.
[0016] Furthermore, based on the cross-attention mechanism, the first feature is optimized to obtain the second feature, including:
[0017] The first feature is used as the V vector, the Gaussian feature as the K vector, and the text entity feature as the Q vector. These are then input into the cross-attention model for computation to obtain the third feature.
[0018] The third feature is denoised using the UNet model to obtain the second feature.
[0019] Furthermore, both the encoder and decoder modules of the UNet model are composed of multi-layer cross-attention models;
[0020] The third feature is denoised using the UNet model to obtain the second feature, including:
[0021] The text entity features are used as V vectors, and the third features are used as Q and K vectors, which are then input into the encoder module of the UNet model to obtain intermediate features.
[0022] The text entity features are used as V vectors, and the intermediate features are used as Q and K vectors, which are then input into the decoder module of the UNet model to obtain noise features.
[0023] The fourth feature is obtained by calculating the difference between the third feature and the noise feature.
[0024] Determine whether the preset number of executions has been reached. If the preset number of executions has not been reached, set the fourth feature as the new third feature and re-execute the denoising process. If the preset number of executions has been reached, set the fourth feature as the second feature and end the denoising process.
[0025] Further, generating spectral information based on the second feature includes:
[0026] The second feature is input into the decoder module of the VAE model to generate Mel spectrum information.
[0027] Further, the spectral information is restored to an audio signal, including:
[0028] The Mel spectrum information is input into the HiFiGAN model to generate an audio signal with a WAV waveform.
[0029] Some embodiments of this application also provide an audio generation device, wherein the device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the aforementioned audio generation method.
[0030] Other embodiments of this application also provide a computer-readable medium having computer program instructions stored thereon, which can be executed by a processor to implement the audio generation method described above.
[0031] Compared to existing technologies, this application provides an audio generation scheme that first extracts entity information describing the audio from the input text, performs feature extraction on the entity information to obtain text entity features, and simultaneously samples based on a preset Gaussian distribution to obtain Gaussian features. A first feature is obtained by summing the text entity features and the Gaussian features. Then, based on a cross-attention mechanism, the first feature is optimized to obtain a second feature. Spectral information is generated based on the second feature, and the spectral information is then reconstructed into an audio signal. Because a Gaussian feature with randomness is introduced during the synthesis process, even if the input text is the same, the second feature used in each generation of spectral information will have a certain degree of random difference, avoiding completely identical audio content without variation. Furthermore, since random sampling based on a Gaussian distribution is used to introduce randomness, the degree of variation can be controlled within a reasonable range while generating random variations, avoiding excessive variation that would cause the generated audio content to differ too much from the input text. Attached Figure Description
[0032] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0033] Figure 1 A flowchart illustrating an audio generation method provided in an embodiment of this application;
[0034] Figure 2 This is a schematic diagram of the structure of the UNet model in the embodiments of this application;
[0035] Figure 3 The flowchart illustrates the process of generating random audio from text using the solution provided in this application embodiment.
[0036] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation
[0037] The present application will now be described in further detail with reference to the accompanying drawings.
[0038] In a typical configuration of this application, the terminal and the service network devices each include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.
[0039] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0040] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer program instructions, data structures, program devices, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disc (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.
[0041] This application provides an audio generation method. The method first extracts entity information describing the audio from the input text, performs feature extraction on the entity information to obtain text entity features, and simultaneously samples based on a preset Gaussian distribution to obtain Gaussian features. A first feature is obtained by summing the text entity features and the Gaussian features. Then, based on a cross-attention mechanism, the first feature is optimized to obtain a second feature. Spectral information is generated based on the second feature, and the spectral information is then reconstructed into an audio signal. Because a Gaussian feature with randomness is introduced during the synthesis process, even with the same input text, the second feature used in each generation of spectral information will have a certain degree of random difference, preventing the generated audio content from being completely identical and without variation. Furthermore, since random sampling based on a Gaussian distribution is used to introduce randomness, the degree of variation can be controlled within a reasonable range while generating random variations, avoiding excessive variation that would cause the generated audio content to differ too much from the input text.
[0042] In practical scenarios, the execution subject of this method can be a user device, a network device, or a device composed of user devices and network devices integrated through a network, or it can be an application running on the aforementioned devices. The user device includes, but is not limited to, various terminal devices such as computers, mobile phones, and tablets; the network device includes, but is not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets. Here, the cloud consists of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.
[0043] Figure 1 The present application illustrates a processing flow of an audio generation method according to an embodiment of the present application. The method includes at least the following processing steps:
[0044] Step S101: Extract entity information used to describe the audio from the input text.
[0045] The input text is text information entered by the user to generate specific audio content, and may include a description of the audio content to be generated. For example, in a real-world scenario, the input text could be "a mouse made a soft squeak" or "a strong wind blew across the lake."
[0046] The entity information refers to words in the input text that can describe the content of the audio. For example, in this embodiment, the entity information may include nouns, adjectives, and verbs in the input text. Taking the aforementioned input text "The mouse made a soft squeaking sound" as an example, the nouns, adjectives, and verbs in the text can be extracted as "mouse," "made," "squeak," and "squeaking sound," respectively. After extracting these words, they can be concatenated to form the required entity information. For example, in this embodiment, commas can be used as separators to concatenate the extracted words into entity information: mouse, made, squeak, squeaking sound.
[0047] In some embodiments of this application, when extracting entity information, BERT (Bidirectional Enoceder Representations from Transformers), LSTM (Long Short-Term Memory), and CRF (Conditional Random Field) models can be used to extract entity information describing audio from the input text. This can improve the accuracy of extraction, so as to extract entity information that reflects the relevant features of the audio more accurately from the input text.
[0048] Step S102: Extract features from entity information to obtain text entity features (feature_text). In real-world scenarios, pre-trained models can be used to extract features from entity information. For example, in this embodiment, the BERT model is used to extract features from entity information to obtain the text entity features corresponding to the entity information. The dimension of the text entity features can be set according to the needs of the actual scenario. For example, in this embodiment, 768 dimensions can be set, meaning that the text prompt feature_text of each entity information item can be represented as {f1,f2,f3,…,f766,f767,f768}, where fn (n=1~768) represents the feature value of each dimension.
[0049] Step S103: Sampling is performed based on a preset Gaussian distribution to obtain Gaussian features. In this embodiment, the mean of the Gaussian distribution can be set to 0 and the variance to 1. Sampling on this Gaussian distribution will yield the Gaussian feature feature_gaussian. In real-world scenarios, each time audio is generated, resampling is required to obtain new Gaussian features, thus ensuring that the Gaussian features introduced each time are different.
[0050] When sampling based on a Gaussian distribution, the dimension of the obtained Gaussian features can be the same as the dimension of the aforementioned text entity feature, feature_text. For example, when the dimension of the text entity feature is 768, the dimension of the sampled Gaussian features also needs to be the same as the dimension of the text entity feature, which is also 768.
[0051] Step S104: Summate the text entity features and Gaussian features to obtain the first feature. In practical scenarios, the text entity features and Gaussian features can be represented as n-dimensional vectors. For example, the text entity feature feature_text can be represented as {a1,a2,a3,…,an}, and the Gaussian feature feature_gaussian can be represented as {b1,b2,b3,…,bn}. After summing them, the first feature feature_sum = {a1+b1,a2+b2,a3+b3,…,an+bn} can be obtained.
[0052] Step S105: Based on the cross attention mechanism, the first feature is optimized to obtain the second feature.
[0053] During optimization, the first feature is plotted as a vector V (feature_sum), the Gaussian feature as a vector K (feature_gaussian), and the text entity feature as a vector Q. These are then input into a cross-attention model for computation to obtain the third feature. Next, the UNet model is used to denoise the third feature to obtain the second feature (feature_new).
[0054] In some embodiments of this application, the encoder module and decoder module of the UNet model can both be composed of multi-layer cross-attention models. Figure 2 The structure of the UNet model is shown, where the encoder module consists of N layers of cross-attention models, and the decoder module has the same structure as the encoder module.
[0055] During denoising, the text entity feature `feature_text` is first used as the V vector, and the third feature `feature` is used as the Q and K vectors. These are input into the encoder module (decoder) of the UNet model. After processing through N layers of cross-attention, the intermediate feature `feature_tem` is obtained. Then, the text entity feature `feature_text` is used as the V vector again, and the intermediate feature `feature_tem` is used as the Q and K vectors. This is input into the decoder module (decoder) of the UNet model to obtain the noise feature `feature_noise`. After obtaining the noise feature, the difference between the third feature and the noise feature is calculated, and the result can be denoted as the fourth feature.
[0056] In practical scenarios, to achieve better denoising results, the above denoising process can be repeated multiple times. In this embodiment, a preset number of executions for the denoising process can be set, such as 100, 200, or 250 times. After calculating the difference between the third feature and the noise feature to obtain the fourth feature, it can be determined whether the preset number of executions has been reached. If the preset number of executions has not been reached, the fourth feature is set as the new third feature, and the new third feature (feature) is used as the input Q vector and K vector, while the text entity feature (feature_text) is used as the input V vector. The aforementioned denoising process is then re-executed. If the preset number of executions has been reached, the fourth feature is set as the second feature (feature_new), and the entire denoising process ends. The second feature (feature_new) obtained at this point is the optimized feature information.
[0057] Step S106: Generate spectral information based on the second feature, and reconstruct the spectral information into an audio signal. The second feature obtained through the aforementioned processing contains implicit information about the audio content described by the input text. Therefore, the corresponding spectral information can be reconstructed using this second feature in a specific way. Because a Gaussian feature with randomness is introduced during the synthesis process, even with the same input text, the second feature used to generate spectral information will have a certain degree of random difference each time, preventing the generated audio content from being completely identical and without variation. Furthermore, since randomness is introduced using a Gaussian distribution-based random sampling method, the degree of variation can be controlled within a reasonable range while generating random changes, avoiding excessive variation that would cause the generated audio content to differ too much from the input text.
[0058] In some embodiments of this application, spectral information can be generated using the decoder module of a VAE (Variational Autoencoder) model. Utilizing a pre-trained model allows for faster and more accurate generation of spectral information. Furthermore, since human frequency perception is not linear in real-world scenarios, and low-frequency signals are perceived more readily than high-frequency signals, mel spectrogram information can be used to generate spectral information in order to obtain sound that is easier for users to perceive.
[0059] Furthermore, when restoring the spectral information to an audio signal, the scheme in this embodiment can input the Mel spectral information into a HiFiGAN (High Fidelity Generative Adversarial Networks) model to generate a WAV waveform audio signal. The WAV waveform audio signal refers to an audio signal file saved in .wav format, which can record relatively complete audio information and ensure that the sound is not distorted.
[0060] Because the HiFiGAN model can use multiple sub-discriminators to process the modes corresponding to different frequencies of signals in the spectrum, it can obtain better audio quality. At the same time, because the model can process these modes in parallel, the processing efficiency can also be effectively guaranteed, so that the audio signal generation process can be both efficient and high-quality.
[0061] Figure 3 The present application illustrates a processing flow for generating random audio from text using an embodiment of this application, including the following processing steps:
[0062] Step S301: When the user inputs text describing the audio content, this solution can use the BERT+LSTM+CRF model to extract the nouns n, adjectives adj, and verbs v from the text. Then, the concat function is used to combine these nouns n, adjectives adj, and verbs v, and they are concatenated with commas as separators to form entity information text.
[0063] Step S302: Use BERT to extract features from the entity information text obtained in the previous step to obtain the text entity feature feature_text.
[0064] Step S303: Set a Gaussian distribution with a mean of 0 and a variance of 1, and sample this Gaussian distribution to obtain the Gaussian feature used for this audio generation. A new Gaussian feature needs to be resampled for each audio generation.
[0065] Step S304: The text entity feature feature_text and the Gaussian feature feature_gaussian are summed to obtain the first feature feature_sum.
[0066] Step S305: The first feature, feature_sum, is used as the V vector, the Gaussian feature, feature_gaussian, is used as the K vector, and the text entity feature, feature_text, is used as the Q vector. These are then input into the cross-attention model to obtain the third feature.
[0067] Step S306: The third feature is input into the UNet model, and the noise feature `feature_noise` is output. The UNet model consists of two parts: an encoder module and a decoder module. The encoder module comprises an N-layer cross-attention model. Its input vector V is the text entity feature `feature_text` obtained in step S302, and its input vectors Q and K are both the current third feature. The decoder module has the same structure as the encoder module. Its input vector V is also the text entity feature `feature_text` obtained in step S302, and its input vectors Q and K are intermediate vectors output by the encoder module. It can also output the noise feature `feature_noise`.
[0068] In step S307, the current third feature is subtracted from the noise feature obtained in step S306 to obtain the fourth feature.
[0069] Step S308: Set the fourth feature as the current third feature. Repeat steps S306 to S308 200 times, and then set the fourth feature as the final second feature, feature_new.
[0070] Step S309: The second feature, feature_new, is processed by the decoder module in the VAE model to generate a mel spectrogram.
[0071] Step S310: Using the HiFiGAN model, the mel spectragram is restored to the audio signal wav.
[0072] Based on another aspect of this application, embodiments of this application also provide an audio generation device, the device including a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the aforementioned audio generation method.
[0073] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processing unit, it performs the functions defined in the methods of this application.
[0074] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0075] In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.
[0076] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0077] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0078] In another aspect, this application also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The aforementioned computer-readable medium carries one or more computer program instructions, which may be executed by a processor to implement the methods and / or technical solutions of the various embodiments of this application.
[0079] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.
[0080] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device in software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any particular order. Numbers corresponding to steps are used to label and distinguish different steps, and the magnitude of these numbers does not limit any particular order of execution.
Claims
1. An audio generation method, characterized in that, The method includes: Extract entity information from the input text to describe the audio; Feature extraction is performed on entity information to obtain text entity features; Sampling is performed based on a preset Gaussian distribution to obtain Gaussian features; The first feature is obtained by summing the text entity features and the Gaussian features. The first feature is used as the V vector, the Gaussian feature as the K vector, and the text entity feature as the Q vector. These are then input into the cross-attention model for computation to obtain the third feature. The third feature is denoised using the UNet model to obtain the second feature; Spectrum information is generated based on the second feature, and the spectrum information is then restored into an audio signal; The encoder and decoder modules of the UNet model are both composed of multi-layer cross-attention models. The UNet model is used to denoise the third feature to obtain the second feature, including: The text entity features are used as V vectors, and the third features are used as Q and K vectors, which are then input into the encoder module of the UNet model to obtain intermediate features. The text entity features are used as V vectors, and the intermediate features are used as Q and K vectors, which are then input into the decoder module of the UNet model to obtain noise features. The fourth feature is obtained by calculating the difference between the third feature and the noise feature. Determine whether the preset number of executions has been reached. If the preset number of executions has not been reached, set the fourth feature as the new third feature and re-execute the denoising process. If the preset number of executions has been reached, set the fourth feature as the second feature and end the denoising process.
2. The method according to claim 1, characterized in that, The entity information includes nouns, adjectives, and verbs in the input text.
3. The method according to claim 1, characterized in that, Extract entity information describing the audio from the input text, including: BERT, LSTM, and CRF models are used to extract entity information describing audio from input text.
4. The method according to claim 1, characterized in that, Feature extraction is performed on entity information to obtain text entity features, including: The BERT model is used to extract features from entity information and obtain text entity features.
5. The method according to claim 1, characterized in that, Generate spectral information based on the second feature, including: The second feature is input into the decoder module of the VAE model to generate Mel spectrum information.
6. The method according to claim 5, characterized in that, Reconstructing the spectral information into an audio signal includes: The Mel spectrum information is input into the HiFiGAN model to generate an audio signal with a WAV waveform.
7. An audio generation device, wherein, The device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to perform the method of any one of claims 1 to 6.
8. A computer-readable medium having stored thereon computer program instructions that can be executed by a processor to implement the method as described in any one of claims 1 to 6.