Method and device for generating audio through text

Abstract

The embodiment of the invention provides a method and device for generating an audio through a text, and the method comprises the steps: extracting a first feature based on a first text used for audio description; obtaining first noise, first rhythm information and first chord information, determining a first rhythm feature according to the first rhythm information, and determining a first chord feature according to the first chord information; obtaining a first fusion feature according to the first rhythm feature, the first chord feature and the first noise, and inputting the first fusion feature and the first feature into a pre-trained control network to obtain a first control feature; inputting the first feature, the first noise, the first control feature and preset time into a pre-trained correction flow network to obtain a first predicted velocity field; and determining a first audio feature according to the first noise and the first predicted velocity field, the first audio feature being used for generating a first audio.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of deep learning technology, and more particularly, to a method and apparatus for generating audio from text. Background Art

[0002] Text-to-Audio Generation (TTA) technology generates audio matching text input. With the continuous advancement of machine learning, TTA solutions based on machine learning models have gained widespread application in fields such as music generation. However, existing TTA solutions based on machine learning models still suffer from the high computational resources consumed, long computation times, and low audio quality. Summary of the Invention

[0003] The embodiments in this specification aim to provide a method and apparatus for generating audio through text, which can improve the efficiency of generating audio through text and reduce the time consumed in generating audio, so that it can be used for low-latency, high-real-time audio generation tasks. It can also reduce the negative effects of unclear descriptive text on the quality of audio generation, greatly improve the quality of audio generated in actual production scenarios, and address the shortcomings of the existing technology.

[0004] According to a first aspect, a method for generating audio from text is provided, comprising:

[0005] extracting a first feature based on a first text for audio description;

[0006] Acquiring first noise, first rhythm information, and first chord information, determining a first rhythm feature based on the first rhythm information, and determining a first chord feature based on the first chord information; obtaining a first fusion feature based on the first rhythm feature, the first chord feature, and the first noise, and inputting the first fusion feature and the first feature into a pre-trained control network to obtain a first control feature;

[0007] The first feature, the first noise, the first control feature and the preset time are input into a pre-trained correction flow network to obtain a first predicted velocity field; a first audio feature is determined based on the first noise and the first predicted velocity field, and the first audio feature is used to generate a first audio.

[0008] In a possible implementation, obtaining a first fusion feature according to the first rhythm feature, the first chord feature, and the first noise includes:

[0009] After the first rhythm feature and the first chord feature are spliced together, they are input into the convolution layer to obtain an intermediate fusion feature. Based on the intermediate fusion feature and the first noise, the first fusion feature is obtained.

[0010] In a possible implementation, the first audio feature is used to generate the first audio, including: inputting the first audio feature into an audio decoder to obtain a second audio feature, and inputting the second audio feature into an audio generator to obtain the first audio.

[0011] In one possible implementation, the rectification flow network is pre-trained by the following process:

[0012] determining a second feature based on a second text used for audio description or a preset audio corresponding to the second text;

[0013] Obtaining a second noise and a first target feature of a first target audio signal corresponding to the second noise, determining a first difference based on the second noise and the first target feature; determining a first interpolation feature based on the second noise, the first target feature, and a preset time; inputting the second feature, the first interpolation feature, and the first time point into the correction flow network to obtain a second predicted velocity field;

[0014] A first training loss is determined according to the difference between the first difference and the second predicted velocity field, and parameters of the rectification flow network are updated with the goal of minimizing the first training loss.

[0015] In a possible implementation, determining a first training loss according to a difference between the first difference and the second predicted velocity field includes:

[0016] A first training loss is determined according to an expected value of a Euclidean norm of a difference between the first difference and the second predicted velocity field and an integration result in a preset period.

[0017] In one possible implementation, the control network is pre-trained through the following process:

[0018] Initialize the parameters of the control network based on the updated parameters of the correction flow network;

[0019] determining a third feature based on a third text used for audio description or a preset audio corresponding to the third text;

[0020] Acquiring second rhythm information and second chord information, determining a second rhythm feature based on the second rhythm information, determining a second chord feature based on the second chord information, obtaining a second fusion feature based on the second rhythm feature, the second chord feature, and third noise, and inputting the second fusion feature and the third feature into the control network to obtain a second control feature;

[0021] Obtaining a third noise and a second target feature of a second target audio corresponding to the third noise, and determining a second difference based on the third noise and the second target feature;

[0022] determining a second interpolation feature based on the third noise, the second target feature, and a preset time, and inputting the third feature, the second interpolation feature, and the preset time into an updated corrected flow network to obtain a third predicted velocity field;

[0023] A second training loss is determined according to the difference between the second difference and the third predicted velocity field, and the parameters of the control network are updated with the goal of reducing the second training loss.

[0024] In one possible implementation, determining the second feature based on the second text used for audio description or the preset audio corresponding to the second text includes:

[0025] A second text for audio description is obtained, and the second text is input into a text encoder to obtain a second feature.

[0026] In one possible implementation, determining the second feature based on the second text used for audio description or the preset audio corresponding to the second text includes:

[0027] Obtain a preset audio corresponding to the second text, input the preset audio into an audio encoder, and obtain a second feature.

[0028] In a possible implementation, the audio decoder is a variational audio decoder.

[0029] According to a second aspect, there is provided an apparatus for generating audio from text, the apparatus comprising:

[0030] a text feature acquisition unit configured to extract a first feature based on a first text for audio description;

[0031] a control feature acquisition unit configured to acquire first noise and first rhythm information, determine a first rhythm feature based on the first rhythm information; obtain a first fusion feature based on the first rhythm feature and the first noise, and input the first fusion feature and the first feature into a pre-trained control network to obtain a first control feature;

[0032] The audio generation unit is configured to input the first feature, the first noise, the first control feature and the preset time into a pre-trained correction flow network to obtain a first predicted velocity field; determine a first audio feature based on the first noise and the first predicted velocity field, and the first audio feature is used to generate a first audio.

[0033] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect.

[0034] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect is implemented.

[0035] By utilizing one or more of the methods, devices, computing devices, and storage media in the above aspects, the efficiency of generating audio through text can be improved, and the time consumed in generating audio can be reduced, so that it can be used for low-latency, high-real-time audio generation tasks. It can also reduce the negative effects of unclear descriptive text on the quality of audio generation, and greatly improve the quality of audio generated in actual production scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0037] Figure 1 A schematic diagram illustrating a scheme for generating audio from text;

[0038] Figure 2 A schematic diagram illustrating a method for generating audio from text according to an embodiment of this specification;

[0039] Figure 3 A flowchart showing a method for generating audio from text according to an embodiment of this specification is shown;

[0040] Figure 4 A schematic diagram illustrating a method for generating audio from text according to another embodiment of the present specification;

[0041] Figure 5 A schematic diagram illustrating a training rectification flow network according to an embodiment of the present specification is shown;

[0042] Figure 6 A schematic diagram illustrating a training control network according to an embodiment of the present specification is shown;

[0043] Figure 7 A structural diagram of an apparatus for generating audio from text according to an embodiment of this specification is shown. DETAILED DESCRIPTION

[0044] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments derived by those skilled in the art based on the embodiments in this specification without creative effort shall fall within the scope of protection of this specification.

[0045] As mentioned earlier, Text-to-Audio Generation (TTA) technology generates audio matching text based on input text. With the continuous advancement of machine learning, TTA solutions based on machine learning models have found widespread application in fields such as music generation and speech generation. However, existing TTA solutions based on machine learning models still suffer from the high computational resources consumed and long computational time required to generate audio. Figure 1 A schematic diagram showing a method for generating audio from text is shown. Figure 1 As shown, the audio description text and initial noise can be input into a diffusion network-based audio model to obtain generated audio. However, this solution also has the following problems: on the one hand, generating audio based on a diffusion network often requires a large number of intermediate iterative steps (e.g., hundreds to thousands of steps) to gradually reverse denoise, which not only takes a long time to generate the audio, but also consumes a large amount of memory (or video memory) during the audio generation process. Moreover, due to the long audio generation time, it is often difficult to use for audio generation tasks with low response time and high real-time performance. On the other hand, the objective function used to train the diffusion network is usually highly complex, and the convergence speed of the neural network is slow during the training process, resulting in a large amount of computing resources consumed in the training process and a long training time.

[0046] In order to solve the above technical problems, the embodiments of this specification provide a method for generating audio from text. Figure 2 Schematic diagram showing a method for generating audio through text according to an embodiment of this specification. Figure 2 As shown, through this method, text features can be obtained based on the text description of the audio to be generated; fusion features can be obtained based on the initial noise, rhythm information and chord information of the audio to be generated, the fusion features and text features are input into a pre-trained control network to obtain control features, the text features, initial noise and control features are input into a pre-trained rectified flow (RectifiedFlow) network to obtain a predicted velocity field, audio features are obtained based on the predicted velocity field, and audio is generated based on the audio features.

[0047] This method has the following advantages: First, it can generate audio based on a pre-trained rectified flow network and control network. Compared to audio generation based on diffusion models, this approach consumes less video memory or memory and generates audio in a shorter time, making it suitable for low-response, high-real-time audio generation tasks. Second, compared to diffusion models, the objective function used to train the rectified flow network is less complex, resulting in faster neural network convergence during training, requiring less computational resources and shortening training time. Third, in real-world production scenarios, descriptive text for audio often contains unclear semantics or inherent ambiguity. If audio generation relies entirely on unclear descriptions, the quality of the generated audio can be low. In particular, this ambiguity often results in a lack of effective control over audio elements such as rhythm and chords, resulting in poor musicality and aural appeal in the final output audio. In this method, the control features are determined through the rhythm information, chord information and pre-trained control network of the audio to be generated. The audio features of the audio to be generated are obtained by integrating the control features with the features of the descriptive text and the initial noise input into the correction flow network. This allows the audio generated according to the audio features to obtain basic control over audio elements such as rhythm and harmony, reducing the negative impact of unclear descriptive text on the quality of audio generation, greatly improving the quality of the audio generated in actual production scenarios, and being more in line with human preferences.

[0048] The detailed process of this method is further explained below. Figure 3 A flowchart of a method for generating audio from text according to an embodiment of this specification is given. Figure 3 Said method comprises at least the following steps:

[0049] Step S301: extracting a first feature based on a first text for audio description;

[0050] Step S303: obtaining first noise and first rhythm information, determining a first rhythm feature based on the first rhythm information; obtaining a first fusion feature based on the first rhythm feature and the first noise, and inputting the first fusion feature and the first feature into a pre-trained control network to obtain a first control feature;

[0051] Step S305: Input the first feature, the first noise, the first control feature, and the preset time into a pre-trained correction flow network to obtain a first predicted velocity field; determine a first audio feature based on the first noise and the first predicted velocity field, and the first audio feature is used to generate a first audio.

[0052] First, in step S301, a first feature can be extracted based on the first text used for audio description. The first text refers to the text used to describe the audio. In different embodiments, the specific content described in the first text may be different. In one example, it may be "café background music, with guitar as the main melody, incorporating the environmental sampling prelude of the sound of coffee machine steam and the sound of turning pages (0:00-0:15): finger-plucked steel-string guitar, verse (0:15-0:45): superimposed nylon-string guitar (high pitch area)". The first feature is the text feature of the first text. In one embodiment, the first feature can be obtained by inputting the first text into a text encoder. In different embodiments, the first feature can be obtained by using different specific types of text encoders, and this specification does not limit this.

[0053] In step S303, the first noise and the first rhythm information can be obtained, and the first rhythm feature can be determined based on the first rhythm information. The first noise is the initial noise used to generate the audio. In different embodiments, the first noise can be different specific noises. In one embodiment, for example, it can be Gaussian noise. The first rhythm information is used to indicate the rhythm of the generated audio. In different embodiments, the specific format of the first rhythm information and the specific content indicated can be different. In one embodiment, for example, it can be "[X,X,X,X|_,X,_,X|X,X,X,X|_,X,_,X]", where X represents the strong beat position of the quarter note, _ represents the eighth rest, and | represents the division of the measure. In one instance, after obtaining the first rhythm information, the first rhythm information can be input into the embedding layer to obtain the first rhythm feature, such as Figure 4 shown.

[0054] After obtaining the first rhythm feature, a first fusion feature can be obtained based on the first rhythm feature and the first noise, and the first fusion feature and the first feature are input into a pre-trained control network to obtain a first control feature.

[0055] In one embodiment, the first chord information can also be obtained, and the first chord feature can be determined based on the first chord information. The first chord information is used to indicate the chord of the generated audio. In different embodiments, the specific format of the first chord information and the specific content of the indication can also be different. In one embodiment, for example, it can be "[Cmin,_,Cmin,_|Eb,_,Eb,_]", where Cmin represents the C minor triad, _ represents the eighth rest, | represents the measure division, and Eb represents the Eb major triad. In one embodiment, after obtaining the first chord information, the first chord information can be input into the embedding layer to obtain the first chord feature, such as Figure 4 shown.

[0056] Furthermore, a first fusion feature can be obtained based on the first rhythm feature, the first chord feature and the first noise. In a specific embodiment, the first rhythm feature and the first chord feature can be concatenated and input into a convolution layer to obtain an intermediate fusion feature. The first fusion feature can be obtained based on the intermediate fusion feature and the first noise, such as Figure 4 shown.

[0057] Thereafter, in step S305, the first feature, the first noise, the first control feature, and the preset time can be input into a pre-trained rectified flow network to obtain a first predicted velocity field. A first audio feature is determined based on the first noise and the first predicted velocity field. The first audio feature can be used to generate the first audio.

[0058] The rectified flow network can be used to fit a time-dependent function. The function value output by the time-dependent function, that is, the predicted velocity field output by the rectified flow network, is used to describe the direction and rate of the points on the straight line connecting the initial distribution of the data (such as Gaussian noise) to the target distribution (such as Mel spectrum). In this step, the first feature, the first noise, the first control feature, and the preset time obtained in step S303 are input into the pre-trained rectified flow network to obtain a predicted velocity field, that is, a first predicted velocity field. Furthermore, the audio feature of the audio to be generated, that is, the first audio feature, can be determined based on the first noise and the first predicted velocity field. In an actual production scenario, the first audio feature can be obtained through one or more rounds of iterative processing. For example, in one embodiment, the first audio feature can be obtained through multiple rounds of iterative processing. In the first round of multiple processing, the output features of the current round can be determined based on the first noise and the velocity field predicted in the current round. In each subsequent round of iterative processing, the output features of the current round can be determined based on the output features of the previous round and the velocity field predicted in the current round, until the last round of the multiple processing outputs the first audio feature. In a specific embodiment, the output features of each round of iterative processing can be expressed as: in, Represents the output features of this round of iteration, It represents the output feature of the previous iteration, ε represents the iteration step of this round, and v represents the velocity field predicted in this round.

[0059] After obtaining the first audio feature, the first audio can be generated according to the first audio feature. In different embodiments, the specific method of generating the first audio according to the first audio feature can be different. In one embodiment, the first audio feature can be input into an audio decoder to obtain a second audio feature, and the second audio feature can be input into an audio generator to obtain the first audio, such as Figure 4 shown.

[0060] In different specific embodiments, the audio decoder and the audio generator may be different specific types of audio decoders and audio generators. In one specific embodiment, the audio decoder may be a variational audio decoder, which may output a second audio feature (e.g., a Mel spectrum) of the first audio based on the input first audio feature, and the audio generator may generate the first audio based on the Mel spectrum of the first audio.

[0061] In different specific embodiments, the specific type of the generated first audio can be different, and this specification does not limit this. In one embodiment, the first audio can be music audio. In another embodiment, the first audio can be human voice audio.

[0062] In different embodiments, the specific training methods of the correction flow network and the control network may be different. In one embodiment, the correction flow network is pre-trained through the following process: based on the second text used for audio description or the preset audio corresponding to the second text, a second feature is determined; the second noise and the first target feature of the first target audio corresponding to the second noise are obtained, and a first difference is determined based on the second noise and the first target feature; based on the second noise, the first target feature and the preset time, a first interpolation feature is determined, and the second feature, the first interpolation feature and the first time point are input into the correction flow network to obtain a second predicted velocity field; based on the difference between the first difference and the second predicted velocity field, a first training loss is determined, and the parameters of the correction flow network are updated with the goal of minimizing the first training loss, such as Figure 5 In a specific embodiment, the first training loss may be determined according to an integration result of an expected value of the Euclidean norm of the difference between the first difference and the second predicted velocity field in a preset period.

[0063] In a specific embodiment, the first training loss is determined, which can be expressed as:

[0064]

[0065] Among them, L rf1 represents the first training loss, represents the second noise, represents the first target feature, represents the first difference, t represents the preset time, t∈[0,1]; represents the first interpolation feature, v θ () represents the correction flow network, e tx1 represents the second feature, |||| 2represents the Euclidean form, E[] represents the expected value, ∫ is the integral symbol, and d is the differential symbol.

[0066] In different specific embodiments, the specific method of determining the second feature may be different. In a specific embodiment, a second text for audio description may be obtained, and the second text may be input into a text encoder to obtain a second feature (text feature), such as Figure 6 In another specific embodiment, a preset audio corresponding to the second text can be obtained, and the preset audio is input into an audio encoder to obtain a second feature (audio feature), such as Figure 6 As shown. In different examples, the second feature can be obtained by different specific types of audio encoders, and this specification does not limit this. In this embodiment, the reason for using audio features instead of text features for training is that in some generation scenarios, high-quality audio description text is often difficult to obtain or the acquisition cost is high, while audio with semantic information of the audio to be generated is often easy to obtain. Therefore, this type of audio can be used instead of the description text to train the correction flow network and the control network (see below) to reduce the acquisition cost of training samples.

[0067] After training the correction flow network, the control network can be further trained. Therefore, in one embodiment, the control network can be pre-trained through the following process: initialize the parameters of the control network based on the updated parameters of the correction flow network; determine the third feature based on the third text used for audio description or the preset audio corresponding to the third text; obtain the second rhythm information and the second chord information, determine the second rhythm feature according to the second rhythm information, determine the second chord feature according to the second chord information, obtain the second fusion feature according to the second rhythm feature, the second chord feature and the third noise, input the second fusion feature and the third feature into the control network to obtain the second control feature; obtain the third noise and the second target feature of the second target audio corresponding to the third noise, determine the second difference according to the third noise and the second target feature; determine the second interpolation feature according to the third noise, the second target feature and the preset time, input the third feature, the second interpolation feature and the preset time into the updated correction flow network to obtain the third predicted velocity field; determine the second training loss according to the difference between the second difference and the third predicted velocity field, and update the parameters of the control network with the goal of making the second training loss tend to become smaller, such as Figure 6 In one example, during the training of the control network, the parameters of the correction flow network can also be solidified.

[0068] In a specific embodiment, the second training loss is determined as follows:

[0069]

[0070] Among them, L rf2 represents the second training loss, represents the third noise, represents the second target feature, represents the second difference, t represents the preset time, t∈[0,1]; represents the second interpolation feature, e tx2 represents the third characteristic, c represents the second control characteristic, v μ () represents the control network and the correction flow network, |||| 2 represents the Euclidean form, E[] represents the expected value, ∫ is the integral symbol, and d is the differential symbol.

[0071] According to yet another embodiment, a device for generating a report is provided. Figure 7 A structural diagram of an apparatus for generating audio through text according to an embodiment of this specification is shown, Figure 7 As shown, the apparatus 700 includes:

[0072] A text feature acquisition unit 702 is configured to extract a first feature based on a first text for audio description;

[0073] The control feature acquisition unit 704 is configured to acquire first noise and first rhythm information, determine a first rhythm feature based on the first rhythm information, obtain a first fusion feature based on the first rhythm feature and the first noise, and input the first fusion feature and the first feature into a pre-trained control network to obtain a first control feature.

[0074] The audio generation unit 706 is configured to input the first feature, the first noise, the first control feature and the preset time into a pre-trained correction flow network to obtain a first predicted velocity field; determine a first audio feature based on the first noise and the first predicted velocity field, and the first audio feature is used to generate the first audio.

[0075] Another aspect of the embodiments of this specification provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute any one of the above methods.

[0076] On the other hand, the embodiments of this specification provide a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, any one of the above methods is implemented.

[0077] It should be understood that the descriptions such as “first” and “second” in this article are only used to distinguish similar concepts for the sake of simplicity of description and do not have any other limiting effect.

[0078] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD through their own programming, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0079] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also understand that in addition to implementing the controller in pure computer-readable program code, the controller can also be implemented in the form of logic gates, switches, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0080] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude that with the future development of computer technology, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0081] Although one or more embodiments of this specification provide method operation steps as described in the embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any particular order.

[0082] For the convenience of description, the above devices are described in terms of functions divided into various modules. Of course, when implementing one or more of the present specifications, the functions of each module can be implemented in the same or multiple software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0083] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0084] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0086] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0087] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0088] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0089] Those skilled in the art will appreciate that one or more embodiments of this specification may be provided as a method, system, or computer program product. Thus, one or more embodiments of this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0091] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between the various embodiments can be referenced across them. Each embodiment focuses on the differences from the other embodiments. In particular, since the system embodiments are generally similar to the method embodiments, their description is relatively simple. For relevant parts, reference can be made to the description of the method embodiments. Throughout this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and integrate the different embodiments or examples, and features of different embodiments or examples, described in this specification, without conflict.

[0092] The foregoing description is merely an example of one or more embodiments of this specification and is not intended to limit the one or more embodiments of this specification. Those skilled in the art will appreciate that various modifications and variations of one or more embodiments of this specification are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this specification are intended to be included within the scope of the claims.

Claims

1. A method for generating audio from text, comprising: extracting a first feature based on a first text for audio description; Acquiring first noise and first rhythm information, and determining a first rhythm feature according to the first rhythm information; Obtaining a first fusion feature based on the first rhythm feature and the first noise, and inputting the first fusion feature and the first feature into a pre-trained control network to obtain a first control feature; The first feature, the first noise, the first control feature and the preset time are input into a pre-trained correction flow network to obtain a first predicted velocity field; a first audio feature is determined based on the first noise and the first predicted velocity field, and the first audio feature is used to generate a first audio.

2. The method according to claim 1, further comprising: obtaining first chord information, and determining a first chord feature based on the first chord information; Obtaining a first fusion feature according to the first rhythm feature and the first noise includes: A first fusion feature is obtained according to the first rhythm feature, the first chord feature and the first noise.

3. The method according to claim 1, wherein Obtaining a first fusion feature according to the first rhythm feature, the first chord feature, and the first noise includes: After the first rhythm feature and the first chord feature are spliced together, they are input into the convolution layer to obtain an intermediate fusion feature. Based on the intermediate fusion feature and the first noise, the first fusion feature is obtained.

4. The method according to claim 1, wherein The first audio feature is used to generate the first audio, including: inputting the first audio feature into an audio decoder to obtain a second audio feature, and inputting the second audio feature into an audio generator to obtain the first audio.

5. The method according to claim 1, wherein The rectified flow network is pre-trained by the following process: determining a second feature based on a second text used for audio description or a preset audio corresponding to the second text; Obtaining a second noise and a first target feature of a first target audio signal corresponding to the second noise, determining a first difference based on the second noise and the first target feature; determining a first interpolation feature based on the second noise, the first target feature, and a preset time; inputting the second feature, the first interpolation feature, and the first time point into the correction flow network to obtain a second predicted velocity field; A first training loss is determined according to the difference between the first difference and the second predicted velocity field, and parameters of the rectification flow network are updated with the goal of minimizing the first training loss.

6. The method according to claim 5, wherein: Determining a first training loss according to a difference between the first difference and the second predicted velocity field includes: A first training loss is determined according to an expected value of a Euclidean norm of a difference between the first difference and the second predicted velocity field and an integration result in a preset period.

7. The method according to claim 5, wherein: The control network is pre-trained through the following process: Initialize the parameters of the control network based on the updated parameters of the correction flow network; determining a third feature based on a third text used for audio description or a preset audio corresponding to the third text; Acquiring second rhythm information and second chord information, determining a second rhythm feature based on the second rhythm information, determining a second chord feature based on the second chord information, obtaining a second fusion feature based on the second rhythm feature, the second chord feature, and third noise, and inputting the second fusion feature and the third feature into the control network to obtain a second control feature; Obtaining a third noise and a second target feature of a second target audio corresponding to the third noise, and determining a second difference based on the third noise and the second target feature; determining a second interpolation feature based on the third noise, the second target feature, and a preset time, and inputting the third feature, the second interpolation feature, and the preset time into an updated corrected flow network to obtain a third predicted velocity field; A second training loss is determined according to the difference between the second difference and the third predicted velocity field, and the parameters of the control network are updated with the goal of reducing the second training loss.

8. The method according to claim 5, wherein: Determining a second feature based on a second text used for audio description or a preset audio corresponding to the second text includes: A second text for audio description is obtained, and the second text is input into a text encoder to obtain a second feature.

9. The method according to claim 5, wherein: Determining a second feature based on a second text used for audio description or a preset audio corresponding to the second text includes: Obtain a preset audio corresponding to the second text, input the preset audio into an audio encoder, and obtain a second feature.

10. The method according to claim 1, wherein The audio decoder is a variational audio decoder.

11. A device for generating audio from text, the device comprising: a text feature acquisition unit configured to extract a first feature based on a first text for audio description; a control feature acquisition unit configured to acquire first noise and first rhythm information, determine a first rhythm feature based on the first rhythm information; obtain a first fusion feature based on the first rhythm feature and the first noise, and input the first fusion feature and the first feature into a pre-trained control network to obtain a first control feature; The audio generation unit is configured to input the first feature, the first noise, the first control feature and the preset time into a pre-trained correction flow network to obtain a first predicted velocity field; determine a first audio feature based on the first noise and the first predicted velocity field, and the first audio feature is used to generate a first audio.

12. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 10.

13. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 10 is implemented.

Images