Audio generation method and device, equipment and medium

Abstract

The invention relates to the field of computers, and provides an audio generation method and device, equipment and a medium, the method comprises the following steps: obtaining to-be-generated audio information and audio generation task information, the to-be-generated audio information being used for representing information for generating required audio, and the audio task information being used for representing a task generation type of the to-be-generated audio; inputting the audio information to be generated and the audio generation task information into an audio generation model to obtain an audio generation result output by the audio generation model; wherein the audio generation model is obtained by training the same speed vector field based on audio training information corresponding to different task generation types and audio samples corresponding to the audio training information; the audio generation model is used for generating corresponding audio by using the velocity vector field based on the audio generation task information. According to the invention, the audio generation model switches the speed vector field based on the task generation type to quickly generate the audio, so that the audio generation quality is ensured and different audio generation requirements are met.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an audio generation method, device, equipment and medium. Background Art

[0002] In recent years, artificial intelligence technology has developed rapidly at an astonishing pace and has gradually been widely used in various industries. Among them, audio is an indispensable modality in applications and has been widely adopted in many scenarios.

[0003] Currently, autoregressive models are mostly used to generate audio. However, due to their gradual generation characteristics, autoregressive models have a slow generation speed and are prone to instability when processing long or complex audio signals, making it difficult to meet application scenarios with high real-time requirements. Summary of the Invention

[0004] The present invention provides an audio generation method, apparatus, device and medium to address the drawback of slow audio generation speed of the autoregressive model in the prior art. The method can quickly generate audio, improve the audio generation efficiency and ensure the audio generation quality.

[0005] The present invention provides an audio generation method, comprising: obtaining audio information to be generated and audio generation task information, wherein the audio information to be generated is used to represent information for generating required audio, and the audio task information is used to represent the task generation type of the audio to be generated; inputting the audio information to be generated and the audio generation task information into an audio generation model to obtain an audio generation result output by the audio generation model; wherein the audio generation model is obtained by training the same velocity vector field based on audio training information corresponding to different task generation types and audio samples corresponding to the audio training information; and the audio generation model is used to generate corresponding audio based on the audio generation task information using the velocity vector field.

[0006] According to an audio generation method provided by the present invention, audio information to be generated and audio generation task information are input into an audio generation model to obtain an audio generation result output by the audio generation model, including: inputting the audio information to be generated and the audio generation task information into the audio generation model, and gradually converting the random noise distribution into audio along the velocity vector field based on the audio information to be generated, starting from the random noise distribution; decoding the audio to obtain the audio generation result.

[0007] According to an audio generation method provided by the present invention, before inputting the audio information to be generated and the audio generation task information into the audio generation model, the method includes: obtaining audio training information of at least one task generation type and audio samples corresponding to the audio training information; inputting the obtained audio training information of at least one task generation type and the audio samples corresponding to the audio training information into the previously created audio generation model, so as to predict the first position representation of the corresponding audio sample in the embedding space based on the audio training information for each task generation type, and predicting the second position representation of the corresponding audio sample in the embedding space based on the audio training information; wherein the embedding space is used to characterize a high-dimensional vector space that maps different types of data into continuous vector representations; according to the first position representation and the second position representation, combined with the optimization objective, a target loss function is obtained; adjusting the model parameters, and re-inputting the obtained audio training information of at least one task generation type and the audio samples corresponding to the audio training information into the audio generation model to iteratively train the audio generation model until a preset maximum number of iterations is reached, ending the training, selecting the model parameters that minimize the target loss function, and obtaining a trained audio generation model.

[0008] According to an audio generation method provided by the present invention, a first position representation of a corresponding audio sample in an embedded space is predicted based on audio training information, and a second position representation of the corresponding audio sample in the embedded space is predicted based on the audio training information, including: based on the audio training information, starting from a first random noise distribution, gradually converting the corresponding noise distribution into audio along an initialized velocity vector field, to obtain a first position representation of the audio sample corresponding to the audio training information in the embedded space; based on the audio training information, starting from a second random noise distribution, gradually converting the corresponding noise distribution into audio along an initialized velocity vector field, to obtain a second position representation of the audio sample corresponding to the audio training information in the embedded space; wherein the second random noise distribution is different from the first random noise distribution.

[0009] According to an audio generation method provided by the present invention, model parameters are adjusted, and audio training information of at least one task generation type and audio samples corresponding to the audio training information are re-input into the audio generation model to iteratively train the audio generation model, including: adjusting model parameters, and re-inputting audio training information of at least one task generation type and audio samples corresponding to the audio training information into the audio generation model to iteratively train the velocity vector field of the audio generation model corresponding to the task generation type.

[0010] According to an audio generation method provided by the present invention, after obtaining audio training information of at least one task generation type and audio samples corresponding to the audio training information, the method includes: identifying the data type of the audio training information corresponding to each task generation type and the audio information corresponding to the audio training information; when determining that the data type is text, performing a first preprocessing on the corresponding text information, and converting the corresponding preprocessed text information into a text feature vector; when determining that the data type is audio, performing a second preprocessing on the corresponding audio information, and converting the corresponding preprocessed audio information into an audio feature vector.

[0011] According to an audio generation method provided by the present invention, audio information to be generated and audio generation task information are obtained, including: obtaining the audio information to be generated; when it is determined that the audio information to be generated contains a prompt, determining that the audio task information is a prompt-based speech generation task; when it is determined that the audio information to be generated contains audio, determining that the audio task information is an audio adjustment task; when it is determined that the audio information to be generated is text information, performing semantic recognition on the audio information to be generated, determining the task generation type of the audio to be generated, and obtaining the audio task information, or obtaining the audio task information corresponding to the audio information to be generated.

[0012] The present invention also provides an audio generation device, including: an information acquisition module, which acquires audio information to be generated and audio generation task information, wherein the audio information to be generated is used to characterize the information for generating the required audio, and the audio task information is used to characterize the task generation type of the audio to be generated; an audio generation module, which inputs the audio information to be generated and the audio generation task information into an audio generation model to obtain an audio generation result output by the audio generation model; wherein the audio generation model is obtained by training the same velocity vector field based on audio training information corresponding to different task generation types and audio samples corresponding to the audio training information; the audio generation model is used to generate corresponding audio based on the audio generation task information using the velocity vector field.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described audio generation methods when executing the computer program.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the above-mentioned audio generation methods when executed by a processor.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned audio generation methods.

[0016] The audio generation method, device, equipment and medium provided by the present invention can accurately locate user needs by clarifying the audio information to be generated and the audio generation task information, laying the foundation for generating audio that meets expectations, and inputting the information into the audio generation model so that the audio generation model can switch the corresponding velocity vector field based on the task generation type, thereby flexibly adapting to different audio generation needs. There is no need to use multiple different models to complete different audio generation tasks, which reduces the cost and complexity of use, and also facilitates the deployment and maintenance of the model. Audio is quickly generated based on the selected velocity vector field, which improves the audio generation efficiency and ensures the audio generation quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 It is a flowchart of the audio generation method provided by the present invention; Figure 2 Schematic diagram of the process of training the audio generation model provided by the present invention; Figure 3 It is a structural diagram of the audio generating device provided by the present invention; Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0020] Figure 1 It is a flowchart of the audio generation method provided by the present invention, such as Figure 1 As shown, the method includes: S11, obtaining audio information to be generated and audio generation task information, wherein the audio information to be generated is used to represent information for generating the required audio, and the audio task information is used to represent the task generation type of the audio to be generated; S12, inputting the audio information to be generated and the audio generation task information into the audio generation model to obtain the audio generation result output by the audio generation model; wherein, the audio generation model is obtained by training the same velocity vector field based on the audio training information corresponding to different task generation types and the audio samples corresponding to the audio training information; the audio generation model is used to generate corresponding audio based on the audio generation task information using the velocity vector field.

[0021] It should be noted that the step numbers "S1N" in this manual do not represent the order of the audio generation method. Figure 2 The audio generation method of the present invention is described.

[0022] Step S11: Acquire audio information to be generated and audio generation task information. The audio information to be generated is used to represent information for generating required audio, and the audio task information is used to represent the task generation type of the audio to be generated.

[0023] It should be noted that when obtaining the audio information to be generated and the audio generation task information, the audio information to be generated can be obtained based on the audio information to be generated input by the user, and the audio generation task information can be directly determined based on the task type input by the user, or determined by identifying the audio information to be generated.

[0024] Specifically, obtaining the audio information to be generated and the audio generation task information includes: obtaining the audio information to be generated; when determining that the audio information to be generated contains a prompt, determining that the audio task information is a prompt-based speech generation task; when determining that the audio information to be generated contains audio, determining that the audio task information is an audio adjustment task; when determining that the audio information to be generated is text information, performing semantic recognition on the audio information to be generated, determining the task generation type of the audio to be generated, and obtaining the audio task information, or obtaining the audio task information corresponding to the audio information to be generated.

[0025] It should be noted that the task generation type includes at least one of a speech generation task (Text-to-Speech, abbreviated as TTS), a prompt-based speech generation task (Prompt TTS), an audio generation task (Text-to-Audio, abbreviated as T2A), a music generation task (Text-to-Music) and an audio adjustment task (Audio Edit). When the task generation type is TTS and / or T2A, the audio task information can be determined by identifying the corresponding audio information to be generated, or directly determined according to the task generation type input by the user. No further limitation is made here.

[0026] Step S12: input the audio information to be generated and the audio generation task information into the audio generation model to obtain the audio generation result output by the audio generation model; wherein, the audio generation model is obtained by training the same velocity vector field based on the audio training information corresponding to different task generation types and the audio samples corresponding to the audio training information; the audio generation model is used to generate corresponding audio based on the audio generation task information using the velocity vector field.

[0027] In this embodiment, the audio information to be generated and the audio generation task information are input into the audio generation model to obtain the audio generation result output by the audio generation model, including: inputting the audio information to be generated and the audio generation task information into the audio generation model, and gradually converting the random noise distribution into audio along the velocity vector field based on the audio information to be generated, starting from the random noise distribution; decoding the audio to obtain the audio generation result.

[0028] It should be added that decoding is mainly implemented through the audio decoder in the audio generation model to convert the generated audio features into playable audio signals. In addition, the audio generation model can adopt a consistent stream matching model to ensure that the generated audio is consistent with the input audio information to be generated in the embedding space. This allows the generated audio to better conform to the semantics and intent of the input description, and ensures that data of different modalities are mapped to the same or similar representations in the embedding space to ensure cross-modal consistency, which helps to improve the quality and relevance of the generated audio.

[0029] In an optional embodiment, before inputting the audio information to be generated and the audio generation task information into the audio generation model, it includes: obtaining audio training information of at least one task generation type and audio samples corresponding to the audio training information; inputting the obtained audio training information of at least one task generation type and the audio samples corresponding to the audio training information into the previously created audio generation model, so as to predict the first position representation of the corresponding audio sample in the embedding space based on the audio training information for each task generation type, and predict the second position representation of the corresponding audio sample in the embedding space based on the audio training information; wherein the embedding space is used to characterize a high-dimensional vector space that maps different types of data into continuous vector representations; according to the first position representation and the second position representation, combined with the optimization objective, a target loss function is obtained; the model parameters are adjusted, and the audio training information of at least one task generation type and the audio samples corresponding to the audio training information are re-input into the audio generation model to iteratively train the audio generation model until a preset maximum number of iterations is reached, the training is ended, the model parameters that minimize the target loss function are selected, and a trained audio generation model is obtained.

[0030] It should be added that when obtaining audio training information of at least one task generation type and audio samples corresponding to the audio training information, you can select training data of a specific task generation type based on usage requirements, or you can select training data training models corresponding to all task generation types to achieve a unified audio generation model that supports multiple audio generation tasks such as TTS, Prompt TTS, T2A, Text-to-Music, and Audio Edit. By designing a general model architecture and data processing flow, the model can flexibly adapt to different audio generation requirements, thereby improving the versatility and practicality of the model. Users do not need to use multiple different models to complete different audio generation tasks, which reduces the cost and complexity of use and facilitates the deployment and maintenance of the model.

[0031] In addition, Audio Edit enables the audio generation model to support editing and modification of the generated predicted audio. Users can personalize the generated audio according to their own needs to meet the needs and preferences of different users, provide users with more creative space and possibilities, stimulate users' creativity, and enhance user experience and model flexibility.

[0032] In an alternative embodiment, reference Figure 2 , obtain the training data corresponding to TTS, Prompt TTS, T2A, Text-to-Music and Audio Edit and assign task numbers to the training data of each task type. The training data includes audio training information corresponding to the task generation type and audio samples corresponding to the audio training information, so as to input the training data corresponding to all the task generation types obtained into the audio generation model to be trained, and train the model based on the above training method, so as to facilitate the selection of the corresponding velocity vector field based on the task type to switch the corresponding task during the application process, so that the model can flexibly adapt to different audio generation requirements, adapt to different audio generation tasks, and realize multi-functional audio generation.

[0033] It should be noted that TTS audio training information includes text information used to generate the desired speech, Prompt TTS audio training information includes text information and prompts used to generate the desired speech, T2A audio training information includes text information used to generate the desired audio, Text-to-Music audio training information includes text information used to generate the desired music, and Audio Edit audio training information includes previously generated audio and text information used to represent the required speech adjustments. Furthermore, the audio samples for Text-to-Music are music.

[0034] Specifically, predicting a first position representation of the corresponding audio sample in the embedding space based on the audio training information, and predicting a second position representation of the corresponding audio sample in the embedding space based on the audio training information include: based on the audio training information, starting from a first random noise distribution, gradually converting the corresponding noise distribution into audio along an initialized velocity vector field, and obtaining a first position representation of the audio sample corresponding to the audio training information in the embedding space; based on the audio training information, starting from a second random noise distribution, gradually converting the corresponding noise distribution into audio along an initialized velocity vector field, and obtaining a second position representation of the audio sample corresponding to the audio training information in the embedding space; wherein the second random noise distribution is different from the first random noise distribution.

[0035] In addition, adjusting the model parameters, re-inputting the audio training information of at least one task generation type and the audio samples corresponding to the audio training information into the audio generation model to iteratively train the audio generation model, including: adjusting the model parameters, re-inputting the audio training information of at least one task generation type and the audio samples corresponding to the audio training information into the audio generation model to iteratively train the velocity vector field of the task generation type corresponding to the audio generation model.

[0036] It should be noted that the velocity vector field is defined by the conversion between the audio training information of the corresponding task generation type and its corresponding audio samples, so as to iteratively train the velocity vector fields corresponding to each task generation type of the audio generation model, so that the audio generation model learns the mapping relationship between the audio training information of the corresponding task generation type and the audio information corresponding to the audio training information, optimizes the velocity vector field, realizes data stream matching in the audio generation process, and makes the audio generation process more stable and efficient.

[0037] In addition, the objective loss function is expressed as: in, Indicates the corresponding model parameters The target loss function of represents the expectation of cross-modal consistency at time step t; Represents the time step t and time steps Time expectations; represents the first random noise distribution, t represents the time step corresponding to the first random noise distribution, Represents the point in the embedding space corresponding to the first random noise distribution; It indicates that starting from the first random noise distribution, the first position representation of the corresponding audio sample in the embedding space is predicted based on the audio training information; represents the second random noise distribution, represents the corresponding time step of the second random noise distribution, Represents the point in the embedding space corresponding to the second random noise distribution; It indicates that starting from the second random noise distribution, a second position representation of the corresponding audio sample in the embedding space is predicted based on the audio training information; represents the square of the Euclidean norm, that is, the sum of the squares of vector differences; represents the regularization parameter, used to balance and the significance of both losses; Represents the velocity vector field, which is used to describe how the corresponding noise distribution changes from a point at time step t Move to another point, i.e. point The velocity vector at time step t, Used to describe the corresponding noise distribution expressed in time steps Time point The corresponding velocity vector.

[0038] It should be added that and For the same velocity vector field, through self-consistency constraints, the model can efficiently learn the mapping relationship between the audio training information of the corresponding task generation type and the audio information corresponding to the audio training information to optimize the velocity vector field, thereby enabling the model to start from different starting points. and Generating the same audio sample along the same velocity vector field quickly generates high-quality audio samples, avoids uncertainty in the generation results, and improves learning efficiency and generation speed. In an optional embodiment, after obtaining audio training information of at least one task generation type and audio samples corresponding to the audio training information, the method includes: identifying the data type of the audio training information corresponding to each task generation type and the audio information corresponding to the audio training information; when determining that the data type is text, performing a first preprocessing on the corresponding text information and converting the corresponding preprocessed text information into a text feature vector; when determining that the data type is audio, performing a second preprocessing on the corresponding audio information and converting the corresponding preprocessed audio information into an audio feature vector.

[0039] It should be noted that the first preprocessing can be performed through methods such as word segmentation and word embedding, and vector conversion can be implemented using existing model encoding techniques, such as an improved convolutional neural network (ConvNeXt V2 blocks) model, to encode the corresponding text information. This can be configured based on actual design requirements and is not further defined here. Furthermore, the second preprocessing can be performed through methods such as sampling and quantization, and vector conversion can be implemented using existing model encoding techniques, such as Mel-spectrogram, to convert the audio signal into a spectrogram. This can be configured based on actual design requirements and is not further defined here.

[0040] To sum up, the embodiments of the present invention clarify the audio information to be generated and the audio generation task information, so that the system can accurately locate user needs, lay the foundation for generating audio that meets expectations, and input the information into the audio generation model, so that the audio generation model can switch the corresponding velocity vector field based on the task generation type, thereby flexibly adapting to different audio generation needs. There is no need to use multiple different models to complete different audio generation tasks, which reduces the cost and complexity of use, and also facilitates the deployment and maintenance of the model. It also quickly generates audio based on the selected velocity vector field, improves the audio generation efficiency and ensures the audio generation quality.

[0041] The audio generating device provided by the present invention is described below. The audio generating device described below and the audio generating method described above can be referenced to each other.

[0042] Figure 3 A schematic structural diagram of an audio generating device is shown, the device comprising: An information acquisition module 31 acquires audio information to be generated and audio generation task information, wherein the audio information to be generated is used to represent information for generating the required audio, and the audio task information is used to represent a task generation type of the audio to be generated; The audio generation module 32 inputs the audio information to be generated and the audio generation task information into the audio generation model to obtain the audio generation result output by the audio generation model; wherein, the audio generation model is obtained by training the same velocity vector field based on the audio training information corresponding to different task generation types and the audio samples corresponding to the audio training information; the audio generation model is used to generate corresponding audio based on the audio generation task information using the velocity vector field.

[0043] In this embodiment, the information acquisition module 31 includes: an information acquisition unit, which acquires the audio information to be generated; a task type determination unit, which determines that the audio task information is a prompt-based speech generation task when it is determined that the audio information to be generated contains a prompt; when it is determined that the audio information to be generated contains audio, it determines that the audio task information is an audio adjustment task; when it is determined that the audio information to be generated is text information, it performs semantic recognition on the audio information to be generated, determines the task generation type of the audio to be generated, and obtains the audio task information, or obtains the audio task information corresponding to the audio information to be generated.

[0044] In addition, the audio generation module 32 is used to: input the audio information to be generated and the audio generation task information into the audio generation model, so as to gradually convert the random noise distribution into audio along the velocity vector field based on the audio information to be generated, starting from the random noise distribution; and decode the audio to obtain the audio generation result.

[0045] In an optional embodiment, the device further includes: a sample acquisition module, which acquires audio training information of at least one task generation type and audio samples corresponding to the audio training information before inputting the audio information to be generated and the audio generation task information into the audio generation model; a training module, which inputs the acquired audio training information of at least one task generation type and the audio samples corresponding to the audio training information into the previously created audio generation model, so as to predict the first position representation of the corresponding audio sample in the embedding space based on the audio training information for each task generation type, and predict the second position representation of the corresponding audio sample in the embedding space based on the audio training information; wherein the embedding space is used to represent a high-dimensional vector space that maps different types of data into continuous vector representations; a function acquisition module, which obtains a target loss function based on the first position representation and the second position representation in combination with the optimization objective; an iteration module, which adjusts the model parameters, re-inputs the acquired audio training information of at least one task generation type and the audio samples corresponding to the audio training information into the audio generation model, so as to iteratively train the audio generation model until a preset maximum number of iterations is reached, terminates the training, selects the model parameters that minimize the target loss function, and obtains a trained audio generation model.

[0046] Specifically, the training module is used to: based on audio training information, starting from a first random noise, gradually convert the corresponding noise distribution into audio along an initialized velocity vector field, and obtain a first position representation of the audio sample corresponding to the audio training information in the embedding space; based on audio training information, starting from a second random noise distribution, gradually convert the corresponding noise distribution into audio along an initialized velocity vector field, and obtain a second position representation of the audio sample corresponding to the audio training information in the embedding space; wherein the second random noise distribution is different from the first random noise distribution.

[0047] In addition, the iteration module is used to: adjust the model parameters, re-input the acquired audio training information of at least one task generation type and the audio samples corresponding to the audio training information into the audio generation model, so as to iteratively train the velocity vector field of the audio generation model corresponding to the task generation type.

[0048] In an optional embodiment, the device further includes: an information processing module, which, after obtaining audio training information of at least one task generation type and audio samples corresponding to the audio training information, identifies the data type of the audio training information corresponding to each task generation type and the audio information corresponding to the audio training information; when it is determined that the data type is text, performs a first preprocessing on the corresponding text information, and converts the corresponding preprocessed text information into a text feature vector; when it is determined that the data type is audio, performs a second preprocessing on the corresponding audio information, and converts the corresponding preprocessed audio information into an audio feature vector.

[0049] To sum up, the embodiment of the present invention clarifies the audio information to be generated and the audio generation task information through the information acquisition module. The system can accurately locate user needs, lay the foundation for generating audio that meets expectations, and input the information into the audio generation model through the audio generation module, so that the audio generation model can switch the corresponding velocity vector field based on the task generation type, thereby flexibly adapting to different audio generation needs. There is no need to use multiple different models to complete different audio generation tasks, which reduces the cost and complexity of use, and also facilitates the deployment and maintenance of the model. It also quickly generates audio based on the selected velocity vector field, improves the audio generation efficiency and ensures the audio generation quality.

[0050] Figure 4 An example of a physical structure diagram of an electronic device is shown below. Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communications bus 440. The processor 410 may call logic instructions in the memory 430 to execute an audio generation method, which includes: obtaining audio information to be generated and audio generation task information, wherein the audio information to be generated is used to represent information for generating the desired audio, and the audio task information is used to represent the task generation type of the audio to be generated; inputting the audio information to be generated and the audio generation task information into an audio generation model to obtain an audio generation result output by the audio generation model; wherein the velocity vector field is trained based on audio training information corresponding to the task generation type and audio samples corresponding to the audio training information; and the audio generation model is used to select a corresponding velocity vector field based on the audio generation task information, and generate corresponding audio using the selected velocity vector field according to the audio information to be generated.

[0051] Furthermore, the logic instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0052] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the audio generation method provided by the above methods, which includes: obtaining audio information to be generated and audio generation task information, the audio information to be generated is used to characterize the information for generating the required audio, and the audio task information is used to characterize the task generation type of the audio to be generated; inputting the audio information to be generated and the audio generation task information into the audio generation model to obtain the audio generation result output by the audio generation model; wherein the velocity vector field is obtained by training based on audio training information of the corresponding task generation type and audio samples corresponding to the audio training information; the audio generation model is used to select the corresponding velocity vector field based on the audio generation task information, and generate the corresponding audio according to the audio information to be generated using the selected velocity vector field.

[0053] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the audio generation method provided by the above-mentioned methods, the method comprising: obtaining audio information to be generated and audio generation task information, the audio information to be generated being used to characterize information for generating required audio, and the audio task information being used to characterize the task generation type of the audio to be generated; inputting the audio information to be generated and the audio generation task information into an audio generation model to obtain an audio generation result output by the audio generation model; wherein the velocity vector field is obtained by training based on audio training information of the corresponding task generation type and audio samples corresponding to the audio training information; the audio generation model is used to select a corresponding velocity vector field based on the audio generation task information, and generate corresponding audio using the selected velocity vector field according to the audio information to be generated.

[0054] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0055] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An audio generation method, characterized in that: include: Acquire audio information to be generated and audio generation task information, wherein the audio information to be generated is used to represent information for generating required audio, and the audio task information is used to represent a task generation type of the audio to be generated; Inputting the audio information to be generated and the audio generation task information into an audio generation model to obtain an audio generation result output by the audio generation model; wherein the audio generation model is obtained by training the same velocity vector field based on audio training information corresponding to different task generation types and audio samples corresponding to the audio training information; The audio generation model is used to generate corresponding audio based on the audio generation task information using the velocity vector field.

2. The audio generation method according to claim 1, wherein Inputting the audio information to be generated and the audio generation task information into an audio generation model to obtain an audio generation result output by the audio generation model includes: Inputting the audio information to be generated and the audio generation task information into an audio generation model, so as to gradually convert the random noise distribution into audio along the velocity vector field based on the audio information to be generated, starting from the random noise distribution; The audio is decoded to obtain an audio generation result.

3. The audio generation method according to claim 1, wherein Before inputting the to-be-generated audio information and the audio generation task information into the audio generation model, the method includes: Obtaining audio training information of at least one task generation type and an audio sample corresponding to the audio training information; Inputting the acquired audio training information of at least one task generation type and the audio samples corresponding to the audio training information into a previously created audio generation model, so as to predict, for each task generation type, a first position representation of the corresponding audio sample in an embedding space based on the audio training information, and a second position representation of the corresponding audio sample in the embedding space based on the audio training information; wherein the embedding space is used to represent a high-dimensional vector space that maps different types of data into continuous vector representations; Obtaining a target loss function based on the first position representation and the second position representation in combination with an optimization objective; Adjust the model parameters, re-input the acquired audio training information of at least one task generation type and the audio samples corresponding to the audio training information into the audio generation model to iteratively train the audio generation model until a preset maximum number of iterations is reached, end the training, select the model parameters that minimize the target loss function, and obtain a trained audio generation model.

4. The audio generation method according to claim 3, characterized in that Predicting a first position representation of the corresponding audio sample in the embedding space based on the audio training information, and predicting a second position representation of the corresponding audio sample in the embedding space based on the audio training information, comprising: Based on the audio training information, starting from a first random noise distribution, the corresponding noise distribution is gradually converted into audio along an initialized velocity vector field to obtain a first position representation of an audio sample corresponding to the audio training information in an embedding space; Based on the audio training information, starting from a second random noise distribution, the corresponding noise distribution is gradually converted into audio along the initialized velocity vector field to obtain a second position representation of the audio sample corresponding to the audio training information in the embedding space; wherein the second random noise distribution is different from the first random noise distribution.

5. The audio generation method according to claim 4, characterized in that Adjusting model parameters, re-inputting the acquired audio training information of at least one task generation type and the audio samples corresponding to the audio training information into the audio generation model to iteratively train the audio generation model, including: Adjust the model parameters, and re-input the acquired audio training information of at least one task generation type and the audio samples corresponding to the audio training information into the audio generation model to iteratively train the velocity vector field of the audio generation model corresponding to the task generation type.

6. The audio generation method according to claim 3, characterized in that After obtaining audio training information of at least one task generation type and an audio sample corresponding to the audio training information, the method includes: Identifying the audio training information corresponding to each of the task generation types and the data type of the audio information corresponding to the audio training information; When it is determined that the data type is text, performing a first preprocessing on the corresponding text information, and converting the corresponding preprocessed text information into a text feature vector; When it is determined that the data type is audio, a second preprocessing is performed on the corresponding audio information, and the corresponding preprocessed audio information is converted into an audio feature vector.

7. The audio generation method according to claim 1, wherein: Get the audio information to be generated and the audio generation task information, including: Obtaining the audio information to be generated; When it is determined that the audio information to be generated includes a prompt, determining that the audio task information is a prompt-based speech generation task; When it is determined that the audio information to be generated includes audio, determining that the audio task information is an audio adjustment task; When it is determined that the audio information to be generated is text information, semantic recognition is performed on the audio information to be generated, the task generation type of the audio to be generated is determined, and audio task information is obtained, or audio task information corresponding to the audio information to be generated is obtained.

8. An audio generating device, characterized in that include: An information acquisition module is configured to acquire audio information to be generated and audio generation task information, wherein the audio information to be generated is used to represent information for generating the required audio, and the audio task information is used to represent a task generation type of the audio to be generated; An audio generation module inputs the audio information to be generated and the audio generation task information into an audio generation model to obtain an audio generation result output by the audio generation model; wherein the audio generation model is obtained by training the same velocity vector field based on audio training information corresponding to different task generation types and audio samples corresponding to the audio training information; The audio generation model is used to generate corresponding audio based on the audio generation task information using the velocity vector field.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the audio generation method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the audio generation method according to any one of claims 1 to 7 is implemented.

Images