Audio generation model training method and device and audio generation method and device

Abstract

The embodiment of the invention provides an audio generation model training method and device and an audio generation method and device, and relates to the technical field of computers, and the audio generation model training method comprises the steps: determining a video sample and an audio sample corresponding to the video sample; determining a target audio feature sample according to the audio feature sample of the audio sample and a preset audio feature sample; determining a target audio and video feature sample according to the target audio feature sample and a video feature sample of the video sample; according to the target audio and video feature sample, the audio feature sample and a preset audio feature sample, an audio generation model is obtained through training, and the audio generation model is used for outputting a target audio corresponding to the target video according to the input target video and preset audio features. The audio generation model obtained by training according to the method can greatly improve the detail matching degree of the target audio and the target video.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a method for training an audio generation model. One or more embodiments of this specification also relate to an audio generation model training device, an audio generation method, and an audio generation device. Background Art

[0002] Currently, there are many technical solutions for composing music for videos, but the generated music usually achieves the ability to match the video globally, but cannot achieve detailed matching of the video music.

[0003] Therefore, there is an urgent need for an audio generation model that also matches the video details to solve the above technical problems. Summary of the invention

[0004] In view of this, an embodiment of the present specification provides an audio generation model training method. One or more embodiments of the present specification also relate to an audio generation model training device, an audio generation method, an information processing method based on an audio generation model, an audio generation device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defect that the audio and video do not match in details after the music is generated in the prior art.

[0005] According to a first aspect of an embodiment of this specification, there is provided an audio generation model training method, comprising:

[0006] Determine a video sample and an audio sample corresponding to the video sample;

[0007] Determining a target audio feature sample according to the audio feature sample of the audio sample and a preset audio feature sample;

[0008] Determine a target audio and video feature sample according to the target audio feature sample and the video feature sample of the video sample;

[0009] An audio generation model is trained based on the target audio and video feature samples, the audio feature samples, and the preset audio feature samples, wherein the audio generation model is used to output the target audio corresponding to the target video based on the input target video and the preset audio features.

[0010] According to a second aspect of the embodiments of this specification, there is provided an audio generation method, including:

[0011] Determine target video and preset audio features;

[0012] The target video and the preset audio features are input into an audio generation model to obtain a target audio corresponding to the target video, wherein the audio generation model is trained by the above-mentioned audio generation model training method.

[0013] According to a third aspect of the embodiments of this specification, an information processing method based on an audio generation model is provided, which is applied to a task platform and includes:

[0014] Receiving a model request sent by a terminal device, wherein the model request includes a scene identifier of a target scene, scene input data of the target scene, and at least one of a model specification parameter;

[0015] Based on the model request, a target audio generation model is determined from multiple audio generation models, wherein the multiple audio generation models are trained based on the above-mentioned audio generation model training method.

[0016] According to a fourth aspect of the embodiments of this specification, there is provided an audio generation model training device, comprising:

[0017] A sample determination module, configured to determine a video sample and an audio sample corresponding to the video sample;

[0018] An audio feature determination module, configured to determine a target audio feature sample according to the audio feature sample of the audio sample and a preset audio feature sample;

[0019] An audio and video feature determination module, configured to determine a target audio and video feature sample based on the target audio feature sample and the video feature sample of the video sample;

[0020] The model training module is configured to train and obtain an audio generation model based on the target audio and video feature samples, the audio feature samples, and the preset audio feature samples, wherein the audio generation model is used to output the target audio corresponding to the target video based on the input target video and the preset audio features.

[0021] According to a fifth aspect of the embodiments of this specification, there is provided an audio generating device, configured to:

[0022] A determination module, configured to determine a target video and a preset audio feature;

[0023] The audio acquisition module is configured to input the target video and the preset audio features into an audio generation model to obtain the target audio corresponding to the target video, wherein the audio generation model is trained by the above-mentioned audio generation model training method.

[0024] According to a sixth aspect of an embodiment of this specification, a computing device is provided, including:

[0025] Memory and processor;

[0026] The memory is used to store computer programs / instructions, and the processor is used to execute computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above method are implemented.

[0027] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and the steps of the above method are implemented when the computer program / instruction is executed by a processor.

[0028] According to an eighth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0029] One or more embodiments of the present specification provide an audio generation model training method, which uses video feature samples as conditions to diffuse preset audio feature samples to audio feature samples through training the audio generation model, so that the video feature samples of each frame of video in the video sample can learn the information of the audio feature samples and the preset audio feature samples, so that the trained audio generation model can accurately match the detailed actions, style, rhythm, etc. in the target video, thereby improving the degree of detail matching between the target audio generated by the subsequent audio generation model and the target video. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a specific application scenario diagram of an audio generation method provided by an embodiment of this specification;

[0031] Figure 2 is a flowchart of an audio generation model training method provided by an embodiment of this specification;

[0032] Figure 3 is a flow chart of an audio generation method provided by an embodiment of this specification;

[0033] Figure 4 is a processing flow chart of an audio generation method provided by an embodiment of this specification;

[0034] Figure 5 is a processing flow chart of an audio generation model training method provided by an embodiment of this specification;

[0035] Figure 6 It is a structural schematic diagram of an audio generation model training device provided by an embodiment of this specification;

[0036] Figure 7is a structural schematic diagram of an audio generating device provided by an embodiment of this specification;

[0037] Figure 8 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0038] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0039] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0040] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0041] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0042] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM, Large Language Model), a multi-modal pre-training model, etc.

[0043] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image description (IC, Image Caption), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0044] First, the terms involved in one or more embodiments of this specification are explained.

[0045] Vision Transformer: Vision Transformer, ViT, is a deep learning model based on the Transformer architecture, which views an image as a series of fixed-size image patches and maps them into sequence vectors, which are then input into the Transformer encoder for processing, similar to the way text is processed in natural language processing (NLP).

[0046] DiT: Deep Iterative Training, a strategy that gradually optimizes the model in an iterative manner so that it can more accurately match the target data distribution.

[0047] Classifier Free Guidance: CFG, a technique used to guide the generation process in diffusion generative models and latent space models, is used to enhance the responsiveness of the generative model to specific conditions without using additional classifiers, thereby generating images or other modal data that are more in line with user intent.

[0048] Make-an-audio: A deep learning model designed to generate audio from text prompts. The model uses the architecture of a diffusion model and can receive text descriptions as input and generate corresponding audio content based on these descriptions. For example, if the input text description is "a cat is meowing, a young woman's voice", Make-An-Audio can generate audio of the corresponding scene, which will contain the sound of a cat meowing and the voice of a woman speaking.

[0049] HiFi-GAN: High-Fidelity Generative Adversarial Networks, a deep learning model based on Generative Adversarial Network (GAN), specifically designed for efficient and high-fidelity speech synthesis, used to generate synthetic audio that is almost indistinguishable from real audio while maintaining computational efficiency and reducing memory consumption.

[0050] V2A: Video-to-Audio, video to sound synthesis, a technology that synthesizes sound based on video.

[0051] V2M: Video-to-Music, video to music synthesis, a technology for synthesizing music based on video.

[0052] MIDI: Musical Instrument Digital Interface, generally used as a carrier for digital music scores.

[0053] DiffusionModel: Diffusion generation model, a model that uses random diffusion technology to generate samples.

[0054] RFM: RectifiedFlowMatching, a generation diffusion model based on optimal transmission theory.

[0055] VAE: VariationalAutoencoer, a variational autoencoder, an autoencoder based on variational inference.

[0056] GT: GroundTruth, true value, the reference standard used for comparison in model training.

[0057] Usually, in video production projects, music production occupies a relatively large proportion of importance. For example, the soundtrack of a video will greatly affect the viewing interest and video experience of video viewers. Therefore, there is a large project demand for video-to-music synthesis technology.

[0058] At present, music synthesis technology has achieved great success mainly in text-to-music synthesis technology, while video-to-music synthesis technology is still underdeveloped. For example, although the current video-to-music synthesis technology can focus on specific scenes, such as the soundtrack for dance videos, or can generate symbolic music scores or symbolic MIDIA, these methods cannot generate more music styles, and the generated music corresponds to the global video semantics in a broad sense, and cannot achieve the effects of music and video in fine-grained rhythm correspondence, control, etc.

[0059] In view of this, in this specification, an audio generation model training method is provided. One or more embodiments of this specification also relate to an audio generation model training device, an audio generation method, an information processing method based on an audio generation model, an audio generation device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0060] See also Figure 1 , Figure 1 A specific application scenario diagram of an audio generation method provided by an embodiment of this specification is shown.

[0061] In such Figure 1 In the application scenario shown, the audio generation model is deployed in the server 104, which includes but is not limited to a physical server, a cloud server, etc. The server 104 can be connected to one or more clients 102 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client 102 here can include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a car device, etc. The client 102 can interact with the user through the user interaction interface to obtain the user's needs, and through the use of the audio generation model by the server 104, the audio generation method provided in the embodiment of this specification is implemented.

[0062] In the embodiment of this specification, the server 104 trains the audio generation model, and the execution steps are as follows:

[0063] Determine a video sample and an audio sample corresponding to the video sample;

[0064] Determining a target audio feature sample according to the audio feature sample of the audio sample and a preset audio feature sample;

[0065] Determine a target audio and video feature sample according to the target audio feature sample and the video feature sample of the video sample;

[0066] An audio generation model is trained based on the target audio and video feature samples, the audio feature samples, and the preset audio feature samples, wherein the audio generation model is used to output the target audio corresponding to the target video based on the input target video and the preset audio features.

[0067] After obtaining the trained audio generation model, the server 104 can receive the target video sent by the client 102, determine the preset audio features, input the target video and the preset audio features into the audio generation model, obtain the target audio corresponding to the target video, and send the target audio to the client 102. The client 102 displays the target audio to the user through a graphical interactive interface to meet the user's audio generation needs.

[0068] It should be noted that, when the client's operating resources can meet the deployment and operating conditions of the audio generation model, the embodiments of the present application can be carried out in the client. In addition, the audio generation model can also be deployed in a third-party server, which includes but is not limited to a third-party physical server, a third-party cloud server, etc.

[0069] One or more embodiments of the present specification provide a method for training an audio generation model, in which a server uses video feature samples as conditions to diffuse initial audio feature samples to audio feature samples by training the audio generation model, so that the trained audio generation model can accurately match the detailed actions, style, rhythm, etc. in the target video, thereby improving the degree of detail matching between the target audio generated by the audio generation model and the target video sent by the client.

[0070] See also Figure 2 , Figure 2 A flowchart of an audio generation model training method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0071] Step 202: Determine a video sample and an audio sample corresponding to the video sample.

[0072] Among them, video samples can be understood as samples in video format, including but not limited to videos uploaded by users, surveillance videos used in compliance with regulations, recorded videos, demonstration videos, animation videos, movie videos, live broadcast videos, etc.

[0073] The audio sample corresponding to the video sample can be understood as an audio sample that matches the video sample with a high degree of detail matching. For example, a video sample containing a kitten can correspond to an audio sample containing the kitten's cry, the kitten's playing sound, etc. A video sample containing a rain scene can correspond to an audio sample containing the sound of rain, thunder, etc.

[0074] In practical applications, after obtaining video samples and audio samples, the audio generation model can be used to extract features from the video samples to analyze the semantics of the video samples. The specific implementation method is as follows:

[0075] The audio generation model includes a video encoding unit;

[0076] After determining the video sample and the audio sample corresponding to the video sample, the method further includes:

[0077] According to the video encoding unit, video features are extracted from the video sample to obtain a video feature sample corresponding to the video sample.

[0078] The video encoding unit may be understood as a unit capable of encoding a video into a video feature vector. For example, the video encoding unit may be understood as a video encoder (Video Enc.), and the video feature sample may be understood as a video feature vector sample corresponding to the video sample.

[0079] In order to improve the effect of video encoding, the video encoding unit can be pre-trained in advance to obtain a pre-trained video encoding unit, and then video features are extracted from the video sample based on the pre-trained video encoding unit to obtain a video feature sample corresponding to the video sample.

[0080] In a specific implementation, according to the video encoding unit, video features are extracted from the video sample to obtain a video feature sample corresponding to the video sample. It can be understood that the video sample is input into the video encoding unit, the video encoding unit extracts or encodes the video features of the video sample, and outputs a video feature sample corresponding to the video sample.

[0081] It should be noted that the video encoding unit can be deployed in the audio generation model, or it can be deployed outside the audio generation model and deployed in a server, or deployed in a third-party server, and the video encoding unit can be called through an interface call.

[0082] One or more embodiments of the present specification provide an audio generation model training method, which reduces the amount of training parameters of the entire audio generation model by utilizing a pre-trained video encoding unit to extract features from video samples, and extracts features from video samples so that analysis of video samples can be achieved by analyzing video feature samples, thereby improving the accuracy of video sample analysis and the granularity of audio and video processing of subsequent audio generation models.

[0083] In practical applications, after obtaining video samples and audio samples, the audio generation model can also be used to process the audio samples to analyze the semantics of the audio samples. The specific implementation method is as follows:

[0084] The audio generation model includes an audio conversion unit and an audio encoding unit;

[0085] After determining the video sample and the audio sample corresponding to the video sample, the method further includes:

[0086] According to the audio conversion unit, the audio sample is converted to obtain an audio spectrum sample corresponding to the audio sample;

[0087] According to the audio encoding unit, audio features are extracted from the audio spectrum samples to obtain audio feature samples corresponding to the audio samples.

[0088] Among them, the audio conversion unit can be understood as a data processing unit that converts audio into an audio spectrum. For example, the audio conversion unit can be understood as a vocoder; the audio encoding unit can be understood as a data processing unit that encodes the audio spectrum to obtain a feature vector corresponding to the audio spectrum. For example, the audio encoding unit can be understood as an audio encoder (MelEnc.); the audio spectrum can be understood as the representation of the audio signal in the frequency domain, and the audio spectrum shows the intensity distribution of different frequency components in the audio. For example, the audio spectrum can be understood as a Mel spectrum, a Fourier transform spectrum, etc.

[0089] It should be noted that the audio conversion unit or the audio encoding unit can also be deployed in the audio generation model, or can be deployed outside the audio generation model and deployed in a server, or deployed in a third-party server, and the audio conversion unit or the audio encoding unit can be called through an interface call.

[0090] During specific implementation, the specific implementation method of performing data processing according to the audio conversion unit and the audio encoding unit can refer to the specific implementation method of performing video feature extraction according to the video encoding unit mentioned above, which will not be repeated here.

[0091] One or more embodiments of the present specification provide an audio generation model training method, which converts audio samples into audio spectra, that is, models the shallow representation of audio waveforms (audio samples), so that the trained audio generation model can directly obtain the audio waveform directly through the generated audio spectrum using an audio conversion unit without the need for intermediate representations such as musical scores, thereby making the audio form richer and more diverse.

[0092] In practical applications, unconditional video features can also be added during the training of the audio generation model to enable the audio generation model to have both conditional and unconditional audio generation capabilities, which facilitates subsequent adjustment of the application of the audio generation model. The specific implementation method is as follows:

[0093] The audio generation model includes a video processing unit;

[0094] After determining the video sample and the audio sample corresponding to the video sample, the method further includes:

[0095] According to the video encoding unit, video feature extraction is performed on the video sample to obtain an initial video feature sample corresponding to the video sample;

[0096] Determining a reference video feature sample and a preset feature replacement strategy for a reference video, wherein the reference video is a video different from the video sample;

[0097] According to the video processing unit, in combination with the preset feature replacement strategy, the reference video feature sample is used to replace the initial video feature sample to obtain the video feature sample of the video sample.

[0098] Among them, the video processing unit can be understood as a data processing unit that performs video frame replacement, video fragment replacement, video frame addition, video frame deletion and other processing on the video. The preset feature replacement strategy includes but is not limited to a preset replacement percentage, a preset replacement position, a preset replacement duration, etc.

[0099] Specifically, one frame can be selected from the video sample and repeated to obtain a reference video, or a preset video frame can be repeated to obtain a reference video. To improve the accuracy of subsequent replacement, the duration of the video sample can be equal to that of the reference video.

[0100] The specific implementation method of determining the initial video feature samples of the video samples and the specific implementation method of determining the reference video feature samples of the reference video can refer to the specific implementation method of determining the video feature samples of the video samples mentioned above, which will not be repeated here.

[0101] In specific implementation, after determining the initial video feature sample and the reference video feature sample, one or more of the above-mentioned preset feature replacement strategies can be used to replace the target video features in the reference video feature sample with the initial video feature sample, thereby obtaining a video feature sample of the video sample.

[0102] One or more embodiments of the present specification provide a method for training an audio generation model, which uses a preset feature replacement strategy to obtain a video feature sample containing conditional video features and unconditional video features, wherein the conditional video feature can be understood as the video feature and the audio feature corresponding to the sample pair consisting of the video sample and the audio sample corresponding to the video sample, and the conditional feature corresponding to the sample pair can enable the model to learn the ability to generate audio based on the video, and the unconditional video feature can be understood as the video feature sample obtained after the above replacement and the audio feature sample of the audio corresponding to the replaced video, and the unconditional video feature obtained after the replacement can enable the model to learn the ability to generate audio even when the video clip is lost in the video. As a result, the audio generation model obtained by training with the video feature sample as a condition can have both unconditional and conditional generation capabilities, and the audio generation model obtained by subsequent training can take into account both diversity and rhythm, thereby improving the realism of the target audio generated by the subsequent audio generation model.

[0103] Step 204: Determine a target audio feature sample according to the audio feature sample of the audio sample and a preset audio feature sample.

[0104] Among them, the preset audio feature samples can be understood as preset noise feature samples. For example, the preset audio feature samples can be obtained by selecting fragments from audio features that obey Gaussian distribution, or the preset audio feature samples can be obtained by selecting fragments from audio features that obey normal distribution, etc. In specific implementation, the preset audio feature samples can be selected according to actual needs.

[0105] In specific implementation, the target audio feature samples are determined based on the audio feature samples of the audio samples and the preset audio feature samples. This can be achieved by weighted summing the audio feature samples of the audio samples and the preset audio feature samples, or by linearly interpolating the audio feature samples of the audio samples and the preset audio feature samples.

[0106] In one or more embodiments of the present specification, a specific implementation method of determining a target audio feature sample according to an audio feature sample of an audio sample and a preset audio feature sample by linear interpolation is as follows:

[0107] The step of determining a target audio feature sample according to the audio feature sample of the audio sample and a preset audio feature sample includes:

[0108] According to the audio feature sample and the preset audio feature sample, a target audio feature sample is calculated using a preset random number, wherein the preset random number is updated according to a training iteration round of the audio generation model.

[0109] Among them, the preset random number can be understood as a preset random number randomly obtained from 0 to 1. The preset random number can be obtained in advance through a random number generation tool or a random number generation function, and the preset random number can be obtained again in each training iteration round.

[0110] Specifically, the audio generation model may include an audio feature processing unit, and utilize the audio feature processing unit to determine the target audio feature sample according to the audio feature sample of the audio sample and the preset audio feature sample.

[0111] In specific implementation, based on the audio feature sample and the preset audio feature sample, the target audio feature sample is calculated using a preset random number. It can be understood that the first product of the audio feature sample and the preset random number is calculated, the second product of the preset audio feature sample and the difference between 1 and the preset random number is calculated, the first product is added to the second product, and the sum is determined as the target audio feature sample.

[0112] One or more embodiments of the present specification provide an audio generation model training method, which reduces the cost of constructing a large number of training samples and improves the accuracy of audio generation model training by randomly selecting an audio feature distribution between an audio feature sample and a preset audio feature sample, i.e., a target audio feature sample, as an audio sample for training the audio generation model.

[0113] Step 206: Determine a target audio and video feature sample according to the target audio feature sample and the video feature sample of the video sample.

[0114] The target audio and video feature samples may be understood as audio feature samples containing video feature information.

[0115] In specific implementation, the target audio and video feature samples are determined based on the target audio feature samples and the video feature samples of the video samples. This can be achieved by fusing the target audio feature samples with the video feature samples, or by inputting the target audio feature samples and the video feature samples into an audio and video processing model. This specification does not limit this embodiment.

[0116] In practical applications, the specific implementation method of fusing the target audio feature sample with the video feature sample is as follows:

[0117] The audio generation model includes an audio and video feature fusion unit;

[0118] The step of determining a target audio and video feature sample according to the target audio feature sample and the video feature sample of the video sample includes:

[0119] According to the audio and video feature fusion unit, the target audio feature sample and the video feature sample of the video sample are fused to obtain the target audio and video feature sample.

[0120] Among them, the audio and video feature fusion unit can be understood as a data processing unit used to unify the audio feature and video feature dimensions and then connect or splice them.

[0121] In a specific implementation, according to the audio and video feature fusion unit, the target audio feature samples and the video feature samples of the video samples are fused. One implementation method for obtaining the target audio and video feature samples is to input the target audio feature samples and the video feature samples into the audio and video feature fusion unit, and the audio and video feature fusion unit splices the target audio feature samples and the video feature samples in the channel dimension, and uses a fully connected layer to map the splicing result to the same dimension as the target audio feature sample to obtain the target audio and video feature sample.

[0122] According to the audio and video feature fusion unit, the target audio feature samples and the video feature samples of the video samples are fused. Another implementation method for obtaining the target audio and video feature samples is to adjust the dimension of the video feature samples so that the dimension of the adjusted video feature samples is the same as the dimension of the target audio feature samples, and then add the adjusted video feature samples to the target audio feature samples to obtain the target audio and video feature samples.

[0123] One or more embodiments of the present specification provide an audio generation model training method, which integrates video information in video feature samples into target audio and video feature samples, so that the training process of the audio generation model can take video information into consideration, thereby improving the matching degree and rhythm synchronization degree between the finally generated target audio and target video.

[0124] Step 208: Train and obtain an audio generation model based on the target audio and video feature samples, the audio feature samples, and the preset audio feature samples.

[0125] The audio generation model is used to output a target audio corresponding to the target video according to an input target video and preset audio features.

[0126] Specifically, according to the target audio and video feature samples, audio feature samples, and preset audio feature samples, training to obtain the audio generation model can be understood as taking the target audio and video feature samples as training samples, taking the difference between the audio feature samples and the preset audio feature samples as training labels, training the audio feature processing unit in the audio generation model, and obtaining the audio feature processing unit when the training is completed.

[0127] In practical applications, the specific implementation of audio feature processing unit training is as follows:

[0128] The audio generation model includes an audio feature processing unit;

[0129] The step of training and obtaining an audio generation model according to the target audio and video feature sample, the audio feature sample, and the preset audio feature sample includes:

[0130] Determining an actual feature difference between the audio feature sample and the preset audio feature sample;

[0131] Inputting the target audio and video feature sample into the audio feature processing unit to obtain a predicted feature difference value output by the audio feature processing unit;

[0132] According to the predicted feature difference and the actual feature difference, the audio feature processing unit is adjusted to obtain the trained audio generation model.

[0133] Among them, the predicted feature difference represents the predicted direction corresponding to the target audio and video feature sample. The direction can be understood as the direction of change in the feature distribution of the audio feature sample pointed to by the preset audio feature sample. The actual feature difference between the audio feature sample and the preset audio feature sample represents the actual direction of change in the feature distribution of the audio feature sample pointed to by the preset audio feature sample.

[0134] In specific implementation, after the target audio and video feature samples are determined as mentioned above, the audio feature processing unit can be used to predict the target audio and video feature samples to obtain the predicted feature difference. Then, based on the predicted feature difference and the actual feature difference, the deviation in the direction predicted by the audio feature processing unit can be determined, thereby adjusting the audio feature processing unit until the adjustment is completed, and the trained audio generation model can be obtained.

[0135] One or more embodiments of the present specification provide an audio generation model training method, which reduces the training cost of the audio generation model by adjusting the audio feature processing unit using predicted feature differences and actual feature differences without adjusting the entire audio generation model, and improves the training accuracy of the audio generation model by adjusting the parameters of the audio feature processing unit (gradient return) using predicted feature differences and actual feature differences in each training iteration round.

[0136] In practical applications, the predicted feature difference and the actual feature difference can be used to determine the loss of the audio feature processing unit, so as to more accurately adjust the audio feature processing unit. The specific implementation method is as follows:

[0137] The step of adjusting the audio feature processing unit according to the predicted feature difference and the actual feature difference to obtain the trained audio generation model includes:

[0138] Determining a loss function according to the predicted feature difference and the actual feature difference;

[0139] When it is determined that the loss function does not meet the preset loss function threshold, the audio feature processing unit is adjusted according to the loss function, and the step of determining the target audio feature sample based on the audio feature sample of the audio sample and the preset audio feature sample is continued until it is determined that the loss function meets the preset loss function threshold, and the trained audio generation model is obtained.

[0140] Among them, the loss function can be understood as the loss function value between the predicted feature difference and the actual feature difference. The loss function can be used to feedback and adjust the training process of the audio feature processing unit to improve the accuracy of the audio generation model.

[0141] The training stop conditions of the audio generation model include but are not limited to the loss function satisfying the preset loss function threshold, the training iteration rounds reaching the preset number threshold, the training time reaching the preset time threshold, etc. The training stop conditions can be set according to actual project requirements.

[0142] The training stop condition of the audio generation model is that the loss function satisfies a preset loss function threshold as an example for exemplary description. For other embodiments, please refer to the embodiments of this specification.

[0143] After the predicted feature difference and the actual feature difference are determined as mentioned above, a loss function between the predicted feature difference and the actual feature difference can be determined based on the predicted feature difference and the actual feature difference. When the loss function meets a preset threshold, it indicates that the direction predicted by the audio feature processing unit is sufficiently accurate. Based on the audio feature processing unit with a sufficiently accurate prediction direction, an audio generation model including an audio feature processing unit with a sufficiently accurate prediction direction can be obtained.

[0144] When the loss function does not meet the preset loss function threshold, the audio feature processing unit can continue to be adjusted, that is, the audio feature processing unit is adjusted according to the loss function, and the step of determining the target audio feature sample according to the audio feature sample of the audio sample and the preset audio feature sample is continued until the loss function meets the preset loss function threshold, thereby obtaining the trained audio generation model.

[0145] One or more embodiments of the present specification provide an audio generation model training method, which greatly improves the adjustment accuracy of the audio feature processing unit by adjusting the audio feature processing unit using a loss function, thereby improving the accuracy of the audio generation model.

[0146] One or more embodiments of the present specification provide a method for training an audio generation model, which uses video feature samples as conditions to diffuse initial audio feature samples to audio feature samples by training the audio generation model, so that the trained audio generation model can accurately match the detailed actions, style, rhythm, etc. in the target video, thereby improving the degree of detail matching between the target audio generated by the subsequent audio generation model and the target video.

[0147] See also Figure 3 , Figure 3 A flowchart of an audio generation method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0148] Step 302: Determine the target video and preset audio features.

[0149] The target video may be understood as a video used as a reference for audio generation, for example, a video to be accompanied by music, a video to be dubbed, and the like.

[0150] The preset audio features may be understood as the preset audio feature samples mentioned above, or may be understood as other audio features preset according to actual projects.

[0151] In order to improve the user experience, the embodiments of this specification can meet the user's needs of generating music corresponding to the video according to the video through the interface interaction between the user, and the specific implementation method is as follows:

[0152] Before determining the target video and presetting the audio features, the method further includes:

[0153] Receiving the target video sent by the client, wherein the target video is generated by a trigger operation of a user on a user interaction interface of the client;

[0154] After obtaining the target audio corresponding to the target video, the method further includes:

[0155] The target audio is sent to the client, so that the client displays the target audio to the user through the user interaction interface.

[0156] Herein, the client can be understood as the above-mentioned client 102.

[0157] The user interaction interface can be understood as the interface for the user to interact with the client, including but not limited to the dialog box interaction interface, the graphical interaction interface, etc.; the trigger operation includes but is not limited to the drop-down list selection operation, the radio button selection operation, the check box selection operation, the list or grid selection operation, the slider selection operation, the voice command selection operation, the keyboard input selection operation, the interface area selection operation, the label or category selection operation, the compressed file upload operation, the video file upload operation, the database data upload operation, the encrypted file upload operation, the peripheral typing operation, etc.

[0158] Specifically, the client can display multiple selectable videos to the user through a user interaction interface, and the user should select a target video, or an upload area can be provided for the user to upload a video. The target video is determined based on this and the target video is sent to the server. The server generates a target audio corresponding to the target video through the following audio generation method, and displays the target audio on the user interaction interface of the client to respond to user needs.

[0159] One or more embodiments of the present specification provide an audio generation method that interacts with a user through a client, so that the user can more intuitively determine the target video for which he or she needs to generate audio, thereby improving user interactivity and making the target audio more in line with user needs, thereby improving the user's usage experience.

[0160] Step 304: Input the target video and the preset audio features into an audio generation model to obtain a target audio corresponding to the target video.

[0161] Wherein, the audio generation model is obtained by training using the above-mentioned audio generation model training method.

[0162] In one or more embodiments of the present specification, the step of inputting the target video and the preset audio features into an audio generation model to obtain a target audio corresponding to the target video includes:

[0163] Inputting the target video and the preset audio features into an audio generation model to obtain target audio features;

[0164] The target audio feature is processed to obtain a target audio corresponding to the target video.

[0165] Specifically, the target video and preset audio features may be input into an audio generation model to obtain target audio features, and the target audio features may be converted to obtain target audio corresponding to the target video.

[0166] During specific implementation, a feature conversion model may be used to achieve conversion from a target audio feature model to a target audio, or other methods may be used for feature processing, which is not limited in the embodiments of this specification.

[0167] In one or more embodiments of the present specification, the step of inputting the target video and the preset audio features into an audio generation model to obtain the target audio features includes:

[0168] Inputting the target video and the preset audio features into an audio generation model;

[0169] In the audio generation model, video feature extraction is performed on the target video to obtain target video features corresponding to the target video;

[0170] The target video feature and the preset audio feature are integrated to obtain the target audio and video feature;

[0171] According to a preset number of iteration steps, feature processing is performed on the target audio and video features to obtain target audio features.

[0172] The preset number of iteration steps can be understood as the preset number of iteration steps for generating the audio distribution of the target audio feature from the audio distribution of the preset audio feature. The specific structure of the audio generation model can refer to the structure of the audio generation model described in the aforementioned audio generation model training method.

[0173] In one or more embodiments of this specification, the preset number of iteration steps is T, and T is a positive integer;

[0174] The step of performing feature processing on the target audio and video features according to a preset number of iteration steps to obtain target audio features includes:

[0175] Determine the current iteration step number, where the current iteration step number is i, where i is a positive integer and starts from 1;

[0176] When i is 1, the target audio and video feature is determined as the input audio feature of the i-th step;

[0177] According to the input audio feature of the i-th step, obtaining the output audio feature of the i-th step;

[0178] Determine whether i is equal to T,

[0179] If not, the output audio feature of the i-th step is determined as the input audio feature of the i+1-th step, and i is incremented by 1, and the step of obtaining the output audio feature of the i-th step according to the input audio feature of the i-th step is continued.

[0180] If so, the output audio feature of the i-th step is determined as the target audio feature.

[0181] The step of obtaining the output audio feature of the i-th step according to the input audio feature of the i-th step includes:

[0182] According to the input audio features of the i-th step, obtaining the initial output audio features of the i-th step;

[0183] Determining reference video features and preset weights;

[0184] The output audio feature of the i-th step is obtained by calculation according to the reference video feature, the preset weight, and the initial output audio feature of the i-th step.

[0185] The preset weight can be understood as the weight between the preset reference video feature and the initial output audio feature of the i-th step. The reference video feature can be understood as the video feature of the reference video.

[0186] By setting preset weights and adding controllable reference video features to the initial output audio features of the i-th step, the controllable reference video features can be used to control the conditional strength of the audio generation reasoning process, thereby achieving compatibility between the rhythm synchronization degree and the diversity degree of the target audio and the target video, thereby improving the naturalness of the target audio.

[0187] In one or more embodiments of the present specification, the processing of the audio features to obtain the target audio corresponding to the target video includes:

[0188] Processing the target audio feature to obtain a target audio spectrum corresponding to the target audio feature;

[0189] The target audio spectrum is processed to obtain a target audio corresponding to the target video.

[0190] Specifically, the target audio feature may be subjected to feature conversion processing to obtain a target audio spectrum corresponding to the target audio feature, and the target audio spectrum may be subjected to spectrum conversion to obtain a target audio corresponding to the target video.

[0191] In practical applications, the generated target audio features (i.e., music latent features) can be reconstructed back into a target audio spectrum (i.e., Mel spectrum) using the VAE decoder, and the target audio spectrum can be converted back into the target audio (i.e., music waveform) using a pre-trained vocoder.

[0192] In one or more embodiments of the present specification, the step of inputting the target video and the preset audio features into an audio generation model to obtain a target audio corresponding to the target video includes:

[0193] According to the audio generation model interface, the audio generation model is called, and the target video and the preset audio features are input into the audio generation model. In the audio generation model, the target audio corresponding to the target video is obtained by using the audio generation model according to the target video and the preset audio features.

[0194] In specific implementation, the specific implementation method of the audio generation model in the embodiment of this specification to process the target video to obtain the target audio can refer to the specific implementation method of the above-mentioned audio generation model training method. The only difference is the order of steps and the processing objects, which will not be repeated here.

[0195] One or more embodiments of the present specification provide an audio generation method, which integrates the video information of a target video into the audio generation process so that the finally generated target audio better matches the target video and is more synchronized with the rhythm of the target video. In addition, by referring to video features and preset weights, the conditional strength control of the audio generation reasoning process is achieved, so that the audio generation model can learn the feature information of the reference video according to the preset weight information, so that the target audio output by the audio generation model can take into account the degree of rhythm synchronization and diversity with the target video, thereby improving the naturalness of the target audio. Furthermore, the entire process of the audio generation method does not require multi-level audio intermediate feature analysis, thereby improving the speed and efficiency of audio generation.

[0196] The following combination Figure 4 , further illustrating the audio generation method provided in this specification, wherein, Figure 4 A processing flow chart of an audio generation method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0197] Step 402: Video encoding.

[0198] During specific implementation, the target video input by the user is first confirmed, and the target video is encoded using a video encoder to obtain target video features corresponding to the target video.

[0199] In order to improve the subsequent synchronization between the target audio and the target video, the video encoder can be understood as a self-supervised trained video encoding neural network model, such as the VideoMAE video encoder.

[0200] Step 404: Determine reference audio features.

[0201] The reference audio feature can be determined by a preset number of steps and an initial reference audio feature. The initial reference audio feature is understood as an unconditional audio feature, such as a reference audio feature that satisfies a Gaussian distribution. Figure 4The x0 shown in the figure is based on the preset number of steps (such as Figure 4 T), the reference audio features can be determined, such as Figure 3 shown Among them, i is the number of iterations, the value range of i is 1~T, t=i / T, the initial time

[0202] Step 406: audio and video fusion.

[0203] Among them, audio and video fusion can be understood as fusing the above reference audio features with the target video features to obtain the target audio and video features (such as Figure 4 W shown in ), for example, the above-mentioned reference audio features and target video features are spliced ​​in the channel dimension, and then the spliced ​​features are mapped to the same dimension as the reference audio features using a fully connected layer to obtain the target audio and video features. For another example, the target video features are dimensionally adjusted so that the dimension of the adjusted target video features is the same as the dimension of the reference audio features, and then the adjusted target video features are added to the reference audio features to obtain the target audio and video features, and so on.

[0204] Step 408: Audio generation model prediction.

[0205] Among them, the audio generation model can be understood as a rectified flow matching model (Rectified Flow Matching, RFM).

[0206] In the specific implementation, taking the i-th iteration as an example, that is, t = i / T, the above target audio and video features are input into the audio generation model to obtain the predicted audio feature difference (i.e., u) output by the audio generation model, and the predicted audio feature difference (i.e., u), the preset number of steps, and the reference audio feature (i.e., ), determine the audio features (i.e. ).

[0207] If the current number of iterations is less than the preset number of steps, the audio feature (i.e. ) as reference audio features.

[0208] And continue to execute steps 406-408 until the current number of iterations is equal to the preset step number position, and obtain the target audio feature (i.e. ).

[0209] In one or more embodiments of the present specification, in order to improve the effect of the final generated audio, the audio generation process may be controlled in terms of strength.

[0210] Specifically, a reference video may be determined, and the reference video may be used as a target video, and the above steps 402 to 408 may be executed to obtain initial reference audio features corresponding to the reference video.

[0211] Afterwards, the initial reference audio feature is weighted and summed with the above reference audio feature (the weight is set according to the actual project situation) to obtain the updated reference audio feature. According to the updated reference audio feature, steps 406-408 are continued to be executed, and the strength control is performed in each iteration process until the target audio feature (i.e. ).

[0212] Step 410: Audio feature decoding.

[0213] Specifically, after the target audio feature is obtained as described above, an audio decoder (such as a VAE decoder) is used to decode the target audio feature to obtain a Mel spectrum corresponding to the target audio feature.

[0214] Step 412: Generate target audio.

[0215] Specifically, the Mel spectrum obtained above is converted into an audio waveform using a vocoder (such as HiFi-GAN) to obtain a target audio corresponding to the Mel spectrum, that is, a target audio that matches the target video.

[0216] One or more embodiments of the present specification provide an audio generation method, which integrates the video information of a target video into the audio generation process so that the target audio finally generated is more matched and the rhythm is more synchronized. In addition, by performing weighted summation of initial reference audio features and reference audio features to update the reference audio features, the reference video can be used to implement the strength control of the conditions in the audio generation reasoning process, so that the target audio can take into account the degree of rhythm synchronization and diversity with the target video, thereby improving the naturalness of the target audio. Furthermore, the entire process of the audio generation method does not require multi-level audio intermediate feature analysis, thereby improving the speed and efficiency of audio generation.

[0217] In one or more embodiments of the present specification, in order to improve the quality of the target audio, the audio generation model can be pre-trained, and a pre-trained video encoder and voice coder can be used to reduce the number of parameters that need to be adjusted for model training, reduce training costs and improve training efficiency.

[0218] Based on this, the embodiment of this specification also provides an audio generation model training method, such as Figure 5 As shown, Figure 5 An audio generation model training method provided in an embodiment of this specification specifically includes the following steps.

[0219] Step 502: Video sample encoding.

[0220] In specific implementation, firstly, a video sample is confirmed, and then a video encoder is used to encode the video sample to obtain an initial video feature sample (conditional video feature sample) corresponding to the video sample.

[0221] Furthermore, a preset reference video feature sample (unconditional video feature sample) and a preset sample replacement percentage are determined, and the video feature samples in the initial video feature sample that meet the preset sample replacement percentage are replaced with video feature samples in the preset reference video feature sample to obtain a target video feature sample corresponding to the target video.

[0222] Step 504: Determine target audio feature samples.

[0223] The target audio feature sample may be understood as the audio feature of the audio sample corresponding to the video sample.

[0224] Step 506: Determine a reference audio feature sample.

[0225] The reference audio feature sample can be determined by a preset random number and an initial reference audio feature sample. The initial reference audio feature sample (i.e. Figure 5 x0) can satisfy the audio characteristics of Gaussian distribution, according to the preset random number (such as Figure 5 As shown in t, t is a random number between 0 and 1), the initial reference audio feature sample, the target audio feature sample, the reference audio feature sample can be determined, such as Figure 5 As shown, It is the reference audio feature sample.

[0226] Step 508: audio and video fusion.

[0227] Among them, audio and video fusion can be understood as fusing the above reference audio feature samples with the target video feature samples to obtain the target audio and video feature samples (such as Figure 5 W shown in ).

[0228] Step 510: Audio generation model training.

[0229] During specific implementation, the above-mentioned target audio and video feature samples are input into the audio generation model to obtain the predicted audio feature difference samples (u) output by the audio generation model, and the loss function is calculated based on the predicted audio feature difference samples (u), the target audio feature samples and the reference audio feature samples.

[0230] When it is determined that the loss function is greater than or equal to the preset loss function threshold, or the number of iterations is less than the preset number of iterations threshold, the preset random number is re-determined, and steps 506 to 510 are continued.

[0231] When it is determined that the loss function is less than a preset loss function threshold, or the number of iterations is greater than or equal to a preset iteration number threshold, the training of the audio generation model is completed.

[0232] One or more embodiments of the present specification provide an audio generation model training method, which trains the audio generation model to use the target video as a condition to diffuse the initial audio to the target audio, so that the quality of the trained audio generation model is higher and can accurately match the action, style, rhythm, etc. of the target video. By presetting the sample replacement percentage, the audio generation model has both unconditional and conditional generation capabilities, so that the audio generation model obtained by subsequent training can take into account both diversity and rhythm, thereby improving the realism of the target audio generated by the subsequent audio generation model.

[0233] Corresponding to the above method embodiment, this specification also provides an audio generation model training device embodiment, Figure 6 FIG. 1 is a schematic diagram showing the structure of an audio generation model training device provided by an embodiment of the present specification. Figure 6 As shown, the device comprises:

[0234] A sample determination module 602 is configured to determine a video sample and an audio sample corresponding to the video sample;

[0235] The audio feature determination module 604 is configured to determine a target audio feature sample according to the audio feature sample of the audio sample and a preset audio feature sample;

[0236] The audio and video feature determination module 606 is configured to determine a target audio and video feature sample based on the target audio feature sample and the video feature sample of the video sample;

[0237] The model training module 608 is configured to train and obtain an audio generation model based on the target audio and video feature samples, the audio feature samples, and the preset audio feature samples, wherein the audio generation model is used to output the target audio corresponding to the target video based on the input target video and the preset audio features.

[0238] Optionally, the audio generation model includes an audio feature processing unit;

[0239] The model training module 608 is further configured to:

[0240] Determining an actual feature difference between the audio feature sample and the preset audio feature sample;

[0241] Inputting the target audio and video feature sample into the audio feature processing unit to obtain a predicted feature difference value output by the audio feature processing unit;

[0242] According to the predicted feature difference and the actual feature difference, the audio feature processing unit is adjusted to obtain the trained audio generation model.

[0243] Optionally, the model training module 608 is further configured to:

[0244] Determining a loss function according to the predicted feature difference and the actual feature difference;

[0245] When it is determined that the loss function does not meet the preset loss function threshold, the audio feature processing unit is adjusted according to the loss function, and the step of determining the target audio feature sample based on the audio feature sample of the audio sample and the preset audio feature sample is continued until it is determined that the loss function meets the preset loss function threshold, and the trained audio generation model is obtained.

[0246] Optionally, the audio feature determination module 604 is further configured to:

[0247] According to the audio feature sample and the preset audio feature sample, a target audio feature sample is calculated using a preset random number, wherein the preset random number is updated according to a training iteration round of the audio generation model.

[0248] Optionally, the audio generation model includes a video encoding unit;

[0249] The device further includes a video encoding module configured to:

[0250] According to the video encoding unit, video features are extracted from the video sample to obtain a video feature sample corresponding to the video sample.

[0251] Optionally, the audio generation model includes an audio conversion unit and an audio encoding unit;

[0252] The device further includes an audio processing module configured to:

[0253] According to the audio conversion unit, the audio sample is converted to obtain an audio spectrum sample corresponding to the audio sample;

[0254] According to the audio encoding unit, audio features are extracted from the audio spectrum samples to obtain audio feature samples corresponding to the audio samples.

[0255] Optionally, the audio generation model includes a video processing unit;

[0256] The device further includes a video processing module configured to:

[0257] According to the video encoding unit, video feature extraction is performed on the video sample to obtain an initial video feature sample corresponding to the video sample;

[0258] Determining a reference video feature sample and a preset feature replacement strategy for a reference video, wherein the reference video is a video different from the video sample;

[0259] According to the video processing unit, in combination with the preset feature replacement strategy, the reference video feature sample is used to replace the initial video feature sample to obtain the video feature sample of the video sample.

[0260] Optionally, the audio generation model includes an audio and video feature fusion unit;

[0261] The audio and video feature determination module 606 is further configured to:

[0262] According to the audio and video feature fusion unit, the target audio feature sample and the video feature sample of the video sample are fused to obtain the target audio and video feature sample.

[0263] One or more embodiments of the present specification provide an audio generation model training device, which uses video feature samples as conditions to diffuse initial audio feature samples to audio feature samples by training the audio generation model, so that the trained audio generation model can accurately match the detailed actions, style, rhythm, etc. in the target video, thereby improving the degree of detail matching between the target audio generated by the subsequent audio generation model and the target video.

[0264] The above is a schematic scheme of an audio generation model training device of this embodiment. It should be noted that the technical scheme of the audio generation model training device and the technical scheme of the above audio generation model training method belong to the same concept, and the details not described in detail in the technical scheme of the audio generation model training device can be found in the description of the technical scheme of the above audio generation model training method.

[0265] Corresponding to the above method embodiment, this specification also provides an audio generation device embodiment, Figure 7 FIG. 1 is a schematic diagram showing the structure of an audio generating device provided by an embodiment of the present specification. Figure 7 As shown, the device comprises:

[0266] A determination module 702 is configured to determine a target video and preset audio features;

[0267] The audio acquisition module 704 is configured to input the target video and the preset audio features into an audio generation model to obtain a target audio corresponding to the target video, wherein the audio generation model is trained by the above-mentioned audio generation model training method.

[0268] Optionally, the audio acquisition module 704 is further configured to:

[0269] Inputting the target video and the preset audio features into an audio generation model to obtain target audio features;

[0270] The target audio feature is processed to obtain a target audio corresponding to the target video.

[0271] Optionally, the audio acquisition module 704 is further configured to:

[0272] Inputting the target video and the preset audio features into an audio generation model;

[0273] In the audio generation model, video feature extraction is performed on the target video to obtain target video features corresponding to the target video;

[0274] The target video feature and the preset audio feature are integrated to obtain the target audio and video feature;

[0275] According to a preset number of iteration steps, feature processing is performed on the target audio and video features to obtain target audio features.

[0276] Optionally, the preset number of iteration steps is T, and T is a positive integer;

[0277] The audio acquisition module 704 is further configured to:

[0278] Determine the current iteration step number, where the current iteration step number is i, where i is a positive integer and starts from 1;

[0279] When i is 1, the target audio and video feature is determined as the input audio feature of the i-th step;

[0280] According to the input audio feature of the i-th step, obtaining the output audio feature of the i-th step;

[0281] Determine whether i is equal to T,

[0282] If not, the output audio feature of the i-th step is determined as the input audio feature of the i+1-th step, and i is incremented by 1, and the step of obtaining the output audio feature of the i-th step according to the input audio feature of the i-th step is continued.

[0283] If so, the output audio feature of the i-th step is determined as the target audio feature.

[0284] Optionally, the audio acquisition module 704 is further configured to:

[0285] According to the input audio features of the i-th step, obtaining the initial output audio features of the i-th step;

[0286] Determining reference video features and preset weights;

[0287] The output audio feature of the i-th step is obtained by calculation according to the reference video feature, the preset weight, and the initial output audio feature of the i-th step.

[0288] Optionally, the audio acquisition module 704 is further configured to:

[0289] Processing the target audio feature to obtain a target audio spectrum corresponding to the target audio feature;

[0290] The target audio spectrum is processed to obtain a target audio corresponding to the target video.

[0291] Optionally, the device further includes an interaction module configured to:

[0292] Receiving the target video sent by the client, wherein the target video is generated by a trigger operation of a user on a user interaction interface of the client;

[0293] After obtaining the target audio corresponding to the target video, the method further includes:

[0294] The target audio is sent to the client, so that the client displays the target audio to the user through the user interaction interface.

[0295] Optionally, the audio acquisition module 704 is further configured to:

[0296] According to the audio generation model interface, the audio generation model is called, and the target video and the preset audio features are input into the audio generation model. In the audio generation model, the target audio corresponding to the target video is obtained by using the audio generation model according to the target video and the preset audio features.

[0297] One or more embodiments of the present specification provide an audio generation device, which integrates the video information of a target video into the audio generation process so that the finally generated target audio better matches the target video and is more synchronized with the rhythm of the target video. Furthermore, by referring to video features and preset weights, the conditional strength control of the audio generation reasoning process is achieved, so that the target audio can take into account the degree of rhythm synchronization and diversity with the target video, thereby improving the naturalness of the target audio. Furthermore, the entire process of the audio generation method does not require multi-level audio intermediate feature analysis, thereby improving the speed and efficiency of audio generation.

[0298] The above is a schematic scheme of an audio generation device of this embodiment. It should be noted that the technical scheme of the audio generation device and the technical scheme of the above audio generation method belong to the same concept, and the details not described in detail in the technical scheme of the audio generation device can be referred to the description of the technical scheme of the above audio generation method.

[0299] Figure 8 The block diagram of a computing device 800 according to an embodiment of the present specification is shown. The components of the computing device 800 include but are not limited to a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0300] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0301] In one embodiment of the present specification, the above components of the computing device 800 and Figure 8 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 8 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0302] The computing device 800 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 800 may also be a mobile or stationary server.

[0303] Among them, the memory 810 is used to store computer programs / instructions, and the processor 820 is used to execute the computer programs / instructions stored in the memory 810. When the computer program / instructions are executed by the processor, the steps of the above-mentioned audio generation model training method or audio generation method are implemented.

[0304] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the audio generation model training method or the audio generation method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the audio generation model training method or the audio generation method mentioned above.

[0305] An embodiment of the present specification also provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned audio generation model training method or audio generation method when executed by a processor.

[0306] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the audio generation model training method or the audio generation method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the audio generation model training method or the audio generation method.

[0307] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned audio generation model training method or audio generation method.

[0308] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the audio generation model training method or the audio generation method described above belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be found in the description of the technical scheme of the audio generation model training method or the audio generation method described above.

[0309] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0310] The computer program / instruction includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0311] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0312] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0313] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for training an audio generation model, comprising: Determine a video sample and an audio sample corresponding to the video sample; Determining a target audio feature sample according to the audio feature sample of the audio sample and a preset audio feature sample; Determine a target audio and video feature sample according to the target audio feature sample and the video feature sample of the video sample; An audio generation model is trained based on the target audio and video feature samples, the audio feature samples, and the preset audio feature samples, wherein the audio generation model is used to output the target audio corresponding to the target video based on the input target video and the preset audio features.

2. The audio generation model training method according to claim 1, wherein the audio generation model comprises an audio feature processing unit; The step of training and obtaining an audio generation model according to the target audio and video feature sample, the audio feature sample, and the preset audio feature sample includes: Determining an actual feature difference between the audio feature sample and the preset audio feature sample; Inputting the target audio and video feature sample into the audio feature processing unit to obtain a predicted feature difference value output by the audio feature processing unit; According to the predicted feature difference and the actual feature difference, the audio feature processing unit is adjusted to obtain the trained audio generation model.

3. The audio generation model training method according to claim 2, wherein the audio feature processing unit is adjusted according to the predicted feature difference and the actual feature difference to obtain the trained audio generation model, comprising: Determining a loss function according to the predicted feature difference and the actual feature difference; When it is determined that the loss function does not meet the preset loss function threshold, the audio feature processing unit is adjusted according to the loss function, and the step of determining the target audio feature sample based on the audio feature sample of the audio sample and the preset audio feature sample is continued until it is determined that the loss function meets the preset loss function threshold, and the trained audio generation model is obtained.

4. The audio generation model training method according to any one of claims 1 to 3, wherein determining a target audio feature sample based on the audio feature sample of the audio sample and the preset audio feature sample comprises: According to the audio feature sample and the preset audio feature sample, a target audio feature sample is calculated using a preset random number, wherein the preset random number is updated according to a training iteration round of the audio generation model.

5. The audio generation model training method according to any one of claims 1 to 3, wherein the audio generation model comprises a video encoding unit; After determining the video sample and the audio sample corresponding to the video sample, the method further includes: According to the video encoding unit, video features are extracted from the video sample to obtain a video feature sample corresponding to the video sample.

6. The audio generation model training method according to any one of claims 1 to 3, wherein the audio generation model comprises an audio conversion unit and an audio encoding unit; After determining the video sample and the audio sample corresponding to the video sample, the method further includes: According to the audio conversion unit, the audio sample is converted to obtain an audio spectrum sample corresponding to the audio sample; According to the audio encoding unit, audio features are extracted from the audio spectrum samples to obtain audio feature samples corresponding to the audio samples.

7. The audio generation model training method according to any one of claims 1 to 3, wherein the audio generation model comprises a video encoding unit; After determining the video sample and the audio sample corresponding to the video sample, the method further includes: According to the video encoding unit, video feature extraction is performed on the video sample to obtain an initial video feature sample corresponding to the video sample; Determining a reference video feature sample and a preset feature replacement strategy for a reference video, wherein the reference video is a video different from the video sample; According to the video processing unit, in combination with the preset feature replacement strategy, the reference video feature sample is used to replace the initial video feature sample to obtain the video feature sample of the video sample.

8. The audio generation model training method according to claim 1, wherein the audio generation model comprises an audio and video feature fusion unit; The step of determining a target audio and video feature sample according to the target audio feature sample and the video feature sample of the video sample includes: According to the audio and video feature fusion unit, the target audio feature sample and the video feature sample of the video sample are fused to obtain the target audio and video feature sample.

9. An audio generation method, comprising: Determine target video and preset audio features; The target video and the preset audio features are input into an audio generation model to obtain a target audio corresponding to the target video, wherein the audio generation model is obtained by training using the audio generation model training method described in any one of claims 1-8.

10. The audio generation method according to claim 9, wherein the step of inputting the target video and the preset audio features into an audio generation model to obtain the target audio corresponding to the target video comprises: Inputting the target video and the preset audio features into an audio generation model to obtain target audio features; The target audio feature is processed to obtain a target audio corresponding to the target video.

11. The audio generation method according to claim 10, wherein the step of inputting the target video and the preset audio features into an audio generation model to obtain the target audio features comprises: Inputting the target video and the preset audio features into an audio generation model; In the audio generation model, video feature extraction is performed on the target video to obtain target video features corresponding to the target video; The target video feature and the preset audio feature are integrated to obtain the target audio and video feature; According to a preset number of iteration steps, feature processing is performed on the target audio and video features to obtain target audio features.

12. The audio generation method according to claim 11, wherein the preset number of iteration steps is T, and T is a positive integer; The step of performing feature processing on the target audio and video features according to a preset number of iteration steps to obtain target audio features includes: Determine the current iteration step number, where the current iteration step number is i, where i is a positive integer and starts from 1; When i is 1, the target audio and video feature is determined as the input audio feature of the i-th step; According to the input audio feature of the i-th step, obtaining the output audio feature of the i-th step; Determine whether i is equal to T, If not, the output audio feature of the i-th step is determined as the input audio feature of the i+1-th step, and i is incremented by 1, and the step of obtaining the output audio feature of the i-th step according to the input audio feature of the i-th step is continued; If so, the output audio feature of the i-th step is determined as the target audio feature.

13. The audio generation method according to claim 12, wherein obtaining the output audio features of the i-th step according to the input audio features of the i-th step comprises: According to the input audio feature of the i-th step, obtaining the initial output audio feature of the i-th step; Determining reference video features and preset weights; The output audio feature of the i-th step is obtained by calculation according to the reference video feature, the preset weight, and the initial output audio feature of the i-th step.

14. The audio generation method according to claim 10, wherein the processing of the audio features to obtain the target audio corresponding to the target video comprises: Processing the target audio feature to obtain a target audio spectrum corresponding to the target audio feature; The target audio spectrum is processed to obtain a target audio corresponding to the target video.

15. The audio generation method according to claim 9, before determining the target video and presetting the audio features, further comprising: Receiving the target video sent by the client, wherein the target video is generated by a trigger operation of a user on a user interaction interface of the client; After obtaining the target audio corresponding to the target video, the method further includes: The target audio is sent to the client, so that the client displays the target audio to the user through the user interaction interface.

16. The audio generation method according to claim 9, wherein the step of inputting the target video and the preset audio features into an audio generation model to obtain the target audio corresponding to the target video comprises: According to the audio generation model interface, the audio generation model is called, and the target video and the preset audio features are input into the audio generation model. In the audio generation model, the target audio corresponding to the target video is obtained by using the audio generation model according to the target video and the preset audio features.

17. An information processing method based on an audio generation model, applied to a task platform, comprising: Receiving a model request sent by a terminal device, wherein the model request includes a scene identifier of a target scene, scene input data of the target scene, and at least one of a model specification parameter; Based on the model request, a target audio generation model is determined from a plurality of audio generation models, wherein the plurality of audio generation models are trained based on the audio generation model training method according to any one of claims 1 to 8.

18. An audio generation model training device, comprising: A sample determination module, configured to determine a video sample and an audio sample corresponding to the video sample; An audio feature determination module, configured to determine a target audio feature sample according to the audio feature sample of the audio sample and a preset audio feature sample; An audio and video feature determination module, configured to determine a target audio and video feature sample based on the target audio feature sample and the video feature sample of the video sample; The model training module is configured to train and obtain an audio generation model based on the target audio and video feature samples, the audio feature samples, and the preset audio feature samples, wherein the audio generation model is used to output the target audio corresponding to the target video based on the input target video and the preset audio features.

19. An audio generating device, configured to: A determination module, configured to determine a target video and a preset audio feature; The audio acquisition module is configured to input the target video and the preset audio features into an audio generation model to obtain a target audio corresponding to the target video, wherein: The audio generation model is obtained by training through the audio generation model training method described in any one of claims 1-8.

20. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 17 are implemented.

21. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 17.

22. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 17.

Images