Audio and video generation method, electronic device and computer readable storage medium

Abstract

The present application provides an audio and video generation method, an electronic device and a computer-readable storage medium, the method comprising: obtaining a target text, wherein the target text comprises a target text feature; inputting the target text into a pre-trained audio and video generation diffusion model to obtain a target video feature and a target audio feature output by the audio and video generation diffusion model; respectively decoding the target video feature and the target audio feature to obtain a target video and a target audio corresponding to the target text, since the audio and video generation diffusion model uses the video feature as a diffusion consideration condition in the diffusion transformation process of generating the target audio feature, and the audio and video generation diffusion model uses the audio feature as a diffusion consideration condition in the spatiotemporal diffusion transformation process of generating the target video feature, therefore, the generated target audio feature is associated with the video feature, so that the target video and audio generated by the scheme have high synchronization, thereby making the synchronization and quality of the audio and video higher.

Description

Technical Field

[0001] The present application relates to the technical field of audio and video conversion, and in particular to an audio and video generation method, an electronic device and a computer-readable storage medium. Background Art

[0002] Currently, audio generation and video generation mostly generate corresponding audio or video given text. There are currently two commonly used frameworks. One is to generate audio and video separately through the diffusion model framework; the other is to generate audio and video separately through the autoregressive structure.

[0003] In the diffusion model architecture, there is no structure that can generate audio and video at the same time. Text is input to generate audio and video separately, making the audio and video independent and unrelated. As a result, the generated audio and video have no direct correlation, which in turn causes the generated audio and video to be out of sync and of low quality. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an audio and video generation method, an electronic device and a computer-readable storage medium to solve the problem of audio and video being out of sync and of low quality caused by the lack of direct correlation between the audio and video generated by the current diffusion model framework.

[0005] In a first aspect, the present invention provides an audio and video generation method, the method comprising: obtaining a target text, wherein the target text includes a target text feature; inputting the target text into a pre-trained audio and video generation diffusion model to obtain a target video feature and a target audio feature output by the audio and video generation diffusion model, wherein the target video feature is generated by performing a spatiotemporal diffusion transformation according to initial video features, initial audio features and target text features through the audio and video generation diffusion model, the target audio feature is generated by performing a diffusion transformation according to initial audio features, initial video features and target text features through the audio and video generation diffusion model, the initial video feature is generated by performing a spatiotemporal diffusion transformation according to first Gaussian noise and target text features through the audio and video generation diffusion model, and the initial audio feature is generated by performing a diffusion transformation according to second Gaussian noise and target text features through the audio and video generation diffusion model; the target video feature and the target audio feature are respectively decoded to obtain a target video and a target audio corresponding to the target text.

[0006] The audio and video generation method designed above, this scheme first obtains the target text with text features, and then inputs the target text into the pre-trained audio and video generation diffusion model, and generates the target video features and target audio features corresponding to the output target text through the audio and video generation diffusion model, and then decodes the target video features and target audio features respectively to obtain the target video and target audio corresponding to the target text. Since the audio and video generation diffusion model takes the video features as diffusion consideration conditions in the diffusion transformation process of generating the target audio features, the generated target audio features are associated with the video features. Similarly, the audio and video generation diffusion model takes the audio features as diffusion consideration conditions in the spatiotemporal diffusion transformation process of generating the target video features. Therefore, the generated target video features are associated with the audio features, and then the generated target video features are associated with the target audio features, so that the target video features and the target audio features are associated, so that the target video features generated by this scheme are highly synchronized, and then the synchronization and quality of audio and video are higher.

[0007] In an optional implementation of the first aspect, the training process of the audio and video generation diffusion model includes: obtaining a training set, wherein the training set includes multiple samples, each sample includes a text sample feature and a video sample feature and an audio sample feature corresponding to the text sample feature; performing spatiotemporal segmentation on the video sample feature of each sample and adding a first Gaussian noise to obtain a latent space feature of the video noise of each sample; performing spatiotemporal segmentation on the audio sample feature of each sample and adding a second Gaussian noise to obtain a latent space feature of the audio noise of each sample; training a preset diffusion model according to the text sample features, the latent space features of the video noise, and the latent space features of the audio noise of the multiple samples to obtain the audio and video generation diffusion model.

[0008] In an optional implementation of the first aspect, a preset diffusion model is trained according to text sample features, latent space features of video noise, and latent space features of audio noise of multiple samples to obtain an audio and video generation diffusion model, including: obtaining a target diffusion step number; performing spatiotemporal diffusion transformation according to text sample features and latent space features of video noise to achieve noise removal, to obtain an initial video noise removal feature, and performing diffusion transformation noise removal according to text sample features and latent space features of audio noise to obtain an initial audio noise removal feature; determining whether the number of diffusion steps reaches a preset number of diffusion steps; if the number of diffusion steps reaches the preset number of diffusion steps, performing a spatiotemporal diffusion transformation according to the initial video noise removal feature, the initial audio noise removal feature, The noise removal features and text sample features are subjected to spatiotemporal diffusion transformation to achieve noise removal, and the initial video interaction features are obtained. Diffusion transformation is performed based on the initial video noise removal features, the initial audio noise removal features and the text sample features to achieve noise removal, and the initial audio interaction features are obtained. It is determined whether the number of diffusion steps reaches the target number of diffusion steps. If the number of diffusion steps reaches the target number of diffusion steps, the loss value of the preset diffusion model is calculated based on the initial video interaction features, the video sample features, the initial audio interaction features and the audio sample features. It is determined whether the loss value of the preset diffusion model reaches the target loss value. If the loss value of the preset diffusion model reaches the target loss value, the audio and video generation diffusion model is obtained.

[0009] In an optional implementation of the first aspect, after determining whether the number of diffusion steps reaches a preset number of diffusion steps, the method further includes: if it is determined that the number of diffusion steps does not reach the preset number of diffusion steps, performing a spatiotemporal diffusion transformation according to the initial video noise removal feature and the text sample feature to achieve noise removal, and obtaining a second video noise removal feature, and performing a diffusion transformation according to the initial audio noise removal feature and the text sample feature to achieve noise removal, and obtaining a second audio noise removal feature; determining whether the number of diffusion steps reaches the preset number of diffusion steps; if the number of diffusion steps reaches the preset number of diffusion steps, performing a spatiotemporal diffusion transformation according to the initial video noise removal feature and the text sample feature to achieve noise removal, and obtaining a second audio noise removal feature. The features are subjected to spatiotemporal diffusion transformation to achieve noise removal, and the initial video interaction features are obtained. Diffusion transformation is performed based on the second video noise removal features, the second audio noise removal features, and the text sample features to achieve noise removal, and the initial audio interaction features are obtained. It is determined whether the number of diffusion steps reaches the target number of diffusion steps. If the number of diffusion steps reaches the target number of diffusion steps, the loss value of the preset diffusion model is calculated based on the initial video interaction features, the video sample features, the initial audio interaction features, and the audio sample features. It is determined whether the loss value of the preset diffusion model reaches the target loss value. If the loss value of the preset diffusion model reaches the target loss value, the audio and video generation diffusion model is obtained.

[0010] In an optional implementation of the first aspect, after determining whether the number of diffusion steps has reached the target number of diffusion steps, the method further includes: if it is determined that the number of diffusion steps has not reached the target number of diffusion steps, performing a spatiotemporal diffusion transformation according to the initial video interaction features, the initial audio interaction features, and the text sample features to achieve noise removal, and obtaining a second video interaction feature, and performing a diffusion transformation according to the initial video interaction features, the initial audio interaction features, and the text sample features to achieve noise removal, and obtaining a second audio interaction feature; determining whether the number of diffusion steps has reached the target number of diffusion steps; if the number of diffusion steps has reached the target number of diffusion steps, calculating a loss value of a preset diffusion model according to the second video interaction features, the video sample features, the second audio interaction features, and the audio sample features; determining whether the loss value of the preset diffusion model has reached the target loss value; if the loss value of the preset diffusion model has reached the target loss value, obtaining an audio and video generation diffusion model.

[0011] In an optional implementation of the first aspect, after determining whether the loss value of the preset diffusion model reaches the target loss value, the method further includes: if it is determined that the loss value of the preset diffusion model does not reach the target loss value, updating the spatiotemporal diffusion transformation parameters and diffusion transformation parameters of the preset diffusion model; training the preset diffusion model according to the updated spatiotemporal diffusion transformation parameters, the updated diffusion transformation parameters and the next sample among multiple samples until the calculated loss value reaches the target loss value, thereby obtaining an audio and video generation diffusion model.

[0012] In the above-mentioned various implementation modes, in the process of training a preset diffusion model, the present scheme uses video features as diffusion consideration conditions in the diffusion transformation process of generating target audio features when the preset diffusion steps are reached, and uses audio features as diffusion consideration conditions in the spatiotemporal diffusion transformation process of generating target video features, thereby associating video features with audio features, and furthermore, the video features and audio features output by the trained preset diffusion model for the input text are also associated accordingly, so that the target video features and target audio features recognized and output by the audio and video generation diffusion model trained by the present scheme are associated, so that the target video and audio generated by the present scheme are highly synchronized, and furthermore, the synchronization and quality of audio and video are higher.

[0013] In an optional implementation of the first aspect, obtaining text sample features includes: obtaining text information, encoding the text information, and obtaining text sample features.

[0014] In an optional implementation of the first aspect, obtaining video sample features and audio sample features corresponding to text sample features includes: obtaining video information and audio information corresponding to the text sample features; compressing and quantizing the video information corresponding to the text sample features to obtain video sample features corresponding to the text sample features; and quantizing and compressing the audio information corresponding to the text sample features to obtain audio sample features corresponding to the text sample features.

[0015] In a second aspect, the present application provides an audio and video generation device, which includes an acquisition module, an input module and a decoding module. The acquisition module is used to acquire a target text, wherein the target text includes target text features; the input module is used to input the target text into a pre-trained audio and video generation diffusion model to obtain target video features and target audio features output by the audio and video generation diffusion model, wherein the target video features are generated by performing a spatiotemporal diffusion transformation on the audio and video generation diffusion model according to initial video features, initial audio features and target text features, the target audio features are generated by performing a diffusion transformation on the audio and video generation diffusion model according to initial audio features, initial video features and target text features, the initial video features are generated by performing a spatiotemporal diffusion transformation on the audio and video generation diffusion model according to first Gaussian noise and target text features, and the initial audio features are generated by performing a diffusion transformation on the audio and video generation diffusion model according to second Gaussian noise and target text features; the decoding module is used to decode the target video features and the target audio features respectively to obtain the target video and target audio corresponding to the target text.

[0016] The audio and video generation device designed above, this scheme first obtains the target text with text features, and then inputs the target text into the pre-trained audio and video generation diffusion model, and generates the target video features and target audio features corresponding to the output target text through the audio and video generation diffusion model, and then decodes the target video features and target audio features respectively to obtain the target video and target audio corresponding to the target text. Since the audio and video generation diffusion model takes the video features as diffusion consideration conditions in the diffusion transformation process of generating the target audio features, the generated target audio features are associated with the video features. Similarly, the audio and video generation diffusion model takes the audio features as diffusion consideration conditions in the spatiotemporal diffusion transformation process of generating the target video features. Therefore, the generated target video features are associated with the audio features, and then the generated target video features are associated with the target audio features, so that the target video features and the target audio features are associated, so that the target video features generated by this scheme are highly synchronized, and then the synchronization and quality of audio and video are higher.

[0017] In an optional implementation of the second aspect, the device also includes a training module for obtaining a training set, wherein the training set includes multiple samples, each sample includes a text sample feature and a video sample feature and an audio sample feature corresponding to the text sample feature; the video sample feature of each sample is segmented in time and space and a first Gaussian noise is added to obtain a latent space feature of the video noise of each sample; the audio sample feature of each sample is segmented in time and space and a second Gaussian noise is added to obtain a latent space feature of the audio noise of each sample; a preset diffusion model is trained according to the text sample features, the latent space features of the video noise, and the latent space features of the audio noise of the multiple samples to obtain an audio and video generation diffusion model.

[0018] In an optional implementation of the second aspect, the training module is specifically used to obtain a target number of diffusion steps; perform spatiotemporal diffusion transformation according to text sample features and latent space features of video noise to achieve noise removal, and obtain an initial video noise removal feature, and perform diffusion transformation noise removal according to text sample features and latent space features of audio noise to obtain an initial audio noise removal feature; determine whether the number of diffusion steps reaches a preset number of diffusion steps; if the number of diffusion steps reaches the preset number of diffusion steps, perform spatiotemporal diffusion transformation according to the initial video noise removal feature, the initial audio noise removal feature and the text sample feature to achieve noise removal, and obtain an initial video interaction feature, and perform diffusion transformation according to the initial video noise removal feature, the initial audio noise removal feature and the text sample feature to achieve noise removal, and obtain an initial audio interaction feature; determine whether the number of diffusion steps reaches the target number of diffusion steps; if the number of diffusion steps reaches the target number of diffusion steps, calculate the loss value of the preset diffusion model according to the initial video interaction feature, the video sample feature, the initial audio interaction feature and the audio sample feature; determine whether the loss value of the preset diffusion model reaches the target loss value; if the loss value of the preset diffusion model reaches the target loss value, obtain an audio and video generation diffusion model.

[0019] In an optional implementation manner of the second aspect, the training module is also specifically used to, if it is determined that the number of diffusion steps has not reached a preset number of diffusion steps, perform a spatiotemporal diffusion transformation according to the initial video noise removal feature and the text sample feature to achieve noise removal, and obtain a second video noise removal feature, and perform a diffusion transformation according to the initial audio noise removal feature and the text sample feature to achieve noise removal, and obtain a second audio noise removal feature; determine whether the number of diffusion steps has reached a preset number of diffusion steps; if the number of diffusion steps has reached the preset number of diffusion steps, perform a spatiotemporal diffusion transformation according to the second video noise removal feature, the second audio noise removal feature and the text sample feature to achieve noise removal, and obtain an initial video interaction feature, and perform a diffusion transformation according to the second video noise removal feature, the second audio noise removal feature and the text sample feature to achieve noise removal, and obtain an initial audio interaction feature; determine whether the number of diffusion steps has reached a target number of diffusion steps; if the number of diffusion steps has reached the target number of diffusion steps, calculate the loss value of the preset diffusion model according to the initial video interaction feature, the video sample feature, the initial audio interaction feature and the audio sample feature; determine whether the loss value of the preset diffusion model has reached the target loss value; if the loss value of the preset diffusion model has reached the target loss value, obtain an audio and video generation diffusion model.

[0020] In an optional implementation of the second aspect, the training module is also specifically used to, if it is determined that the number of diffusion steps has not reached the target number of diffusion steps, perform spatiotemporal diffusion transformation according to the initial video interaction features, the initial audio interaction features and the text sample features to achieve noise removal, and obtain the second video interaction features, and perform diffusion transformation according to the initial video interaction features, the initial audio interaction features and the text sample features to achieve noise removal, and obtain the second audio interaction features; determine whether the number of diffusion steps has reached the target number of diffusion steps; if the number of diffusion steps has reached the target number of diffusion steps, calculate the loss value of the preset diffusion model according to the second video interaction features, the video sample features, the second audio interaction features and the audio sample features; determine whether the loss value of the preset diffusion model has reached the target loss value; if the loss value of the preset diffusion model has reached the target loss value, obtain the audio and video generation diffusion model.

[0021] In an optional implementation of the second aspect, the training module is further specifically used to update the spatiotemporal diffusion transformation parameters and diffusion transformation parameters of the preset diffusion model if it is determined that the loss value of the preset diffusion model does not reach the target loss value; train the preset diffusion model according to the updated spatiotemporal diffusion transformation parameters, the updated diffusion transformation parameters and the next sample among multiple samples until the calculated loss value reaches the target loss value, thereby obtaining an audio and video generation diffusion model.

[0022] In an optional implementation of the second aspect, the acquisition module is specifically used to acquire text information, encode the text information, and obtain text sample features.

[0023] In an optional implementation of the second aspect, the acquisition module is also specifically used to obtain video information and audio information corresponding to the text sample features; compress and quantize the video information corresponding to the text sample features to obtain video sample features corresponding to the text sample features; and quantize and compress the audio information corresponding to the text sample features to obtain audio sample features corresponding to the text sample features.

[0024] In a third aspect, the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes any optional method described in the first aspect when executing the computer program.

[0025] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs any optional method described in the first aspect.

[0026] In a fifth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, performs the steps of any optional method described in the first aspect.

[0027] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 A schematic diagram of a first process flow of the audio and video generation method provided in an embodiment of the present application;

[0030] Figure 2 A second flow chart of the audio and video generation method provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of the structure of the space-time diffusion transformation provided in the embodiment of the present application;

[0032] Figure 4 A schematic diagram of the structure of the diffusion transformation provided in the embodiment of the present application;

[0033] Figure 5 A schematic diagram of the structure of an audio and video generating device provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0035] Icon: 500 - acquisition module; 510 - input module; 520 - decoding module; 530 - training module; 6 - electronic device; 601 - processor; 602 - memory; 603 - communication bus. DETAILED DESCRIPTION

[0036] The following embodiments of the technical solution of the present application will be described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0038] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0039] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0040] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0041] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0042] In the description of the embodiments of the present application, the technical terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the embodiments of the present application.

[0043] In the description of the embodiments of the present application, unless otherwise clearly specified and limited, technical terms such as "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0044] In the prior art, audio generation and video generation mostly generate corresponding audio or video from a given text. There are currently two commonly used frameworks: one is to generate audio and video separately through a diffusion model framework; the other is to generate audio and video separately through an autoregressive structure.

[0045] In the diffusion model architecture, there is no structure that can generate audio and video at the same time. Text is input to generate audio and video separately, making the audio and video independent and unrelated. As a result, the generated audio and video have no direct correlation, which in turn causes the generated audio and video to be out of sync and of low quality.

[0046] Based on the above problems, the present application designs an audio and video generation method, an electronic device and a computer-readable storage medium. By introducing audio features into the spatiotemporal diffusion transformation structure of video generation and video features into the diffusion transformation structure of audio generation during the diffusion process of the diffusion model, the diffusion model generates interactive information of audio and video when generating audio and video, and the audio and video output by the trained diffusion model are correlated, thereby making the synchronization and quality of the audio and video generated by this scheme higher.

[0047] Based on the above ideas, the present application first provides an audio and video generation method, which can be applied to a computing device, including but not limited to a computer, a server, a computing chip, a GPU, a CPU, etc. Figure 1 As shown, the audio and video generation method can be implemented by the following methods, including:

[0048] Step S100: Acquire target text.

[0049] Step S110: input the target text into a pre-trained audio and video generation diffusion model to obtain target video features and target audio features output by the audio and video generation diffusion model.

[0050] Step S120: Decode the target video features and the target audio features respectively to obtain the target video and the target audio corresponding to the target text.

[0051] In the above implementation, the target text represents text information that needs to be converted into video and audio. For example, the target text may be a text description of a certain scenic spot, and the video content and audio content corresponding to the text description of the scenic spot need to be generated. Among them, the target text may include target text features, and the target text features may specifically be text feature vectors, and the text feature vectors may be obtained by encoding the target text features. Specifically, the text feature vectors may be obtained by encoding the target text features using the T5 model.

[0052] When the target text is obtained by the above method, this solution inputs the target text into the pre-trained audio and video generation diffusion model, and outputs the target video features and target audio features corresponding to the target text through the pre-trained audio and video generation diffusion model. The target video features are generated by performing a spatiotemporal diffusion transformation STDIT based on the initial video features, initial audio features and target text features through the audio and video generation diffusion model, and the target audio features are generated by performing a diffusion transformation based on the initial audio features, initial video features and target text features through the audio and video generation diffusion model. Among them, the initial video features are generated by performing a spatiotemporal diffusion transformation based on the first Gaussian noise and the target text features through the audio and video generation diffusion model, and the initial audio features are generated by performing a diffusion transformation based on the second Gaussian noise and the target text features through the audio and video generation diffusion model. Since the audio and video generation diffusion model takes video features as diffusion consideration conditions in the diffusion transformation process of generating target audio features, the generated target audio features are associated with video features. Similarly, the audio and video generation diffusion model takes audio features as diffusion consideration conditions in the spatiotemporal diffusion transformation process of generating target video features. Therefore, the generated target video features are associated with audio features, thereby achieving correlation between the generated target video features and the target audio features. Among them, the diffusion transformation described in this scheme describes the transformation process of generating audio features using the diffusion model (Diffusion Transformer, DIT). Diffusion Transformer is a diffusion model combined with the Transformer architecture, which is used for image and video generation tasks. It can efficiently capture the dependencies in the data and generate high-quality results; the STDiT (Spatial Temporal Diffusion Transformer) model of the spatial-temporal attention mechanism adopts the aforementioned diffusion model Diffusion Transformer architecture, and is based on the high-quality open source cultural model PixArt-α that also uses the DiT architecture. On this basis, the temporal attention layer is introduced to extend it to the model on video data. Among them, DIT and STDIT can use any existing model. This scheme combines the two models to train and generate the aforementioned audio and video generation diffusion model. Gaussian noise refers to a type of noise whose probability density function obeys the Gaussian distribution (i.e., normal distribution). Common Gaussian noise includes fluctuation noise, cosmic noise, thermal noise, shot noise, etc.

[0053] After obtaining the target video features and target audio features corresponding to the target text through the output of the above audio and video generation diffusion model, this solution decodes the target video features and the target audio features respectively, thereby obtaining the target video and target audio corresponding to the target text. Specifically, as a possible example, this solution can use the decoder structure of the code module of the video to decode the target video features to obtain the target video. Similarly, this solution can use the decoder structure of the code module of the audio to decode the target audio features to obtain the target audio.

[0054] The audio and video generation method designed above, this scheme first obtains the target text with text features, and then inputs the target text into the pre-trained audio and video generation diffusion model, and generates the target video features and target audio features corresponding to the output target text through the audio and video generation diffusion model, and then decodes the target video features and target audio features respectively to obtain the target video and target audio corresponding to the target text. Since the audio and video generation diffusion model takes the video features as diffusion consideration conditions in the diffusion transformation process of generating the target audio features, the generated target audio features are associated with the video features. Similarly, the audio and video generation diffusion model takes the audio features as diffusion consideration conditions in the spatiotemporal diffusion transformation process of generating the target video features. Therefore, the generated target video features are associated with the audio features, and then the generated target video features are associated with the target audio features, so that the target video features and the target audio features are associated, so that the target video features generated by this scheme are highly synchronized, and then the synchronization and quality of audio and video are higher.

[0055] In an optional implementation of this embodiment, the above description states that this solution generates output target video features and target audio features through an audio and video generation diffusion model. In this regard, this solution can train the diffusion model in advance to obtain a trained audio and video generation diffusion model. Figure 2 As shown, the training process of the audio and video diffusion generation model may include:

[0056] Step S200: Obtain a training set, wherein the training set includes a plurality of samples, each sample includes a text sample feature and a video sample feature and an audio sample feature corresponding to the text sample feature.

[0057] Step S210: performing spatiotemporal segmentation on the video sample features of each sample and adding the first Gaussian noise to obtain the latent spatial features of the video noise of each sample.

[0058] Step S220: performing spatiotemporal segmentation on the audio sample features of each sample and adding a second Gaussian noise to obtain latent spatial features of the audio noise of each sample.

[0059] Step S230: training a preset diffusion model according to the text sample features of multiple samples, the latent space features of video noise, and the latent space features of audio noise to obtain an audio and video generation diffusion model.

[0060] In the above implementation, the scheme may first obtain a training set, which may include multiple samples, each of which may include a text sample feature and a video sample feature and an audio sample feature corresponding to the text sample feature. Among them, the text sample feature, video sample feature and audio sample feature in the same sample have a corresponding relationship. For example, the sample may be a text content describing a certain scenic spot, a video content corresponding to the text content describing the scenic spot, and an audio content corresponding to the text content describing the scenic spot. The text sample feature may be the text feature vector described above. The video sample feature may be a video feature obtained by compressing and quantizing the video, for example, a video feature obtained by extracting the features of the video through VideoAutoencoder; the audio sample feature may be a compressed audio feature obtained by quantizing and compressing the audio through the encoder model of DAC.

[0061] When multiple samples of the training set are obtained in the above manner, this scheme can perform spatiotemporal segmentation on the video sample features in each sample and add the first Gaussian noise to obtain the latent space features of the video noise of each sample, and perform spatiotemporal segmentation on the audio sample features in each sample and add the second Gaussian noise to obtain the latent space features of the audio noise of each sample.

[0062] Specifically, this scheme can perform spatiotemporal patch segmentation on the video, and then add the first Gaussian noise to obtain the latent spatial features of the video noise; that is: , , here is the number of diffusion steps; ,Here z represents the intermediate features of the video obtained after spatiotemporal segmentation. It represents the latent space features obtained after adding noise; It represents the video features obtained by extracting the video features; Represents video-dependent parameters.

[0063] This scheme can add the second Gaussian noise to the compressed audio features to obtain the latent space features of the audio noise; that is: , here is the number of diffusion steps; ,in, It represents the latent space characteristics of audio noise. Represents the audio features obtained by extracting audio features. Represents audio-dependent parameters.

[0064] Among them, patch segmentation is a technique for dividing an image into small patches (patches), which is often used in image processing and computer vision tasks. This method can divide a large image into multiple small patches, each of which is processed independently, thereby reducing the amount of computation and memory requirements.

[0065] Then, this scheme trains the preset diffusion model according to the text sample features of multiple samples, the latent space features of video noise, and the latent space features of audio noise to obtain the audio and video generation diffusion model.

[0066] Specifically, this scheme can first obtain the target diffusion steps, perform spatiotemporal diffusion transformation according to the text sample characteristics and the latent space characteristics of the video noise to achieve noise removal, and obtain the initial video noise removal features, and perform diffusion transformation noise removal according to the text sample characteristics and the latent space characteristics of the audio noise to obtain the initial audio noise removal features.

[0067] Then determine whether the number of diffusion steps reaches the preset number of diffusion steps. If it is determined that the number of diffusion steps reaches the preset number of diffusion steps, it means that the diffusion has reached the number of steps required for video and audio interaction. This scheme performs spatiotemporal diffusion transformation based on the initial video noise removal features, the initial audio noise removal features, and the text sample features to achieve noise removal and obtain the initial video interaction features. It also performs diffusion transformation based on the initial video noise removal features, the initial audio noise removal features, and the text sample features to achieve noise removal and obtain the initial audio interaction features.

[0068] It is further determined whether the number of diffusion steps reaches the target number of diffusion steps. If the number of diffusion steps reaches the target number of diffusion steps, it means that the number of diffusion steps has been completed. In this case, this scheme calculates the loss value of the preset diffusion model according to the initial video interaction features, video sample features, initial audio interaction features and audio sample features; it is determined whether the loss value of the preset diffusion model reaches the target loss value; if the loss value of the preset diffusion model reaches the target loss value, it means that the parameters of the preset diffusion model meet the requirements, and then the audio and video generation diffusion model is obtained.

[0069] As a possible implementation method, if this scheme determines that the number of diffusion steps has not reached the preset number of diffusion steps, it means that the diffusion has not reached the number of steps required for video and audio interaction. In this case, this scheme continues to perform spatiotemporal diffusion transformation based on the initial video noise removal features and text sample features to achieve noise removal, and obtain the second video noise removal features, and perform diffusion transformation based on the initial audio noise removal features and text sample features to achieve noise removal, and obtain the second audio noise removal features.

[0070] Then determine whether the number of diffusion steps reaches the preset number of diffusion steps; if the number of diffusion steps reaches the preset number of diffusion steps, perform spatiotemporal diffusion transformation according to the second video noise removal feature, the second audio noise removal feature and the text sample feature to achieve noise removal, and obtain the initial video interaction feature, and perform diffusion transformation according to the second video noise removal feature, the second audio noise removal feature and the text sample feature to achieve noise removal, and obtain the initial audio interaction feature.

[0071] Further determine whether the number of diffusion steps reaches the target number of diffusion steps. If the number of diffusion steps reaches the target number of diffusion steps, calculate the loss value of the preset diffusion model based on the initial video interaction features, video sample features, initial audio interaction features and audio sample features; determine whether the loss value of the preset diffusion model reaches the target loss value; if the loss value of the preset diffusion model reaches the target loss value, obtain the audio and video generation diffusion model.

[0072] As a possible implementation method, after determining whether the number of diffusion steps has reached the target number of diffusion steps, this scheme also includes: if it is determined that the number of diffusion steps has not reached the target number of diffusion steps, it means that diffusion needs to continue. In this case, this scheme performs spatiotemporal diffusion transformation according to the initial video interaction features, the initial audio interaction features, and the text sample features to achieve noise removal and obtain the second video interaction features, and performs diffusion transformation according to the initial video interaction features, the initial audio interaction features, and the text sample features to achieve noise removal and obtain the second audio interaction features.

[0073] In the case of obtaining the second video interaction feature and the second audio interaction feature as mentioned above, this scheme continues to determine whether the number of diffusion steps reaches the target number of diffusion steps. If the number of diffusion steps reaches the target number of diffusion steps, the loss value of the preset diffusion model is calculated based on the second video interaction feature, the video sample feature, the second audio interaction feature and the audio sample feature; it is determined whether the loss value of the preset diffusion model reaches the target loss value; if the loss value of the preset diffusion model reaches the target loss value, the audio and video generation diffusion model is obtained.

[0074] As a possible implementation, after determining whether the loss value of the preset diffusion model reaches the target loss value, the method further includes: if it is determined that the loss value of the preset diffusion model does not reach the target loss value, the spatiotemporal diffusion transformation parameters and diffusion transformation parameters of the preset diffusion model are updated; the preset diffusion model is trained according to the updated spatiotemporal diffusion transformation parameters, the updated diffusion transformation parameters, and the next sample in the multiple samples until the calculated loss value reaches the target loss value, thereby obtaining an audio and video generation diffusion model. Among them, the preset diffusion steps and the target diffusion steps described above can be adaptively adjusted according to user needs, and users can also adjust the settings by themselves, and the specific values ​​can be adjusted according to the actual application scenario.

[0075] The above implementation mode can be specifically described by the following examples:

[0076] Assume that the preset diffusion step number is 3 and the target diffusion step number is 5. In this case, this scheme first performs spatiotemporal diffusion transformation according to the text sample features and the latent space features of the video noise to achieve noise removal, and obtains the initial video noise removal features. It also performs diffusion transformation noise removal according to the text sample features and the latent space features of the audio noise, and obtains the initial audio noise removal features. At this time, the diffusion step number + 1 is 2, and the preset diffusion step number is 3, that is, the diffusion step number is less than the preset diffusion step number.

[0077] In the above situation, this scheme continues to perform spatiotemporal diffusion transformation according to the initial video noise removal features and the text sample features to achieve noise removal, and obtains the second video noise removal features, and performs diffusion transformation according to the initial audio noise removal features and the text sample features to achieve noise removal, and obtains the second audio noise removal features. At this time, the number of diffusion steps + 1 is 3, and the preset number of diffusion steps is 3, that is, the number of diffusion steps reaches the preset number of diffusion steps.

[0078] When the above-mentioned number of diffusion steps reaches the preset number of diffusion steps, this scheme performs spatiotemporal diffusion transformation according to the second video noise removal feature, the second audio noise removal feature and the text sample feature to achieve noise removal, and obtains the initial video interaction feature, and performs diffusion transformation according to the second video noise removal feature, the second audio noise removal feature and the text sample feature to achieve noise removal, and obtains the initial audio interaction feature. At this time, it is judged that the diffusion step number 3 is less than the target diffusion step number 5.

[0079] When the above-mentioned diffusion step number 3 is less than the target diffusion step number 5, this scheme performs spatiotemporal diffusion transformation according to the initial video interaction features, the initial audio interaction features and the text sample features to achieve noise removal, and obtains the second video interaction features, and performs diffusion transformation according to the initial video interaction features, the initial audio interaction features and the text sample features to achieve noise removal, and obtains the second audio interaction features. At this time, it is judged that the diffusion step number 4 is less than the target diffusion step number 5.

[0080] When the above-mentioned diffusion step number 4 is less than the target diffusion step number 5, this scheme performs spatiotemporal diffusion transformation according to the second video interaction feature, the second audio interaction feature and the text sample feature to achieve noise removal, and obtains the third video interaction feature, and performs diffusion transformation according to the second video interaction feature, the second audio interaction feature and the text sample feature to achieve noise removal, and obtains the third audio interaction feature. At this time, it is judged that the diffusion step number 5 reaches the target diffusion step number 5, and the loss value of the preset diffusion model is calculated according to the third video interaction feature, the video sample feature, the third audio interaction feature and the audio sample feature; it is judged whether the loss value of the preset diffusion model reaches the target loss value; if the loss value of the preset diffusion model reaches the target loss value, the audio and video generation diffusion model is obtained; if the loss value of the preset diffusion model does not reach the target loss value, the samples in the training set are continued to be used to execute the aforementioned training process to train the preset diffusion model until the loss value reaches the target loss value.

[0081] The space-time diffusion transformation described above can be any existing space-time diffusion transformation structure, for example, Figure 3 As shown in Figure 1, the video noise is input to the spatiotemporal attention mechanism to obtain the output of the spatiotemporal attention mechanism, where the spatial attention is a self-attention model that puts all blocks of the same frame together.

[0082] On the basis of the above, the output of the spatiotemporal attention mechanism, video noise, text features, and the audio features introduced when the number of diffusion steps reaches the preset number of diffusion steps are input together to the cross attention mechanism Cross Attention to obtain the output of the cross attention mechanism Cross Attention. Among them, Cross Attention calculates the attention score between each element in the first sequence and all elements in the second sequence, and learns the relationship between the two sequences in this way.

[0083] On the basis of the above, the output of the cross attention mechanism Cross Attention and the output of the spatiotemporal attention mechanism Space Attention are input to the time attention mechanism TimeAttention after passing through the first multi-layer perceptron, where Time Attention represents the prediction of time series.

[0084] On the basis of the above, the output of the time attention mechanism Time Attention and the output of the first multi-layer perceptron are passed through the second multi-layer perceptron to obtain the output result of the spatiotemporal diffusion transformation.

[0085] It should be noted here that the innovation of this solution lies in the introduction of audio features in the step of the cross attention mechanism, so that the spatiotemporal diffusion transformation process of the video features adds audio features as a reference, thereby associating the video features with the audio features, and the rest of the parts can be conventional spatiotemporal diffusion transformation structures, and the specific contents are not described in detail in this application.

[0086] The diffusion transformation described above can be any existing diffusion transformation structure, for example, Figure 4 As shown in the figure, the audio noise is input into the self-attention model to obtain the output of the self-attention model, and then the output of the self-attention model, the text features, and the video features introduced when the preset diffusion steps are reached are input into the cross-attention mechanism CrossAttention. After obtaining the output of the cross-attention mechanism Cross Attention, the result of the audio diffusion transformation is obtained through a multi-layer perceptron.

[0087] It should be noted here that the innovation of this solution lies in the introduction of video features in the step of the cross-attention mechanism CrossAttention of the diffusion transformation, so that the diffusion transformation process of the audio features adds the video features as a reference, thereby associating the video features with the audio features, and the rest can be conventional diffusion transformation structures, and the specific content will not be described in detail in this application.

[0088] In the above implementation mode, in the process of training the preset diffusion model, when the preset diffusion steps are reached, the video features are taken as diffusion consideration conditions in the diffusion transformation process of generating the target audio features, and the audio features are taken as diffusion consideration conditions in the spatiotemporal diffusion transformation process of generating the target video features, so that the video features are associated with the audio features, and then the video features and audio features output by the trained preset diffusion model for the input text are also associated accordingly, so that the target video features and target audio features recognized and output by the audio and video generation diffusion model trained by the scheme are associated, so that the target video and audio generated by the scheme are highly synchronized, and thus the synchronization and quality of the audio and video are higher.

[0089] Figure 5 The present application provides a schematic structural block diagram of an audio and video generating device. It should be understood that the device is applied to the electronic device described above. Figure 1 and 4The method embodiment executed in the embodiment corresponds to the method embodiment, and can execute the steps involved in the aforementioned method. The specific functions of the device can refer to the description above. To avoid repetition, the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or fixed in the operating system (OS) of the device. Specifically, the device includes: an acquisition module 500, an input module 510 and a decoding module 520, wherein the acquisition module 500 is used to acquire a target text, wherein the target text includes a target text feature; the input module 510 is used to input the target text into a pre-trained audio and video generation diffusion model to obtain a target video feature and a target audio feature output by the audio and video generation diffusion model, wherein the target video feature is generated by performing a spatiotemporal diffusion transformation according to initial video features, initial audio features and target text features through the audio and video generation diffusion model, the target audio feature is generated by performing a diffusion transformation according to initial audio features, initial video features and target text features through the audio and video generation diffusion model, the initial video feature is generated by performing a spatiotemporal diffusion transformation according to a first Gaussian noise and target text features through the audio and video generation diffusion model, and the initial audio feature is generated by performing a diffusion transformation according to a second Gaussian noise and target text features through the audio and video generation diffusion model; the decoding module 520 is used to decode the target video feature and the target audio feature respectively to obtain a target video and a target audio corresponding to the target text.

[0090] The audio and video generation device designed above, this scheme first obtains the target text with text features, and then inputs the target text into the pre-trained audio and video generation diffusion model, and generates the target video features and target audio features corresponding to the output target text through the audio and video generation diffusion model, and then decodes the target video features and target audio features respectively to obtain the target video and target audio corresponding to the target text. Since the audio and video generation diffusion model takes the video features as diffusion consideration conditions in the diffusion transformation process of generating the target audio features, the generated target audio features are associated with the video features. Similarly, the audio and video generation diffusion model takes the audio features as diffusion consideration conditions in the spatiotemporal diffusion transformation process of generating the target video features. Therefore, the generated target video features are associated with the audio features, and then the generated target video features are associated with the target audio features, so that the target video features and the target audio features are associated, so that the target video features generated by this scheme are highly synchronized, and then the synchronization and quality of audio and video are higher.

[0091] According to some embodiments of the present application, the device also includes a training module 530, which is used to obtain a training set, wherein the training set includes multiple samples, each sample includes a text sample feature and a video sample feature and an audio sample feature corresponding to the text sample feature; the video sample feature of each sample is segmented in time and space and a first Gaussian noise is added to obtain a latent space feature of the video noise of each sample; the audio sample feature of each sample is segmented in time and space and a second Gaussian noise is added to obtain a latent space feature of the audio noise of each sample; a preset diffusion model is trained according to the text sample features, the latent space features of the video noise, and the latent space features of the audio noise of multiple samples to obtain an audio and video generation diffusion model.

[0092] According to some embodiments of the present application, the training module 530 is specifically used to obtain a target number of diffusion steps; perform spatiotemporal diffusion transformation according to text sample features and latent space features of video noise to achieve noise removal, obtain an initial video noise removal feature, and perform diffusion transformation noise removal according to text sample features and latent space features of audio noise to obtain an initial audio noise removal feature; determine whether the number of diffusion steps reaches a preset number of diffusion steps; if the number of diffusion steps reaches the preset number of diffusion steps, perform spatiotemporal diffusion transformation according to the initial video noise removal feature, the initial audio noise removal feature and the text sample feature to achieve noise removal, obtain an initial video interaction feature, and perform diffusion transformation according to the initial video noise removal feature, the initial audio noise removal feature and the text sample feature to achieve noise removal, obtain an initial audio interaction feature; determine whether the number of diffusion steps reaches the target number of diffusion steps; if the number of diffusion steps reaches the target number of diffusion steps, calculate the loss value of the preset diffusion model according to the initial video interaction feature, the video sample feature, the initial audio interaction feature and the audio sample feature; determine whether the loss value of the preset diffusion model reaches the target loss value; if the loss value of the preset diffusion model reaches the target loss value, obtain an audio and video generation diffusion model.

[0093] According to some embodiments of the present application, the training module 530 is also specifically used to, if it is determined that the number of diffusion steps has not reached the preset number of diffusion steps, perform a spatiotemporal diffusion transformation according to the initial video noise removal feature and the text sample feature to achieve noise removal, and obtain a second video noise removal feature, and perform a diffusion transformation according to the initial audio noise removal feature and the text sample feature to achieve noise removal, and obtain a second audio noise removal feature; determine whether the number of diffusion steps has reached the preset number of diffusion steps; if the number of diffusion steps has reached the preset number of diffusion steps, perform a spatiotemporal diffusion transformation according to the second video noise removal feature, the second audio noise removal feature and the text sample feature to achieve noise removal, and obtain an initial video interaction feature, and perform a diffusion transformation according to the second video noise removal feature, the second audio noise removal feature and the text sample feature to achieve noise removal, and obtain an initial audio interaction feature; determine whether the number of diffusion steps has reached the target number of diffusion steps; if the number of diffusion steps has reached the target number of diffusion steps, calculate the loss value of the preset diffusion model according to the initial video interaction feature, the video sample feature, the initial audio interaction feature and the audio sample feature; determine whether the loss value of the preset diffusion model has reached the target loss value; if the loss value of the preset diffusion model has reached the target loss value, obtain an audio and video generation diffusion model.

[0094] According to some embodiments of the present application, the training module 530 is also specifically used to, if it is determined that the number of diffusion steps has not reached the target number of diffusion steps, perform spatiotemporal diffusion transformation according to the initial video interaction features, the initial audio interaction features and the text sample features to achieve noise removal, and obtain the second video interaction features, and perform diffusion transformation according to the initial video interaction features, the initial audio interaction features and the text sample features to achieve noise removal, and obtain the second audio interaction features; determine whether the number of diffusion steps has reached the target number of diffusion steps; if the number of diffusion steps has reached the target number of diffusion steps, calculate the loss value of the preset diffusion model according to the second video interaction features, the video sample features, the second audio interaction features and the audio sample features; determine whether the loss value of the preset diffusion model has reached the target loss value; if the loss value of the preset diffusion model has reached the target loss value, obtain the audio and video generation diffusion model.

[0095] According to some embodiments of the present application, the training module 530 is also specifically used to update the spatiotemporal diffusion transformation parameters and the diffusion transformation parameters of the preset diffusion model if it is determined that the loss value of the preset diffusion model has not reached the target loss value; train the preset diffusion model according to the updated spatiotemporal diffusion transformation parameters, the updated diffusion transformation parameters and the next sample among multiple samples until the calculated loss value reaches the target loss value, thereby obtaining an audio and video generation diffusion model.

[0096] According to some embodiments of the present application, the acquisition module 500 is specifically used to acquire text information, encode the text information, and obtain text sample features.

[0097] According to some embodiments of the present application, the acquisition module 500 is also specifically used to obtain video information and audio information corresponding to text sample features; compress and quantize the video information corresponding to the text sample features to obtain video sample features corresponding to the text sample features; and quantize and compress the audio information corresponding to the text sample features to obtain audio sample features corresponding to the text sample features.

[0098] According to some embodiments of the present application, Figure 6 As shown, the present application provides an electronic device 6, including: a processor 601 and a memory 602, the processor 601 and the memory 602 are interconnected and communicate with each other through a communication bus 603 and / or other forms of connection mechanisms (not marked), the memory 602 stores a computer program executable by the processor 601, and when the computing device is running, the processor 601 executes the computer program to execute any optional implementation method, such as steps S100 to S120: obtaining a target text; inputting the target text into a pre-trained audio and video generation diffusion model to obtain a target video feature and a target audio feature output by the audio and video generation diffusion model; respectively decoding the target video feature and the target audio feature to obtain a target video and a target audio corresponding to the target text.

[0099] The present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method in any of the aforementioned optional implementations is executed.

[0100] Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0101] The present application provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the method in any optional implementation manner.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should all be included in the scope of the claims and specification of the present application. In particular, as long as there is no structural conflict, the various technical features mentioned in the various embodiments can be combined in any way. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions that fall within the scope of the claims.

Claims

1. A method for generating audio and video, characterized in that: The method comprises: Acquire a target text, wherein the target text includes a target text feature; Input the target text into a pre-trained audio and video generation diffusion model to obtain target video features and target audio features output by the audio and video generation diffusion model, wherein the target video features are generated by performing spatiotemporal diffusion transformation on the audio and video generation diffusion model according to initial video features, initial audio features and target text features, the target audio features are generated by performing diffusion transformation on the audio and video generation diffusion model according to initial audio features, initial video features and target text features, the initial video features are generated by performing spatiotemporal diffusion transformation on the audio and video generation diffusion model according to first Gaussian noise and target text features, and the initial audio features are generated by performing diffusion transformation on the audio and video generation diffusion model according to second Gaussian noise and target text features; Decoding the target video features and the target audio features respectively to obtain the target video and the target audio corresponding to the target text; The training process of the audio and video diffusion generation model includes: Obtaining a training set, wherein the training set includes a plurality of samples, each sample including a text sample feature and a video sample feature and an audio sample feature corresponding to the text sample feature; Performing spatiotemporal segmentation on the video sample features of each sample and adding the first Gaussian noise to obtain the latent spatial features of the video noise of each sample; Performing spatiotemporal segmentation on the audio sample features of each sample and adding a second Gaussian noise to obtain the latent spatial features of the audio noise of each sample; Get the target diffusion steps; According to the text sample features and the latent space features of the video noise, a spatiotemporal diffusion transformation is performed to achieve noise removal, and an initial video noise removal feature is obtained. According to the text sample features and the latent space features of the audio noise, a diffusion transformation noise removal is performed to obtain an initial audio noise removal feature. Determine whether the number of diffusion steps reaches the preset number of diffusion steps; If the number of diffusion steps reaches the preset number of diffusion steps, a spatiotemporal diffusion transformation is performed according to the initial video noise removal features, the initial audio noise removal features and the text sample features to achieve noise removal, and an initial video interaction feature is obtained. A diffusion transformation is performed according to the initial video noise removal features, the initial audio noise removal features and the text sample features to achieve noise removal, and an initial audio interaction feature is obtained. Determine whether the number of diffusion steps reaches the target number of diffusion steps; If the number of diffusion steps reaches the target number of diffusion steps, the loss value of the preset diffusion model is calculated according to the initial video interaction features, the video sample features, the initial audio interaction features and the audio sample features; Determine whether the loss value of the preset diffusion model reaches the target loss value; If the loss value of the preset diffusion model reaches the target loss value, the audio and video generation diffusion model is obtained.

2. The method according to claim 1, characterized in that After determining whether the number of diffusion steps reaches a preset number of diffusion steps, the method further includes: If it is determined that the number of diffusion steps does not reach the preset number of diffusion steps, a spatiotemporal diffusion transformation is performed according to the initial video noise removal feature and the text sample feature to achieve noise removal, and a second video noise removal feature is obtained. A diffusion transformation is performed according to the initial audio noise removal feature and the text sample feature to achieve noise removal, and a second audio noise removal feature is obtained. Determine whether the number of diffusion steps reaches the preset number of diffusion steps; If the number of diffusion steps reaches the preset number of diffusion steps, a spatiotemporal diffusion transformation is performed according to the second video noise removal feature, the second audio noise removal feature and the text sample feature to achieve noise removal, and an initial video interaction feature is obtained; and a diffusion transformation is performed according to the second video noise removal feature, the second audio noise removal feature and the text sample feature to achieve noise removal, and an initial audio interaction feature is obtained; Determine whether the number of diffusion steps reaches the target number of diffusion steps; If the number of diffusion steps reaches the target number of diffusion steps, the loss value of the preset diffusion model is calculated according to the initial video interaction features, the video sample features, the initial audio interaction features and the audio sample features; Determine whether the loss value of the preset diffusion model reaches the target loss value; If the loss value of the preset diffusion model reaches the target loss value, the audio and video generation diffusion model is obtained.

3. The method according to claim 1, characterized in that After determining whether the number of diffusion steps reaches the target number of diffusion steps, the method further includes: If it is determined that the number of diffusion steps does not reach the target number of diffusion steps, a spatiotemporal diffusion transformation is performed according to the initial video interaction features, the initial audio interaction features, and the text sample features to achieve noise removal, and a second video interaction feature is obtained. A diffusion transformation is performed according to the initial video interaction features, the initial audio interaction features, and the text sample features to achieve noise removal, and a second audio interaction feature is obtained. Determine whether the number of diffusion steps reaches the target number of diffusion steps; If the number of diffusion steps reaches the target number of diffusion steps, a loss value of a preset diffusion model is calculated according to the second video interaction feature, the video sample feature, the second audio interaction feature, and the audio sample feature; Determine whether the loss value of the preset diffusion model reaches the target loss value; If the loss value of the preset diffusion model reaches the target loss value, the audio and video generation diffusion model is obtained.

4. The method according to claim 1, characterized in that: After determining whether the loss value of the preset diffusion model reaches the target loss value, the method further includes: If it is determined that the loss value of the preset diffusion model does not reach the target loss value, the spatiotemporal diffusion transformation parameters and the diffusion transformation parameters of the preset diffusion model are updated; The preset diffusion model is trained according to the updated spatiotemporal diffusion transformation parameters, the updated diffusion transformation parameters and the next sample among the multiple samples until the calculated loss value reaches the target loss value, thereby obtaining the audio and video generation diffusion model.

5. The method according to claim 1, characterized in that in, Get text sample features, including: Acquire text information, encode the text information, and obtain text sample features.

6. The method according to claim 1, characterized in that in, Get the video sample features and audio sample features corresponding to the text sample features, including: Obtain video information and audio information corresponding to text sample features; Compressing and quantizing the video information corresponding to the text sample feature to obtain the video sample feature corresponding to the text sample feature; The audio information corresponding to the text sample feature is quantized and compressed to obtain the audio sample feature corresponding to the text sample feature.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images