Audio-video generation method and device, electronic equipment and storage medium

Abstract

This invention provides a method, apparatus, electronic device, and storage medium for generating audio-visual content, belonging to the field of computer technology. The method includes: preprocessing the text to be reasoned to obtain a text identifier sequence corresponding to the text; inputting the text identifier sequence into an audio-visual generation model to generate an audio-visual video corresponding to the text; wherein the audio-visual generation model includes an autoregressive audio-visual sequence generation model, an audio-visual vector quantization autoencoder, a video frame interpolation model, an audio conversion model, and audio-visual processing tools. This invention generates audio-visual video through the audio-visual generation model, enabling the simultaneous generation of video and audio signals. It then synthesizes an audio-visual video with good generalization that matches the semantics of the text to be reasoned, effectively focusing on audio modal information, providing important data for artificial intelligence research, and effectively meeting user needs.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and storage medium for generating audio-visual content. Background Technology

[0002] The generation of audio-visual content from text is a research area that spans multiple fields and involves multimodal information, and this research is of great significance to artificial intelligence research.

[0003] However, existing video generation methods often only focus on the generation of video images, that is, they often only focus on the generation method from text modality to video modality.

[0004] Therefore, how to generate text-to-video audio has become an urgent problem to be solved in the industry. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, and storage medium for generating audio-visual content, in order to address the technical need for generating text-to-audio-visual content in the prior art.

[0006] In a first aspect, the present invention provides an audio video generation method, comprising: preprocessing a text to be inferred to obtain a text identifier sequence corresponding to the text to be inferred; inputting the text identifier sequence into an audio video generation model to generate an audio video corresponding to the text to be inferred; wherein the audio video generation model includes an autoregressive audio-visual sequence generation model, an audio-visual vector quantization autoencoder, a video frame interpolation model, an audio conversion model, and an audio-visual processing tool.

[0007] According to a method for generating audio-visual content based on the present invention, the step of inputting the text identifier sequence into an audio-visual generation model to generate an audio-visual video corresponding to the text to be inferred includes: Step 21, inputting the text identifier sequence into the autoregressive audio-visual sequence generation model to generate a discrete identifier sequence of video image frames and a discrete identifier sequence of audio spectrum corresponding to the text identifier sequence; Step 22, inputting the discrete identifier sequence of video image frames and the discrete identifier sequence of audio spectrum into the decoder of the audio-visual vector quantization autoencoder to generate video image frames and audio Mel spectrum; Step 23, inputting the generated video image frames into the video frame interpolation model to synthesize a silent video; Step 24, inputting the audio Mel spectrum into the audio conversion model to synthesize an audio signal; the duration of the silent video matches the duration of the audio signal; Step 25, inputting the silent video and the audio signal into the audio-visual processing tool to generate the audio-visual video.

[0008] According to the present invention, a method for generating audio-visual sequences is provided, wherein the autoregressive audio-visual sequence generation model is an autoregressive trimodal joint Transformer neural network decoder model; and / or, the audio-visual vector quantization autoencoder is an SVG-VQGAN model; and / or, the audio conversion model is a HiFiGAN decoder; and / or, the audio-visual processing tool is the ffmpeg multimedia processing tool; and / or, the video frame interpolation model is constructed based on a frame interpolation neural network model.

[0009] According to the present invention, a method for generating audio video is provided, wherein the audio conversion model is trained by the following steps: Step 101, preprocessing the audio and video samples corresponding to each text sample to obtain the audio signal samples and video image frame samples corresponding to each audio and video sample, and obtaining the audio Mel spectrum samples corresponding to each audio signal sample; Step 102, taking any one of the audio Mel spectrum samples as the input to the audio conversion model to be trained, and taking the audio signal sample corresponding to any one of the audio Mel spectrum samples as the output label of the audio conversion model to be trained, and pre-training the audio conversion model to be trained; iteratively executing step 102 until the pre-training of the audio conversion model to be trained is completed, and obtaining the trained audio conversion model.

[0010] According to a method for generating audio-visual content provided by the present invention, step 101 specifically includes: sparsely sampling the audio-visual samples using a preset sampling frame rate, randomly selecting multiple consecutive video frames to form a video segment as the video image frame sample; sampling the audio-visual samples using a preset audio sampling rate to obtain the audio signal sample; obtaining the Mel spectrum corresponding to the audio signal sample; normalizing the Mel spectrum, and extracting the Mel spectrum based on the timestamp information of the randomly selected multiple consecutive video frames to obtain a Mel spectrum segment time-aligned with the randomly selected multiple consecutive video frames, thereby constructing the audio Mel spectrum sample.

[0011] According to the present invention, the encoder of the audio-visual vector quantization autoencoder includes a visual encoder and an audio encoder, and the decoder of the audio-visual vector quantization autoencoder includes a visual decoder and an audio decoder.

[0012] The audio / video vector quantization autoencoder is trained using the following steps: Step 201, obtaining audio Mel spectrum samples and video image frame samples corresponding to any text sample; Step 202, inputting the audio Mel spectrum samples into the audio encoder to obtain audio quantization encoding; inputting the audio quantization encoding into the audio decoder to obtain audio Mel spectrum reconstruction samples; Step 203, inputting the video image frame samples into the visual encoder to obtain visual quantization encoding; inputting the visual quantization encoding into the visual decoder to obtain video image frame reconstruction samples; Step 204, pre-training the audio / video vector quantization autoencoder using the losses between the audio Mel spectrum reconstruction samples and the audio Mel spectrum samples, and between the video image frame reconstruction samples and the video image frame samples; iteratively executing steps 201 to 204 until the pre-training of the audio / video vector quantization autoencoder is completed, obtaining the trained audio / video vector quantization autoencoder; the losses include reconstruction loss, quantization encoding loss, perceptual loss, and adversarial loss.

[0013] According to a method for generating audio video provided by the present invention, before acquiring the audio quantization code and the visual quantization code, the method further includes: acquiring visual features extracted by the visual encoder and audio features extracted by the audio encoder; associating the visual features and the audio features through a cross-modal attention module to acquire global features of video image frames and global features of audio spectrum frames; and training the visual encoder and the audio encoder using a hybrid contrastive learning loss between the global features of video image frames and the visual features, and between the global features of audio spectrum frames and the audio features.

[0014] According to the present invention, a method for generating audio-visual sequences is provided. The autoregressive audio-visual sequence generation model is trained using the following steps: Step 301, obtaining audio Mel-frequency spectrogram samples and video image frame samples corresponding to any text sample, and obtaining text identifier sequence samples corresponding to any text sample; Step 302, inputting the audio Mel-frequency spectrogram samples and the video image frame samples into the audio-visual vector quantization autoencoder to obtain discrete identifier sequence samples of video image frames and discrete identifier sequence samples of audio spectrograms; Step 303, constructing a three-modal joint training sample using the text identifier sequence samples, the discrete identifier sequence samples of video image frames, and the discrete identifier sequence samples of audio spectrograms; Step 304, performing autoregressive training on the autoregressive audio-visual sequence generation model using the three-modal joint training sample; iteratively executing steps 301 to 304 until the pre-training of the autoregressive audio-visual sequence generation model is completed, and obtaining the trained autoregressive audio-visual sequence generation model.

[0015] According to the present invention, a method for generating audio video includes step 303, which specifically includes: performing frame-by-frame splicing processing on the video image frame discrete identifier sequence sample and the audio spectrum discrete identifier sequence sample in chronological order to obtain a spliced ​​bimodal identifier sequence; splicing the text identifier sequence sample and the bimodal identifier sequence to obtain a spliced ​​trimodal identifier sequence; and obtaining the trimodal joint training sample based on the spliced ​​trimodal identifier sequence and a preset sequence length.

[0016] According to the present invention, a method for generating audio-visual content includes preprocessing the text to be inferred to obtain a text identifier sequence corresponding to the text to be inferred. Specifically, this includes encoding the text to be inferred based on a byte pair encoding method to obtain a text identifier sequence corresponding to the text to be inferred.

[0017] Secondly, the present invention also provides an audio-visual generation device, comprising: a text processing module for preprocessing the text to be reasoned to obtain a text identifier sequence corresponding to the text to be reasoned; and a video generation module for inputting the text identifier sequence into an audio-visual generation model to generate an audio-visual video corresponding to the text to be reasoned; wherein the audio-visual generation model includes an autoregressive audio-visual sequence generation model, an audio-visual vector quantization autoencoder, a video frame interpolation model, an audio conversion model, and an audio-visual processing tool.

[0018] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the above-described methods for generating audio-visual content.

[0019] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described methods for generating audio-visual content.

[0020] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described methods for generating audio-visual content.

[0021] The present invention provides a method, apparatus, electronic device, and storage medium for generating audio-visual videos. By generating audio-visual videos through an audio-visual video generation model, it is possible to generate video signals and audio signals simultaneously. Then, based on the video signals and audio signals, it synthesizes audio-visual videos that match the semantics of the text to be inferred and have good generalization. It can effectively focus on audio modal information, provide important data for artificial intelligence research, and effectively meet the user's needs. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the audio-visual generation method provided by the present invention;

[0024] Figure 2 This is a schematic diagram illustrating the principle of the audio-visual generation model provided by the present invention;

[0025] Figure 3 This is a schematic diagram of the structure of the audio-visual generation model provided by the present invention;

[0026] Figure 4 This is a schematic diagram of the pre-training process of the audio / video vector quantization autoencoder provided by the present invention;

[0027] Figure 5 This is a schematic diagram of the pre-training process of the autoregressive audio and video sequence generation model provided by the present invention;

[0028] Figure 6 This is a schematic diagram of the structure of the audio-visual generation device provided by the present invention;

[0029] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0031] It should be noted that in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two elements. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0032] In this invention, the term "and / or" means at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0033] The following is combined with Figures 1-7 This invention describes the audio-visual generation method, apparatus, electronic device, and storage medium provided in embodiments of the present invention.

[0034] It should be noted that the text-to-video generation described in this invention refers to general open-domain text-to-video generation. Based on a general audio-video generation model pre-trained on a large-scale dataset, it can complete the cross-modal generation task of synthesizing audio-videos corresponding to the semantics of the input text, regardless of the content type. In particular, the audio-video generation of this invention refers to the simultaneous modeling of video image signals and audio signals using a unified model.

[0035] Text-guided video generation has broad application prospects, such as demonstration video generation, data augmentation, and virtual reality. In recent years, with the research on two-stage generation methods based on Vector Quantized Variational Auto-encoder (VQVAE) discrete encoding and Transformer sequence generation, and the application of large-scale multimodal pre-training techniques, text-to-video generation has made great progress.

[0036] Research has revealed a significant limitation in current video generation methods: video is a multimodal data format that combines visual and auditory elements, yet existing methods focus only on generating video images while neglecting the crucial audio modality.

[0037] In view of this, the present invention proposes a method, apparatus, electronic device and storage medium for generating audio video, which unifies the generation of video image signals and audio signals into a single framework, and for the first time realizes the generation of audio video from text.

[0038] The generation of audio-visual content from text requires that the input text, the generated video image, and the generated audio satisfy a semantic consistency relationship. Therefore, it is very important to consider the cross-modal relationship modeling between audio and video at each stage of encoding and decoding.

[0039] Further research revealed a co-occurrence consistency relationship between visual and audio content, meaning that video and audio content with the same semantics often co-occur. Based on this finding, this invention proposes to enhance the encoding capability of the audio-video dual-stream vector quantization autoencoder by incorporating a cross-modal attention mechanism and a hybrid contrastive learning loss during the encoding stage. Furthermore, during the Transformer generation stage, it proposes to construct cross-modal associations using alternating sequence patterns.

[0040] Figure 1 This is a flowchart illustrating the audio-visual generation method provided by the present invention, as shown below. Figure 1 As shown, including but not limited to the following steps:

[0041] Step 11: Preprocess the text to be inferred to obtain the text identifier sequence corresponding to the text to be inferred.

[0042] Specifically, the text to be reasoned described in this invention refers to text information used for generating audio-visual videos, which can use one or more sentences of natural language to semantically describe the audio-visual video to be generated.

[0043] Furthermore, the text identifier sequence corresponding to the text to be reasoned can be a discretized element identifier sequence obtained after encoding the text to be reasoned.

[0044] Optionally, a vocabulary can be constructed first. For the text statements in the text to be inferred, they can be split into words in the constructed vocabulary, and the position number of each word in the vocabulary can be taken as its identifier. In this way, the original text string information of the text to be processed can be converted into an identifier sequence, and then the text identifier sequence corresponding to the text to be inferred can be obtained.

[0045] Step 12: Input the text identifier sequence into the audio video generation model to generate the audio video corresponding to the text to be inferred.

[0046] The audio-visual generation model includes an autoregressive audio-visual sequence generation model, an audio-visual vector quantization autoencoder, a video frame interpolation model, an audio conversion model, and audio-visual processing tools.

[0047] Specifically, once the text identifier sequence of the text to be reasoned is obtained, the text identifier sequence can be input into the audio video generation model to generate an audio video that corresponds to the semantics of the text to be reasoned.

[0048] The audio-visual generation model includes an autoregressive audio-visual sequence generation model, an audio-visual vector quantization autoencoder, a video frame interpolation model, an audio conversion model, and audio-visual processing tools.

[0049] Furthermore, the autoregressive audio / video sequence generation model can be used to identify the text identifier sequence corresponding to the user-input text to be inferred, and generate visual and audio identifier sequences through text semantic understanding and cross-modal relationships. Cross-modal relationships refer to the associations between multimodal information, which includes text, video / image, and audio information.

[0050] Audio-video vector quantization autoencoders can be used to restore visual and audio identifier sequences into visual image frames and audio Mel spectra, achieving simultaneous modeling of vision and audio based on the co-occurrence consistency relationship between vision and audio. Here, a video image frame refers to an RGB image sparsely sampled at a fixed frame rate, i.e., one frame is taken at fixed intervals. The audio Mel spectra refer to the Mel spectrum corresponding to the audio at a time-aligned length with the video image frames.

[0051] The video frame interpolation model and audio conversion model further process the visual image frames and audio Mel spectrum to generate silent video and audio signals, and then use audio and video processing tools to synthesize the silent video and audio signals to generate a video with sound.

[0052] Optionally, the above-mentioned audio video generation model can be pre-trained using the AudioSet-Cap dataset with manually annotated text descriptions, or a domain-specific dataset can be used as needed.

[0053] The audio video generation method provided by this invention generates audio video through an audio video generation model, which can simultaneously generate video and audio signals. Then, based on the video and audio signals, it synthesizes an audio video with good generalization that matches the semantics of the text to be inferred. It can effectively focus on audio modal information, provide important data for artificial intelligence research, and effectively meet the user's needs.

[0054] Based on the above embodiments, as an optional embodiment, the text identifier sequence is input into the audio video generation model to generate an audio video corresponding to the text to be inferred, including:

[0055] Step 21: Input the text identifier sequence into the autoregressive audio-visual sequence generation model to generate the discrete identifier sequence of video image frames and the discrete identifier sequence of audio spectrum corresponding to the text identifier sequence.

[0056] Step 22: Input the discrete identifier sequence of video image frames and the discrete identifier sequence of audio spectrum into the decoder of the audio-video vector quantization autoencoder to generate video image frames and audio Mel spectrum.

[0057] Step 23: Input the generated video image frames into the video frame interpolation model to synthesize a silent video.

[0058] Step 24: Input the audio Mel spectrum into the audio conversion model to synthesize the audio signal; the duration of the silent video is matched with the duration of the audio signal.

[0059] Step 25: Input the silent video and audio signals into the audio and video processing tool to generate a video with sound.

[0060] Figure 2 This is a schematic diagram illustrating the principle of the audio-visual generation model provided by the present invention, as shown below. Figure 2 As shown, including but not limited to the following steps:

[0061] First, the text to be inferred is preprocessed using a byte-to-byte encoding method to obtain the text identifier sequence corresponding to the text to be inferred.

[0062] Then, the text identifier sequence corresponding to the text to be inferred is input into the autoregressive audio-visual sequence generation model in the audio-visual generation model to generate a discrete identifier sequence of video image frames and a discrete identifier sequence of audio spectrum corresponding to the text identifier sequence.

[0063] The discrete identifier sequence of the video image frame and the discrete identifier sequence of the audio spectrum are then input into the decoder of the audio-video vector quantization autoencoder to generate the video image frame and the audio Mel spectrum.

[0064] Furthermore, video image frames are input into a video interpolation model to synthesize a silent video.

[0065] Simultaneously, the audio Mel spectrum is input into the audio conversion model to synthesize the audio signal. The duration of the silent video is matched with the duration of the audio signal to ensure semantic consistency.

[0066] Finally, the silent video and audio signals are input into the audio and video processing tool to generate a video with sound.

[0067] As an optional embodiment, the autoregressive audio and video sequence generation model is an autoregressive trimodal joint Transformer neural network Decoder model.

[0068] Among them, the three modalities refer to text, video and audio. The autoregressive three-modal combined with the Transformer neural network Decoder model has good sequence modeling capabilities.

[0069] As an optional embodiment, the audio / video vector quantization autoencoder in this invention can be an SVG-VQGAN model. Here, SVG refers to a Sounding Video Generator (SVG), and the SVG-VQGAN model refers to an audio / video vector quantization autoencoder.

[0070] As an optional embodiment, the audio conversion model in this invention can be a HiFiGAN decoder. The HiFiGAN model can convert the spectrum into high-quality audio and has a good convergence speed.

[0071] As an optional embodiment, the audio and video processing tool in this invention can be the ffmpeg multimedia processing tool, which supports various operations on video and audio (such as video capture and video format conversion) and is equipped with a free, powerful, and massive command library.

[0072] As an optional embodiment, the video frame interpolation model in this invention is built based on a frame interpolation neural network model, such as the open-source FILM model.

[0073] Optionally, Figure 3 This is a schematic diagram of the structure of the audio-visual generation model provided by the present invention, as shown below. Figure 3As shown, [TXT] is the text start identifier, the text identifier sequence is {[TXT],t[1],…,t[L]}, and the length is L. It is input into the autoregressive trimodal joint Transformer neural network Decoder model. [V1] and [A1] are the start identifiers of the discrete identifier sequence of the first frame video image frame and the first frame audio spectrum discrete identifier sequence, respectively. Starting from [V1], the first frame visual identifier sequence {[V1],v1[1],…,v1[M]} is generated, with a length of M. That is, each frame video image frame discrete identifier sequence consists of M visual identifiers. The generation method is to predict the next identifier in an autoregressive manner. That is, when the first identifier v1[1] is generated, v1[1] is called in an autoregressive manner to predict the generation of v1[2], and then v1[2] is called in an autoregressive manner to predict the generation of v1[3]. The above steps are iterated to generate the first frame video image frame discrete identifier sequence.

[0074] Similarly, starting from [A1], the first frame audio spectrum discrete identifier sequence {[A1], a1[1], ..., a1[N]} is generated with a length of N. That is, each frame audio spectrum discrete identifier sequence consists of N audio identifiers. The generation method is also to predict the next identifier in an autoregressive manner. That is, when the first identifier a1[1] is generated, a1[1] is called in an autoregressive manner to predict the generation of a1[2]. The above steps are iterated to generate the first frame audio spectrum discrete identifier sequence. Finally, by repeating the above steps, the generation of a total of K frames of audio video sequence is completed.

[0075] The audio video sequence consists of a discrete identifier sequence of video image frames and a discrete identifier sequence of audio spectrum.

[0076] Through the above steps, the discrete identifier sequences of video image frames and audio spectrum corresponding to the text identifier sequence can be obtained and input into the encoder of the audio-video vector quantization autoencoder. Specifically, the discrete identifier sequence of video image frames is input into the visual decoder to generate video image frames, and the discrete identifier sequence of audio spectrum is input into the audio decoder to generate audio Mel-frequency spectra.

[0077] Furthermore, video image frames are input into a video interpolation model to synthesize silent video, and audio Mel spectrum is input into a HiFiGAN decoder to synthesize audio signals.

[0078] The duration of the silent video is matched with the duration of the audio signal.

[0079] Finally, the silent video and audio signals are integrated using the ffmpeg multimedia processing tool to generate a video with sound.

[0080] The audio video generation method provided by this invention, through an autoregressive audio video sequence generation model, an audio video vector quantization autoencoder, a video frame interpolation model, an audio conversion model, and audio video processing tools, can effectively generate audio videos based on text information, effectively focus on audio modal information, and meet user needs.

[0081] Based on the above embodiments, as an optional embodiment, the audio conversion model is trained using the following steps:

[0082] Step 101: Preprocess the audio and video samples corresponding to each text sample to obtain the audio signal samples and video image frame samples corresponding to each audio and video sample, and obtain the audio Mel spectrum samples corresponding to each audio signal sample.

[0083] Step 102: Use any audio Mel spectrum sample as the input to the audio conversion model to be trained, and use the audio signal sample corresponding to any audio Mel spectrum sample as the output label of the audio conversion model to be trained, and pre-train the audio conversion model to be trained.

[0084] Iteratively execute step 102 until the pre-training of the audio conversion model to be trained is completed, and the trained audio conversion model is obtained.

[0085] Specifically, in this embodiment, multiple text samples can be selected from the AudioSet-Cap dataset with manually annotated text descriptions, and the audio and video samples corresponding to each text sample can be preprocessed to obtain audio signal samples and video image frame samples corresponding to each audio and video sample. For example, the first half of the audio and video samples can be pre-selected as audio signal samples and video image frame samples.

[0086] Furthermore, the above-described implementation of obtaining audio Mel spectrum samples corresponding to each audio signal sample can obtain a linear spectrum of the audio signal sample by performing a fast Fourier transform on the audio signal sample, and then convert the linear spectrum of the audio signal sample into the corresponding audio Mel spectrum sample by using a filter bank with Mel frequency scale.

[0087] Then, any audio Mel spectrum sample is used as the input to the audio conversion model to be trained, and the audio signal sample corresponding to any audio Mel spectrum sample is used as the output label of the audio conversion model to be trained, so as to pre-train the audio conversion model to be trained.

[0088] Furthermore, the pre-training steps of the audio conversion model to be trained are iteratively executed until the pre-training of the audio conversion model to be trained is completed, thereby obtaining a trained audio conversion model. The completion of pre-training of the audio conversion model to be trained can be determined by preset training conditions. For example, the number of pre-training iterations of the audio conversion model to be trained can be preset to 100. After iteratively executing the pre-training steps of the audio conversion model to be trained 100 times, the pre-training of the audio conversion model to be trained can be considered complete.

[0089] The audio video generation method provided by this invention improves the conversion accuracy of the audio conversion model by pre-training it, thereby improving the accuracy of the generated audio signal, so as to facilitate the subsequent synthesis of audio videos with better semantic matching with the text.

[0090] Based on the above embodiments, as an optional embodiment, step 101 preprocesses the audio and video samples corresponding to each text sample to obtain the audio signal samples and video image frame samples corresponding to each audio and video sample, and obtains the audio Mel-frequency spectrum samples corresponding to each audio signal sample, specifically including:

[0091] Using a preset sampling frame rate, sparse sampling is performed on the audio and video samples, and multiple consecutive video frames are randomly selected to form a video segment, which serves as the video image frame sample.

[0092] The audio and video samples are sampled using a preset audio sampling rate to obtain audio signal samples.

[0093] Obtain the Mel spectrum corresponding to the audio signal sample.

[0094] The Mel spectrum is normalized, and Mel spectrum is extracted based on the timestamp information of multiple randomly selected consecutive video frames to obtain Mel spectrum segments that are time-aligned with the timestamps of the multiple randomly selected consecutive video frames, thus constructing audio Mel spectrum samples.

[0095] Specifically, by using a preset sampling frame rate, sparse sampling of audio and video samples can be performed, randomly selecting multiple consecutive video frames to form a video segment, which serves as the video image frame sample. The preset sampling frame rate can be pre-set according to the specific usage requirements of the scenario, for example, set to 8 frames per second.

[0096] It should be noted that obtaining video image frame samples through sparse sampling has the advantage of faster computational convergence. Furthermore, continuous silent video with a higher frame rate can be obtained through video frame interpolation models.

[0097] Furthermore, audio and video samples can be sampled using a preset audio sampling rate to obtain audio signal samples. The preset audio sampling rate can be pre-set according to the specific usage requirements of the scenario.

[0098] Once an audio signal sample is obtained, its linear spectrum can be obtained through a Fast Fourier Transform (FFT). Then, a filter bank with Mel frequency scales can be used to convert the linear spectrum of the audio signal sample into the corresponding Mel spectrum.

[0099] Finally, based on the Mel spectrum corresponding to the obtained audio signal sample, normalization processing is performed, and according to the timestamp information of multiple randomly selected consecutive video frames, the Mel spectrum corresponding to the audio signal sample is extracted to obtain Mel spectrum segments that are time-aligned with multiple randomly selected consecutive video frames, and audio Mel spectrum samples are constructed to ensure that the video image and audio satisfy the semantic consistency relationship.

[0100] The audio video generation method provided by this invention preprocesses audio and video samples to obtain semantically consistent video image frame samples and audio Mel spectrum samples, which facilitates the pre-training of the audio conversion model and improves the performance of the audio conversion model.

[0101] Based on the above embodiments, as an optional embodiment, the encoder of the audio / video vector quantization autoencoder includes a visual encoder and an audio encoder, and the decoder of the audio / video vector quantization autoencoder includes a visual decoder and an audio decoder.

[0102] The audio / video vector quantization autoencoder is trained using the following steps:

[0103] Step 201: Obtain the audio Mel spectrum sample and video image frame sample corresponding to any text sample.

[0104] Step 202: Input the audio Mel spectrum sample into the audio encoder to obtain the audio quantization code; input the audio quantization code into the audio decoder to obtain the audio Mel spectrum reconstruction sample.

[0105] Step 203: Input the video image frame samples into the visual encoder to obtain the visual quantization code; input the visual quantization code into the visual decoder to obtain the video image frame reconstruction samples.

[0106] Step 204: Use the loss between audio Mel spectrum reconstruction samples and audio Mel spectrum samples, and between video image frame reconstruction samples and video image frame samples to pre-train the audio-video vector quantization autoencoder.

[0107] Iteratively execute steps 201 to 204 until the pre-training of the audio-video vector quantization autoencoder is completed, and the trained audio-video vector quantization autoencoder is obtained.

[0108] The losses include reconstruction loss, quantization encoding loss, perception loss, and adversarial loss.

[0109] Figure 4 This is a schematic diagram of the audio / video vector quantization autoencoder pre-training process provided by the present invention, as shown below. Figure 4 As shown, it mainly includes:

[0110] First, the audio and video samples corresponding to any text sample are preprocessed to obtain the audio Mel spectrum samples and video image frame samples corresponding to any text sample.

[0111] Then, the audio Mel-spectrum samples are input to the audio encoder of the audio-video vector quantization autoencoder to obtain the audio quantization code. The audio quantization code includes quantization features and a discrete identifier sequence of audio Mel-spectrum samples corresponding to the quantization features. The audio quantization code is then input to the audio decoder of the audio-video vector quantization autoencoder to obtain the reconstructed audio Mel-spectrum samples.

[0112] Similarly, video image frame samples are input into the visual encoder of the audio-video vector quantization autoencoder to obtain visual quantization encoding. This visual quantization encoding includes quantization features and a discrete identifier sequence of video image frame samples corresponding to the quantization features. The visual quantization encoding is then input into the visual decoder of the audio-video vector quantization autoencoder to obtain reconstructed video image frame samples.

[0113] Finally, the audio-video vector quantization autoencoder is pre-trained using the losses between audio Mel-spectrum reconstruction samples and between video image frame reconstruction samples and video image frame samples. These losses include reconstruction loss, quantization encoding loss, perceptual loss, and adversarial loss.

[0114] By iteratively executing the above pre-training steps until the pre-training of the audio / video vector quantization autoencoder is completed, a trained audio / video vector quantization autoencoder is obtained.

[0115] The pre-training of the audio / video vector quantization autoencoder can be determined by setting pre-defined training conditions. For example, the training objective can be pre-set as follows: minimizing the reconstruction loss between video image frame reconstruction samples and between audio Mel-spectrum reconstruction samples; minimizing the quantization loss of the audio / video vector quantization autoencoder during quantization encoding; maximizing the generator loss during generative adversarial training; and minimizing the discriminator loss.

[0116] Alternatively, the above loss can be calculated using the following formulas (1) and (2):

[0117]

[0118]

[0119] in, The loss of the audio-video vector quantization autoencoder between the video image frame reconstruction sample and the video image frame sample; The loss of the autoencoder is used to quantize the audio-video vectors between the audio Mel spectrum reconstruction samples and the audio Mel spectrum samples; v i:j These are video image frame samples from frame i to frame j; For reconstructing video image frames from frame i to frame j; m i:j The audio Mel spectrum samples are from frame i to frame j. The audio Mel-frequency spectrum reconstructs samples from frame i to frame j; sg is the stopping gradient operation; CNN is the feature extraction module for the image classifier; β is a preset constant that can be obtained in advance through experiments; D v To generate a visual discriminator in adversarial training; D a To generate an audio discriminator in adversarial training.

[0120] Based on the audio Mel spectrum reconstruction samples and audio Mel spectrum samples, and the video image frame reconstruction samples and video image frame samples, the loss of the audio-video vector quantization autoencoder can be calculated using formulas (1) and (2). After reaching the preset training objective, the pre-training of the audio-video vector quantization autoencoder can be considered complete.

[0121] The audio video generation method provided by this invention determines whether the pre-training of the audio-video vector quantization autoencoder is completed by judging whether the loss of the audio-video vector quantization autoencoder during the pre-training process meets the preset training objective. This can effectively improve the performance of the audio-video vector quantization autoencoder, so as to facilitate the subsequent synthesis of audio videos with better semantic matching with the text.

[0122] Based on the above embodiments, as an optional embodiment, before obtaining audio quantization encoding and visual quantization encoding, the method further includes:

[0123] Obtain visual features extracted by the visual encoder and audio features extracted by the audio encoder.

[0124] By associating visual and audio features through a cross-modal attention module, global features of video image frames and global features of audio spectrum frames are obtained.

[0125] The visual encoder and audio encoder are trained by using a mixed contrastive learning loss between global features and visual features of video image frames, and global features and audio features of audio spectrum frames.

[0126] Furthermore, research has found that only the basic... and As for the loss, there is a lack of cross-modal association in the quantization coding stage. Cross-modal association is very important in audio and video quantization coding. Otherwise, visually similar video image frame samples may be quantized and encoded into the same video image frame sample discrete identifier sequence, while when their corresponding audio Mel spectrum samples are significantly different, different audio Mel spectrum sample discrete identifier sequences will be obtained. This limits the quantization coding and reconstruction capabilities of single mode and will also introduce noise in subsequent training.

[0127] In view of this, such as Figure 4 As shown, this embodiment of the invention proposes to obtain global features of video image frames and global features of audio spectrum frames by associating visual features extracted by the visual encoder and audio features extracted by the audio encoder through a cross-modal attention module before obtaining audio quantization encoding and visual quantization encoding.

[0128] Furthermore, the present invention also incorporates hybrid contrastive learning, including intra-modal contrastive learning and inter-modal contrastive learning, to improve the quantization coding performance of audio and video vector quantization autoencoders.

[0129] Intermodal contrastive learning is used to construct cross-modal associations and proposes basic positive and negative sample selection rules to determine how to select positive and negative samples, thereby completing intermodal contrastive learning based on the positive and negative samples. For example, different modal features in the same video image frame sample constitute positive samples, and different modal features in different video image frame samples constitute negative samples. Information from different modalities is used as anchor points to push features with different semantics further away and features with the same semantics closer together.

[0130] To prevent features from deviating from the original modality, this invention introduces intra-modal contrastive learning as a regularization term. For example, features of the same modality in unified video image frame samples are used as positive samples, while features of the same modality in different video image frame samples are used as negative samples.

[0131] Optionally, the visual encoder extracts visual features from the i-th to j-th frames of the video image frame samples. in, Let d represent the visual features of the k-th frame. Video image frame samples are quantized and encoded frame by frame. v For visual feature dimensions, and H represents the size of the visual feature map, and W represents the original image frame size. vThis is the downsampling number of the visual encoder. For example, when H = W = 128, ds v =16, that is, h=w=8, then the length of the discrete identifier sequence of visual features in each frame is M=64.

[0132] Similarly, the audio encoder extracts audio features from audio Mel-spectrum samples by segment. That is, the audio Mel-spectrum segment corresponding to the i-th to j-th frames in the video image frame sample. d a Let f×[(ji)*t] be the audio feature dimension, and f×[(ji)*t] be the size of the audio feature map, corresponding to a frame size of f×t. ds represents the size of the frequency dimension. a Where F is the downsampling number of the audio encoder, and F is the number of frequency bands divided from the original Mel spectrum. The feature map size of each audio frame in the time dimension is given by T, the duration of the video image frame sample is given by K, and the number of frames is given by K.

[0133] For example, when ds a =8, f=t=5, then the length of the discrete identifier sequence of each audio identifier is N=25. Correspondingly, the total duration of the video image frame sample is K=10 frames. According to the audio sampling rate of 22500Hz, the corresponding total duration T is about 9.259 seconds. Each audio frame corresponds to one visual image frame, so the sparse sampling frame rate of the visual image frame is 1.08fps.

[0134] It should be noted that visual features and audio features The visual encoder and audio encoder of the audio-visual vector quantization autoencoder can be further quantized into discrete identifier sequences and quantization features.

[0135] Furthermore, the cross-modal attention module can be used to construct local attention relationships, inputting visual features extracted by the visual encoder and the audio encoder. and audio features Mapping to a common semantic space yields and Furthermore, the global features of the video image frame and the global features of the audio spectrum frame are obtained for each frame through the attention layer.

[0136] The cross-modal attention layer consists of an audio-to-visual attention layer and a visual-to-audio attention layer. Each attention layer can be calculated using formula (3):

[0137]

[0138] The characteristics of the input query, key, and value are e, respectively. q ek and e v Q, K, and V are linear layers; d is the dimension of the hidden layer; h is the output attention result.

[0139] In the audio-to-visual attention layer, audio features are pooled along the temporal dimension. As a query, visual features Using these as keys and values, the location of the sound at each frequency is obtained. The results are then averaged and pooled across all frequencies to obtain the global features of the audio-related video image frames. In the visual-to-audio attention layer, visual global features are used. As a query, audio features in the public semantic space Using keys and values, global features of visually relevant audio spectrum frames can be obtained.

[0140] Furthermore, after obtaining the global features of the video image frame and the global features of the audio spectrum frame for each frame, the parameters of the visual encoder and the audio encoder can be optimized through hybrid contrastive learning. The hybrid contrastive learning loss can be calculated using formula (4):

[0141]

[0142] in, For hybrid contrastive learning loss; For comparison loss functions; H v Global features of all video image frames during a single pre-training process The set of H; a Global features of all audio spectrum frames during a single pre-training process A set of.

[0143] It should be noted that this invention proposes three refined strategies for positive and negative sample screening, including cross-modal positive sample screening based on visual-audio similarity, negative sample screening based on text semantics, and positive sample screening based on time windows.

[0144] Research has also revealed that not all video visual content and audio content are related; that is, not all visual and audio modalities of the same video image frame samples can constitute positive samples. Therefore, the embodiments provided in this invention use CLIP similarity between sentences composed of audio category words and visual image frames as the criterion for determining whether they constitute cross-modal positive samples.

[0145] Furthermore, not all features in different video image frame samples constitute negative samples, because the content of different video image frame samples may be similar or even the same. Therefore, the embodiments provided by this invention use the BERT similarity of text description as the basis for judging whether a negative sample is constituted.

[0146] Finally, not all time positions in the same video constitute positive samples, because video content may change over time. Therefore, the embodiments provided by this invention maintain the selection of positive samples within a fixed-size time window.

[0147] Based on the above-mentioned refined strategy for positive and negative sample selection, the contrastive loss function provided by this invention This can be expressed as:

[0148]

[0149]

[0150]

[0151]

[0152]

[0153] H1 and H2 are the global features of all video image frames during a single pre-training process. global features of a set or all audio spectrum frames A set; As an indicator function, when determining h l When the visual and audio similarity of the video image frame samples is too low, Otherwise, it is 1; As an indicator function, when determining h l ,h n When the text content of the video image frame samples is too similar, Otherwise, it is 1; As an indicator function, when determining h l ,h m When the timestamp positions are more than the set time window size, Otherwise, it is 1; This is the positive sample portion; ζ represents the negative sample portion; ζ is used to balance the numerical fluctuations caused by different numbers of negative samples; τ is a hyperparameter in the mixed contrastive loss learning, which can be set to a value of 0.07.

[0154] Therefore, when obtaining the hybrid contrastive learning loss Then, the loss of the audio / video vector quantization autoencoder can be calculated, which can be obtained by the following formula (5):

[0155]

[0156] in, The loss is the loss of the audio / video vector quantization autoencoder; α is the hybrid contrastive learning loss coefficient, which can be 1.0.

[0157] Therefore, the pre-set training objective can be considered as completing the pre-training of the audio-video vector quantization autoencoder when minimizing the reconstruction loss between video image frame reconstruction samples and between audio Mel spectrum reconstruction samples and between audio Mel spectrum samples, minimizing the quantization coding loss when the encoder of the audio-video vector quantization autoencoder performs quantization coding, maximizing the generator loss in generative adversarial training, and minimizing the discriminator loss. In addition, it is also necessary to minimize the hybrid contrastive learning loss.

[0158] The audio video generation method provided by this invention improves the reconstruction and generation effects by incorporating a hybrid contrastive learning method into the audio-video vector quantization autoencoder, thereby effectively utilizing multimodal information to enhance the quantization encoding capabilities of the visual encoder and audio encoder.

[0159] Based on the above embodiments, as an optional embodiment, the autoregressive audio / video sequence generation model is trained using the following steps:

[0160] Step 301: Obtain the audio Mel spectrum sample and video image frame sample corresponding to any text sample, and obtain the text identifier sequence sample corresponding to any text sample.

[0161] Step 302: Input the audio Mel spectrum sample and the video image frame sample into the audio-video vector quantization autoencoder to obtain the video image frame discrete identifier sequence sample and the audio spectrum discrete identifier sequence sample.

[0162] Step 303: Construct a three-modal joint training sample from the text identifier sequence sample, the video image frame discrete identifier sequence sample, and the audio spectrum discrete identifier sequence sample.

[0163] Step 304: Use the three-modal joint training samples to perform autoregressive training on the autoregressive audio and video sequence generation model.

[0164] Iteratively execute steps 301 to 304 until the pre-training of the autoregressive audio-video sequence generation model is completed, and obtain the trained autoregressive audio-video sequence generation model.

[0165] Figure 5 This is a schematic diagram of the pre-training process of the autoregressive audio / video sequence generation model provided by the present invention, as shown below. Figure 5 As shown, it mainly includes:

[0166] First, obtain any text sample, along with the corresponding audio Mel-frequency spectrum sample and video image frame sample. Then, obtain the text identifier sequence sample X corresponding to the text sample using the Byte Pair Encoder (BPE) method. T =[TXT],t[1]…,t[L]}.

[0167] The video image frame samples are then input into the audio / video vector quantization autoencoder, and the discrete identifier sequence samples of the video image frames are obtained through the visual encoder. The discrete identifier sequence sample of the k-th video image frame is represented as...

[0168] Similarly, the audio Mel-frequency spectrum samples are input into the audio-video vector quantization autoencoder, and the audio encoder obtains the discrete identifier sequence samples of the audio spectrum, where the discrete identifier sequence sample of the audio spectrum of the k-th frame is represented as...

[0169] Then, the text identifier sequence samples, video image frame discrete identifier sequence samples, and audio spectrum discrete identifier sequence samples are used to construct a three-modal joint training sample.

[0170] Finally, the autoregressive audio and video sequence generation model was trained using joint training samples from three modalities.

[0171] For example, such as Figure 5 As shown, a three-modal joint training sample is constructed from a text identifier sequence sample of length L, a total of K video image frame discrete identifier sequence samples with each frame having a length of M, and a total of K audio spectrum discrete identifier sequence samples with each frame having a length of N. The sample is simultaneously input into an autoregressive three-modal joint Transformer neural network Decoder model to process all input identifiers in parallel. Autoregressive pre-training is used, which is a training method that predicts the output of the next identifier by inputting a known identifier. For example, when the input is t[1] in the text identifier sequence sample, the expected output result is the next identifier t[2].

[0172] After processing all the input identifier sequences in parallel, the autoregressive loss is calculated based on the output identifier sequence.

[0173] The above steps are iteratively executed until the pre-training of the autoregressive audio-visual sequence generation model is completed, resulting in the trained autoregressive audio-visual sequence generation model. The completion of pre-training can be determined by setting a preset training objective. For example, the completion of pre-training can be determined by whether the autoregressive loss is minimized.

[0174] Alternatively, the calculation process of autoregressive loss can be expressed as:

[0175]

[0176]

[0177]

[0178]

[0179] in, Autoregressive loss for text-identified sequence samples; Autoregressive loss for discrete identifier sequence samples of video image frames; Autoregressive loss for discrete identifier sequence samples of audio spectrum; γ t γ v γ a These are the loss weights for the autoregressive loss of text identifier sequence samples, the autoregressive loss of video image frame discrete identifier sequence samples, and the autoregressive loss of audio spectrum discrete identifier sequence samples, respectively. Indicator functions for text, video, and audio identifiers, respectively; X i Let S be the i-th identifier in the identifier sequence sample X; S is the length of the identifier sequence sample X; This is the autoregressive loss weighted by the loss weight.

[0180] The audio video generation method provided by this invention improves the performance of the autoregressive audio video sequence generation model by performing autoregressive training on the model and calculating the autoregressive loss to determine whether the pre-training of the model is complete, thus facilitating the subsequent synthesis of audio videos with better semantic matching with the text.

[0181] Based on the above embodiments, as an optional embodiment, step 303 constructs a three-modal joint training sample from text identifier sequence samples, video image frame discrete identifier sequence samples, and audio spectrum discrete identifier sequence samples, specifically including:

[0182] According to the time sequence, the discrete identifier sequence samples of video image frames and discrete identifier sequence samples of audio spectrum are spliced ​​frame by frame to obtain the spliced ​​dual-modal identifier sequence.

[0183] The text identifier sequence sample and the bimodal identifier sequence are concatenated to obtain the concatenated trimodal identifier sequence.

[0184] Based on the spliced ​​trimodal identifier sequence and the preset sequence length, trimodal joint training samples are obtained.

[0185] Specifically, after obtaining discrete identifier sequence samples of video image frames and discrete identifier sequence samples of audio spectrum, frame-by-frame splicing can be performed to obtain the spliced ​​bimodal identifier sequence.

[0186] For example, the discrete identifier sequence sample of the k-th video image frame is The discrete identifier sequence sample of the audio spectrum of the k-th frame is Then, the discrete identifier sequence sample of the k-th frame video image and the discrete identifier sequence sample of the k-th frame audio spectrum are concatenated to form the k-th frame bimodal identifier sequence. Similarly, the discrete identifier sequence samples of the (k+1)th frame of the video image can be used. and the discrete identifier sequence sample of the (k+1)th video image frame The splicing process generates a dual-modal identifier sequence for the (k+1)th frame.

[0187] Furthermore, the text identifier sequence sample and the bimodal identifier sequence are concatenated to obtain the concatenated trimodal identifier sequence;

[0188] For example, the obtained text identifier sequence sample X T After being concatenated with the bimodal identifier sequence, a trimodal identifier sequence is formed.

[0189] Finally, based on the concatenated trimodal identifier sequence and the preset sequence length, trimodal joint training samples are obtained. The preset sequence length can be pre-set according to the specific usage requirements of the scenario, for example, 50 frames.

[0190] It should be noted that generating trimodal joint training samples according to this alternating modal sequence format, focusing on the information of the three modalities from the first frame, not only maintains temporal alignment, but also improves the generation quality of discrete identifier sequences through multimodal information, so as to facilitate the subsequent synthesis of audio videos with better semantic matching with the text.

[0191] The audio video generation method provided by this invention constructs a trimodal joint training sample by combining information from three modalities. It can always pay attention to multimodal information, not only maintaining temporal alignment, but also improving the generation quality of discrete identifier sequences through multimodal information, so as to facilitate the subsequent synthesis of audio videos with better semantic matching with text.

[0192] Based on the above embodiments, as an optional embodiment, the text to be reasoned is preprocessed to obtain a text identifier sequence corresponding to the text to be reasoned, specifically including:

[0193] Based on the byte pair encoding method, the text to be inferred is encoded to obtain the text identifier sequence corresponding to the text to be inferred.

[0194] Specifically, the Byte-Pair Encoding (BPE) method is a common form of data compression encoding. It can construct a vocabulary to discretize and encode the characters, phrases, etc. of the text sentences in the text to be inferred, determine the identifiers of each character, phrase, etc., and thus obtain the text identifier sequence of the text to be inferred.

[0195] The audio-visual generation method provided by this invention can effectively identify the semantics of the text to be inferred through a byte pair encoding method, thereby generating a text identifier sequence with better semantic matching.

[0196] Figure 6 This is a schematic diagram of the structure of the audio-visual generation device provided by the present invention, as shown below. Figure 6 As shown, it mainly includes: a text processing module 601 and a video generation module 602, wherein:

[0197] The text processing module 601 is used to preprocess the text to be reasoned to obtain the text identifier sequence corresponding to the text to be reasoned.

[0198] The video generation module 602 is used to input the text identifier sequence into the audio video generation model to generate an audio video corresponding to the text to be inferred.

[0199] The audio-visual generation model includes an autoregressive audio-visual sequence generation model, an audio-visual vector quantization autoencoder, a video frame interpolation model, an audio conversion model, and audio-visual processing tools.

[0200] It should be noted that the audio-visual generation apparatus provided in this embodiment of the invention can execute the audio-visual generation method described in any of the above embodiments during specific operation, and this embodiment will not elaborate on this.

[0201] The audio-visual generation device provided by this invention generates audio-visual videos through an audio-visual generation model. It can generate video signals and audio signals simultaneously, and then synthesize audio-visual videos with good generalization that match the semantics of the text to be inferred based on the video signals and audio signals. It can effectively focus on audio modal information, provide important data for artificial intelligence research, and effectively meet the user's needs.

[0202] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 7As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a video-on-audio generation method. This method includes: preprocessing the text to be reasoned to obtain a text identifier sequence corresponding to the text to be reasoned; inputting the text identifier sequence into a video-on-audio generation model to generate a video-on-audio corresponding to the text to be reasoned; wherein the video-on-audio generation model includes an autoregressive audio-visual sequence generation model, an audio-visual vector quantization autoencoder, a video frame interpolation model, an audio conversion model, and audio-visual processing tools.

[0203] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0204] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the audio-visual generation method provided in the above embodiments, the method including: preprocessing the text to be reasoned to obtain a text identifier sequence corresponding to the text to be reasoned; inputting the text identifier sequence into an audio-visual generation model to generate an audio-visual video corresponding to the text to be reasoned; wherein, the audio-visual generation model includes an autoregressive audio-visual sequence generation model, an audio-visual vector quantization autoencoder, a video frame interpolation model, an audio conversion model, and an audio-visual processing tool.

[0205] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the audio-visual generation method provided in the above embodiments. The method includes: preprocessing the text to be reasoned to obtain a text identifier sequence corresponding to the text to be reasoned; inputting the text identifier sequence into an audio-visual generation model to generate an audio-visual video corresponding to the text to be reasoned; wherein the audio-visual generation model includes an autoregressive audio-visual sequence generation model, an audio-visual vector quantization autoencoder, a video frame interpolation model, an audio conversion model, and an audio-visual processing tool.

[0206] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0207] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0208] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating video with sound, characterized in that, include: The text to be reasoned is preprocessed to obtain the text identifier sequence corresponding to the text to be reasoned; The text identifier sequence is input into the audio video generation model to generate the audio video corresponding to the text to be inferred; The audio-visual generation model includes an autoregressive audio-visual sequence generation model, an audio-visual vector quantization autoencoder, a video frame interpolation model, an audio conversion model, and audio-visual processing tools. The autoregressive audio-visual sequence generation model is an autoregressive trimodal joint Transformer neural network Decoder model. This model is used to identify the text identifier sequence corresponding to the user-input text to be inferred, and to generate visual and audio identifier sequences through text semantic understanding and cross-modal relationships. The cross-modal relationships refer to the association between multimodal information, which includes text information, video image information, and audio information. The step of inputting the text identifier sequence into the audio video generation model to generate the audio video corresponding to the text to be inferred includes: Step 21: Input the text identifier sequence into the autoregressive audio-visual sequence generation model to generate a discrete identifier sequence of video image frames and a discrete identifier sequence of audio spectrum corresponding to the text identifier sequence; Step 22: Input the discrete identifier sequence of the video image frame and the discrete identifier sequence of the audio spectrum into the decoder of the audio-video vector quantization autoencoder to generate video image frames and audio Mel spectrum; Step 23: Input the generated video image frames into the video frame interpolation model to synthesize a silent video; Step 24: Input the audio Mel spectrum into the audio conversion model to synthesize an audio signal; the duration of the silent video matches the duration of the audio signal. Step 25: Input the silent video and the audio signal into the audio and video processing tool to generate the video with sound.

2. The method for generating audio-visual content according to claim 1, characterized in that, The audio / video vector quantization autoencoder is an SVG-VQGAN model. And / or, the audio conversion model is a HiFiGAN decoder; And / or, the audio and video processing tool is the ffmpeg multimedia processing tool; And / or, the video frame interpolation model is constructed based on a frame interpolation neural network model.

3. The method for generating audio-visual content according to claim 1, characterized in that, The audio conversion model is trained using the following steps: Step 101: Preprocess the audio and video samples corresponding to each text sample to obtain the audio signal samples and video image frame samples corresponding to each audio and video sample, and obtain the audio Mel spectrum samples corresponding to each audio signal sample. Step 102: Take any of the audio Mel spectrum samples as input to the audio conversion model to be trained, and take the audio signal sample corresponding to any of the audio Mel spectrum samples as output label of the audio conversion model to be trained, and pre-train the audio conversion model to be trained. Iteratively execute step 102 until the pre-training of the audio conversion model to be trained is completed, and the trained audio conversion model is obtained.

4. The method for generating audio-visual content according to claim 3, characterized in that, Step 101 specifically includes: Using a preset sampling frame rate, the audio and video samples are sparsely sampled, and multiple consecutive video frames are randomly selected to form a video segment, which is used as the video image frame sample. The audio and video samples are sampled using a preset audio sampling rate to obtain the audio signal samples; Obtain the Mel spectrum corresponding to the audio signal sample; The Mel spectrum is normalized, and based on the timestamp information of the randomly selected multiple consecutive video frames, the Mel spectrum is extracted to obtain Mel spectrum segments that are time-aligned with the randomly selected multiple consecutive video frames, thus constructing the audio Mel spectrum sample.

5. The method for generating audio-visual content according to claim 3, characterized in that, The encoder of the audio / video vector quantization autoencoder includes a visual encoder and an audio encoder, and the decoder of the audio / video vector quantization autoencoder includes a visual decoder and an audio decoder. The audio / video vector quantization autoencoder is trained using the following steps: Step 201: Obtain the audio Mel spectrum sample and video image frame sample corresponding to any text sample; Step 202: Input the audio Mel spectrum sample into the audio encoder to obtain the audio quantization code; The audio quantization code is input into the audio decoder to obtain audio Mel-spectrum reconstruction samples; Step 203: Input the video image frame samples into the visual encoder to obtain visual quantization codes; The visual quantization code is input to the visual decoder to obtain video image frame reconstruction samples; Step 204: Use the loss between the audio Mel spectrum reconstruction samples and the audio Mel spectrum samples, and between the video image frame reconstruction samples and the video image frame samples to pre-train the audio-video vector quantization autoencoder; Iteratively execute steps 201 to 204 until the pre-training of the audio-video vector quantization autoencoder is completed, and the trained audio-video vector quantization autoencoder is obtained. The losses include reconstruction loss, quantization encoding loss, perception loss, and adversarial loss.

6. The method for generating audio-visual content according to claim 5, characterized in that, Before acquiring the audio quantization code and the visual quantization code, the process further includes: Obtain the visual features extracted by the visual encoder and the audio features extracted by the audio encoder; By associating the visual features and the audio features through a cross-modal attention module, global features of video image frames and global features of audio spectrum frames are obtained; The visual encoder and the audio encoder are trained using a mixed contrastive learning loss between the global features of the video image frame and the visual features, and between the global features of the audio spectrum frame and the audio features.

7. The method for generating audio-visual content according to any one of claims 3-6, characterized in that, The autoregressive audio / video sequence generation model is trained using the following steps: Step 301: Obtain the audio Mel spectrum sample and video image frame sample corresponding to any text sample, and obtain the text identifier sequence sample corresponding to any text sample; Step 302: Input the audio Mel spectrum samples and the video image frame samples into the audio-video vector quantization autoencoder to obtain video image frame discrete identifier sequence samples and audio spectrum discrete identifier sequence samples; Step 303: Construct a three-modal joint training sample from the text identifier sequence sample, the video image frame discrete identifier sequence sample, and the audio spectrum discrete identifier sequence sample; Step 304: Use the three-modal joint training samples to perform autoregressive training on the autoregressive audio and video sequence generation model; Iteratively execute steps 301 to 304 until the pre-training of the autoregressive audio-video sequence generation model is completed, and obtain the trained autoregressive audio-video sequence generation model.

8. The method for generating audio-visual content according to claim 7, characterized in that, Step 303 specifically includes: According to the time sequence, the video image frame discrete identifier sequence sample and the audio spectrum discrete identifier sequence sample are spliced ​​frame by frame to obtain the spliced ​​dual-modal identifier sequence. The text identifier sequence sample and the bimodal identifier sequence are concatenated to obtain the concatenated trimodal identifier sequence; Based on the spliced ​​trimodal identifier sequence and the preset sequence length, the trimodal joint training samples are obtained.

9. The method for generating audio-visual content according to claim 1, characterized in that, The preprocessing of the text to be reasoned, to obtain the text identifier sequence corresponding to the text to be reasoned, specifically includes: Based on the byte pair encoding method, the text to be inferred is encoded to obtain the text identifier sequence corresponding to the text to be inferred.

10. A device for generating video with sound, characterized in that, include: The text processing module is used to preprocess the text to be reasoned, and obtain the text identifier sequence corresponding to the text to be reasoned. The video generation module is used to input the text identifier sequence into the audio video generation model to generate an audio video corresponding to the text to be inferred. The audio-visual generation model includes an autoregressive audio-visual sequence generation model, an audio-visual vector quantization autoencoder, a video frame interpolation model, an audio conversion model, and audio-visual processing tools. The autoregressive audio-visual sequence generation model is an autoregressive trimodal joint Transformer neural network Decoder model. This model is used to identify the text identifier sequence corresponding to the user-input text to be inferred, and to generate visual and audio identifier sequences through text semantic understanding and cross-modal relationships. The cross-modal relationships refer to the association between multimodal information, which includes text information, video image information, and audio information. The step of inputting the text identifier sequence into the audio video generation model to generate the audio video corresponding to the text to be inferred includes: Step 21: Input the text identifier sequence into the autoregressive audio-visual sequence generation model to generate a discrete identifier sequence of video image frames and a discrete identifier sequence of audio spectrum corresponding to the text identifier sequence; Step 22: Input the discrete identifier sequence of the video image frame and the discrete identifier sequence of the audio spectrum into the decoder of the audio-video vector quantization autoencoder to generate video image frames and audio Mel spectrum; Step 23: Input the generated video image frames into the video frame interpolation model to synthesize a silent video; Step 24: Input the audio Mel spectrum into the audio conversion model to synthesize an audio signal; the duration of the silent video matches the duration of the audio signal. Step 25: Input the silent video and the audio signal into the audio and video processing tool to generate the video with sound.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the audio-visual generation method as described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the audio-visual generation method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the audio-visual generation method as described in any one of claims 1 to 9.

Images